summaryrefslogtreecommitdiff
path: root/taskflow/jobs
Commit message (Collapse)AuthorAgeFilesLines
* Merge "Fix parsing of zookeeper jobboard backend options"Zuul2023-04-191-1/+3
|\
| * Fix parsing of zookeeper jobboard backend optionsGregory Thiemonge2023-01-121-1/+3
| | | | | | | | | | | | | | | | Fix the zookeeper backend options when values are passed as strings, a "False" string is now treated as the False boolean. Closes-Bug: #1999174 Change-Id: I048faf06d89ebf980efe0598e647f2ec89f65ada
* | Replace abc.abstractproperty with property and abc.abstractmethodljhuang2022-08-031-6/+12
|/ | | | | | | | | Replace abc.abstractproperty with property and abc.abstractmethod, as abc.abstractproperty has been deprecated since python3.3[1] [1]https://docs.python.org/3.8/whatsnew/3.3.html?highlight=deprecated#abc Change-Id: I1bcecd99d8856c26621a5304d9f7f01f8f111918
* Remove sixTakashi Kajinami2022-05-183-19/+14
| | | | | | | | This library no longer supports Python 2, thus usage of six can be removed. This also removes workaround about pickle library used in Python 2 only. Change-Id: I19d298cf0f402d65f0b142dea0bf35cf992332a9
* Handle invalid redis entries in RedisJobBoardGregory Thiemonge2022-01-101-1/+4
| | | | | | | | | | | | | When catching that a redis buffer is not formatted correctly, _fetch_jobs now deletes the data from the hashmap, so the same error will not occur during the next call to the function. The commit also adds JobFailure in the list of caught exceptions (might be raised by _loads) in _fetch_jobs Closes-Bug: #1949950 Change-Id: I9078086cd4c3dbfa954422dfa3d5b19fefd7264f
* Add sentinel redis supportAnn Taraday2020-08-051-1/+14
| | | | | | | | | | | | | Redis client has an ability to connect to Redis server using Sentinel[1] (especially important for Redis clusters), but this ability was missing here. Allow to pass 'sentinel' variable to Redis conf, extend _make_client to use sentinel for that case. [1] - https://github.com/andymccurdy/redis-py#sentinel-support Change-Id: Ia8cc98e701435fd0231da3724f5d7108fc4f96f4
* Update various links in docsVu Cong Tuan2018-07-172-5/+5
| | | | | | Avoid redirects, s/http/https/. Change-Id: Id03bd49f250b48109967a0e04b92b75925f08932
* Avoid log warning when closing is underway (on purpose)Joshua Harlow2017-08-011-1/+8
| | | | | | Related-Bug: #1557107 Change-Id: I8b2f327dadbf038cd050f05fbc46a428282a3d82
* do not allow redis job reclaim by same ownerRick van de Loo2017-05-131-5/+3
| | | | | | | | | | | | | | | Running a nonblocking conductor or two conductors on the same host will re-execute the same job multiple times with the current implementation of 'claim' for the redis jobboard backend. This is different from the ZooKeeper jobboard backend, there the same owner of a job is not allowed to reclaim the job again (https://github.com/openstack/taskflow/blob/master/taskflow/jobs/backends/impl_zookeeper.py#L554). If the same owner is allowed to reclaim the job again there can be no concurrent execution on the same owner because all jobs will be re-claimed and re-executed by the same owner every pass as long as it's on the jobboard. To reproduce this behavior: - Use the redis jobboard backend - Create a flow with a task that sleeps 10 seconds in the execute method - Post that flow as a job - Run a nonblocking conductor It will claim and execute the same job multiple times in a loop until the first worker is finished and consumes the job. After this change it will not re-execute the same job multiple times. Change-Id: I4f6c364211500e510fc496f23b03ce056771417d
* python3.0 has deprecated LOG.warnxhzhf2017-03-072-13/+14
| | | | | | | | python3.0 has deprecated LOG.warn https://docs.python.org/3/library/logging.html#logging.warning Closes-Bug: #1650843 Change-Id: Ib8473cbdb84547f385e619fe08ed723b1ee7bcdb
* Some classes not define __ne__() built-in functionJi-Wei2016-08-312-0/+6
| | | | | | | | | | Some classes defines __eq__() built-in function, but does not define __ne__() built-in function, so self.assertEqual works but self.assertNotEqual does not work at all in this test case in python2. This patch fixes it. Change-Id: I3e4f213081268bad44583a63a84795d39094117f Closes-Bug: #1586268
* Merge "Ensure the fetching jobs does not fetch anything when in bad state"Jenkins2016-07-251-8/+66
|\
| * Ensure the fetching jobs does not fetch anything when in bad stateJoshua Harlow2016-05-311-8/+66
| | | | | | | | | | | | | | | | | | | | When the underlying connection is in LOST or SUSPENDED mode do not allow jobs to be iterated over (and clear the local cache when the connection has been LOST). Related-Bug: #1557107 Change-Id: Ic0a2ab2519ff8a7386d80d9092a0e24579883681
* | Fix some misspellings in the function name and descriptionsPablo Iranzo Gómez2016-06-302-2/+2
|/ | | | Change-Id: I7e3451feb94b1f25b00c5e7b197bb6b527548306
* Merge "Add rundimentary and/or non-optimized job priorities"Jenkins2016-04-283-70/+209
|\
| * Add rundimentary and/or non-optimized job prioritiesJoshua Harlow2016-01-093-70/+209
| | | | | | | | | | | | | | | | | | | | | | | | Allow for providing job priority on job posting and order iteration by priority (this does not store in any special way different priorities under different keys/paths, but it could be done that way in the future). This is backwards compat in that all old jobs posted without a priority assume a normal/default priority. Change-Id: Ie670023d624f4442a16a1278a3aee0d88b102842
* | Merge "Rename '_emit' -> '_try_emit' since it is best-effort (not ensured)"Jenkins2016-01-131-6/+8
|\ \ | |/ |/|
| * Rename '_emit' -> '_try_emit' since it is best-effort (not ensured)Joshua Harlow2016-01-041-6/+8
| | | | | | | | Change-Id: If45e46c35a17f9fd347c7a09ec7b7588c7a266e2
* | Merge "Add ability of job poster/job iterator to wait for jobs to complete"Jenkins2016-01-091-0/+40
|\ \
| * | Add ability of job poster/job iterator to wait for jobs to completeJoshua Harlow2015-12-161-0/+40
| | | | | | | | | | | | | | | | | | | | | | | | It is quite useful for someone who has posted a job to be able to easily wait for its completion (up to a given timeout) so add a wait method onto the job class to allow for this to be possible. Change-Id: Id3a7c724020962591e323da0febfd0c71d1acc50
* | | Use shared util helper for driver name + config extractionJoshua Harlow2015-12-221-11/+1
| |/ |/| | | | | Change-Id: I43465b8f5868e64bdf38d2873417a8a4a403a23b
* | Translate kazoo exceptions into job equivalents if register_entity failsJoshua Harlow2015-12-141-9/+23
|/ | | | Change-Id: I8c0abef6924f16c8b017193451e1ee273ad1d754
* Use conductor entity class constant instead of raw stringJoshua Harlow2015-11-131-1/+2
| | | | | | | | | Now that the conductor base class provides a constant that all entity objects produced from it should provide we can now use that to ensure we are only registering a limited set of entity kinds. Change-Id: I0e71a8da64a228fdc68c9de941aaccf54d493d96
* Add a executor backed conductor and have existing impl. use itJoshua Harlow2015-11-121-1/+6
| | | | | | | | | | | | | | | | | | | This adds a executor backed job dispatching base class and has the existing blocking executor use it by running jobs and dispatching jobs into a sync executor. It also allows for dispatching jobs into a thread executor, or other executor via a new '_executor_factory' method that can generate executors (it can be overriden in the non-blocking executor to provide your own executors instances). This does alter the behavior in that now that jobs are dispatched into an executor we no longer can immediatly know if a job was dispatched and raised an exception or whether it will raise an exception in the future, so we now alter the 'local_dispatched' to just be a boolean that is used to determine if any dispatches happened (failure or not). Change-Id: I485770e8f4c85d3833892a453c9fb5168d8f0407
* Register conductor information on jobboardtonytan4ever2015-10-193-0/+36
| | | | Change-Id: I3bf935280a6e8b265045b09fde43d0ec7dc56f07
* Merge "Make currently implemented jobs use @functools.total_ordering"Jenkins2015-07-242-3/+28
|\
| * Make currently implemented jobs use @functools.total_orderingJoshua Harlow2015-07-082-3/+28
| | | | | | | | | | | | | | | | | | | | | | | | | | This allows jobs to be fully be comparable using the total ordering function to provide the complexity around the various functions needed to achieve/perform comparisons. Also fixes up the various job __eq__ and __lt__ methods to correctly return 'NotImplemented' on unknown types and adds a __hash__ to the redis job (so that it can be used in hashable collections, just like the zookeeper job class). Change-Id: I8820d5cc6b2e7f346ac329f011f41b76fa94b777
* | Merge "Unify the zookeeper/redis jobboard iterators"Jenkins2015-07-213-96/+105
|\ \
| * | Unify the zookeeper/redis jobboard iteratorsJoshua Harlow2015-07-123-96/+105
| |/ | | | | | | | | | | | | | | | | | | | | To make the zookeeper jobboard and redis jobboard iterjobs and wait functions that much similar have both return iterator objects from the same class iterator. This makes the iterator code and the jobboard code that much easier to follow and understand. Change-Id: Ia772cde881c2631002140e06684521fd42441534
* | Merge "Update 'make_client' kazoo docs and link to them"Jenkins2015-07-171-0/+10
|\ \
| * | Update 'make_client' kazoo docs and link to themJoshua Harlow2015-07-171-0/+10
| |/ | | | | | | | | | | | | | | | | Since the creation of a client is somewhat important and knowing what the options that are transfereed to kazoo are we should explicitly document what keys are and what the values should be. Change-Id: I1a5037b274828190270ea5c402be8b2100306de4
* | Merge "Provide a deprecated alias for the now removed stop watch class"Jenkins2015-07-132-4/+3
|\ \
| * | Provide a deprecated alias for the now removed stop watch classJoshua Harlow2015-07-122-4/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This allows those who were using it to still continue using it until 2.0 where it will be removed; this makes it possible for those users to get off that code in a way that will be easily do-able (without totally breaking there code-bases, until we do that in the 2.0 release). This also removes all internal usage of that stop watch class so that the library itself will not reference it anymore. Change-Id: If313d8e7b9bdc8741db2e2e1dfb381aa3260b971
* | | Merge "Integrate futurist (and **remove** taskflow originating code)"Jenkins2015-07-111-2/+2
|\ \ \ | |/ /
| * | Integrate futurist (and **remove** taskflow originating code)Joshua Harlow2015-07-091-2/+2
| |/ | | | | | | Change-Id: If89baa042695f19e42b6368034f3ccf22c2cf0aa
* | Add more useful `__str__` to redis jobJoshua Harlow2015-07-081-0/+7
|/ | | | | | | | The information about the sequence and the owner_key is useful for debugging and for interacting with redis if the owner key ever needs to be manually removed. Change-Id: Ibe751d15938a2bcb83fb057f4c139f8f0022a60b
* Build-out + test a redis backed jobboardJoshua Harlow2015-06-293-134/+1093
| | | | | | Part of blueprint taskflow-redis-jobs Change-Id: I7c94e2201c5d933c8a1ec73fc0cf705962e5eef6
* Remove all 'lock_utils' now that fasteners provides equivalentsJoshua Harlow2015-06-041-3/+3
| | | | | | Depends-On: I442249783da4a6ae10c78b95e0b279409c95d2e6 Change-Id: I877928c858e8d6176d3f01ad9de2765104acf5c3
* Remove 'pass' usage not neededJoshua Harlow2015-05-241-2/+0
| | | | | | | | These usages are not really needed and can just be eliminated since they offer no benefit (and coverage may classify them as dead/useless code). Change-Id: I951c4cb9216a19ce37dc8fc390ae905fd525a3c7
* Remove validation of state on state read property access0.10.0Joshua Harlow2015-05-071-6/+1
| | | | | | | | | | | This should be done (and are done, but not with the best exception messages) in a more appropriate place (in the methods of the jobboard that mutate a job via some manner, aka, in consume, claim, abandon, trash...). Part of fix for bug 1452388 Change-Id: If29bc73cc6e9282ad35a50ce0041efc3ff8f71d9
* Make the default path a constant and tweak class docstringJoshua Harlow2015-05-061-16/+20
| | | | | | | | Have the default path used be a class level constant (that is picked up by sphinx) and tweak the docstring to be formatted better and to read better (with less grammatical issues). Change-Id: I43b78a2b36ed92dbd6dfb4d7376535ad361dc03d
* Move implementations into there own sub-sectionsJoshua Harlow2015-05-011-6/+6
| | | | | | | | Also fixes up some inline-code/examples docs to correctly display in the generated docs (and tweaks some URI capitalization). Change-Id: I001ef2460eb5e9a884ca6db6e8d6f72864191fe7
* Merge "Avoid attribute error by checking executor for being non-none"Jenkins2015-04-291-5/+9
|\
| * Avoid attribute error by checking executor for being non-noneJoshua Harlow2015-04-201-5/+9
| | | | | | | | Change-Id: I47e7a14a93026918f4a469139ca6a53dbaa1f702
* | Move zookeeper jobboard constants to class levelJoshua Harlow2015-04-271-23/+31
| | | | | | | | | | | | | | | | To make it easy to document and inquire on the zookeeper job constants expose them as class level instead of module level constants (and use them where needed). Change-Id: Ice53db104c55ba4a7794a358d56922074d44492b
* | Replace more instance(s) of exception chaining with helperJoshua Harlow2015-04-201-5/+6
|/ | | | Change-Id: I5a35194022656786c329ea3586366d3b00b461ba
* Merge "Add strict job state transition checking"Jenkins2015-04-181-0/+8
|\
| * Add strict job state transition checkingJoshua Harlow2015-04-161-0/+8
| | | | | | | | | | | | | | | | Also adds basic unit tests to test that the states module functions have some level of testing to ensure they don't break. Change-Id: Icedd6e70820bcf484c0dace76f2acf01d9bc967e
* | Merge "Use the ability to chain exceptions correctly"Jenkins2015-04-181-54/+77
|\ \ | |/ |/|
| * Use the ability to chain exceptions correctlyJoshua Harlow2015-04-031-54/+77
| | | | | | | | | | | | | | | | | | | | | | In the zookeeper jobboard (and persistence backends) we are associating the cause of a new exception when raising a exception. Using the new exception helper we can make this work better on py2.x and py3.x so that the py3.x version has the chain setup correctly (while the py2.x version just uses the class 'cause' attribute instead). Change-Id: Ieeac2f70e1834d4612556565762ffd3be3e5b5a1
View Lorry Controller Status