summaryrefslogtreecommitdiff
path: root/taskflow
Commit message (Collapse)AuthorAgeFilesLines
* Replace deprecated import of ABCs from collections4.6.2Takashi Kajinami2021-07-171-2/+2
| | | | | | | | ABCs in collections should be imported from collections.abc and direct import from collections is deprecated since Python 3.3. Closes-Bug: #1936667 Change-Id: Ia6316894a67cc0ea7ee0d2110565b45546c813e7
* Merge "Fix flowdetails meta size"Zuul2021-06-011-0/+34
|\
| * Fix flowdetails meta sizeGregory Thiemonge2021-05-031-0/+34
| | | | | | | | | | | | | | | | | | | | meta field in flowdetails is defined as a JSON data type, but its data type is 'text' in mysql, which is limited to 64kbytes. JSON data type should have the same size as a LONGTEXT. Closes-Bug: #1926304 Change-Id: I9f89badfc697f0f26245ca7f4c22d62e220be5f9
* | Use unittest.mock instead of mockHervé Beraud2021-04-272-2/+2
|/ | | | | | | | The mock third party library was needed for mock support in py2 runtimes. Since we now only support py36 and later, we can use the standard lib unittest.mock module instead. Change-Id: Ib169e3deb7ddb2bc93a206ebec4043552281aa7f
* Fix deprecated Alembic function args4.6.0Adam Harwell2021-01-281-9/+9
| | | | | | | | create_index and create_foreign_key were using long-deprecated argument names which were finally removed as of 1.5.0: https://alembic.sqlalchemy.org/en/latest/changelog.html#change-da0bea3199b905783ddbd0cde968beec Change-Id: I03405e3d0daf085d0849a6e3effc65ec05ceb7a1
* Merge "Add sentinel redis support"4.4.0Zuul2020-08-242-1/+48
|\
| * Add sentinel redis supportAnn Taraday2020-08-052-1/+48
| | | | | | | | | | | | | | | | | | | | | | | | | | Redis client has an ability to connect to Redis server using Sentinel[1] (especially important for Redis clusters), but this ability was missing here. Allow to pass 'sentinel' variable to Redis conf, extend _make_client to use sentinel for that case. [1] - https://github.com/andymccurdy/redis-py#sentinel-support Change-Id: Ia8cc98e701435fd0231da3724f5d7108fc4f96f4
* | Avoid endless loop on StorageFailureAnn Taraday2020-08-241-0/+11
|/ | | | | | | | | | | If an error occures with writing atom detail in database( persistence backend) flow execution enters an endless loop throwing errors and retrying to save details. Avoid this situation and log exception message. Change-Id: Ic6b0a78d20124cc027468ecc6aeff189c25d1a8a Closes-bug: 1889773
* Switch from unittest2 compat methods to Python 3.x methodsmelissaml2020-07-054-50/+50
| | | | | | | | | | | | | | | | | With the removal of Python 2.x we can remove the unittest2 compat wrappers and switch to assertCountEqual instead of assertItemsEqual We have been able to use them since then, because testtools required unittest2, which still included it. With testtools removing Python 2.7 support [3][4], we will lose support for assertItemsEqual, so we should switch to use assertCountEqual. [1] - https://bugs.python.org/issue17866 [2] - https://hg.python.org/cpython/rev/d9921cb6e3cd [3] - testing-cabal/testtools#286 [4] - testing-cabal/testtools#277 Change-Id: Iaa8251a1e9965a00fe99b7a740a104c011260340
* Stop to use the __future__ module.Hervé Beraud2020-06-029-18/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | The __future__ module [1] was used in this context to ensure compatibility between python 2 and python 3. We previously dropped the support of python 2.7 [2] and now we only support python 3 so we don't need to continue to use this module and the imports listed below. Imports commonly used and their related PEPs: - `division` is related to PEP 238 [3] - `print_function` is related to PEP 3105 [4] - `unicode_literals` is related to PEP 3112 [5] - `with_statement` is related to PEP 343 [6] - `absolute_import` is related to PEP 328 [7] [1] https://docs.python.org/3/library/__future__.html [2] https://governance.openstack.org/tc/goals/selected/ussuri/drop-py27.html [3] https://www.python.org/dev/peps/pep-0238 [4] https://www.python.org/dev/peps/pep-3105 [5] https://www.python.org/dev/peps/pep-3112 [6] https://www.python.org/dev/peps/pep-0343 [7] https://www.python.org/dev/peps/pep-0328 Change-Id: Ie7c18f2f1b6294e3d19c36112d35f4472ce77e7b
* Import modules, not classesSean McGinnis2020-05-012-9/+4
| | | | | | | | This addresses comments from I24b13c8654a33d99bf687a44a36cfdace39e3866 to follow guidelines to only import modules, not classes. Change-Id: I0e66c08bfcdff306bac0121602d45b56cfb2c712 Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
* Drop use of deprecated collections classesSean McGinnis2020-04-143-15/+8
| | | | | | | | | | | Many classes have moved from collections to collections.abc. The original imports work, but emit deprecation warnings. Now that we are py3-only, we can drop any references to the older deprecated locations. Change-Id: I24b13c8654a33d99bf687a44a36cfdace39e3866 Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
* Zookeeper backend SSL supportussuri-em4.1.0Ann Kamyshnikova2020-03-061-0/+13
| | | | | | | | | | | | | | | | | | Bump kazoo version to 2.6.0 that supports SSL. Allow to configure ssl config options via conf * keyfile * keyfile_password * certfile * use_ssl * verify_certs https://kazoo.readthedocs.io/en/latest/api/client.html#kazoo.client.KazooClient This should be done in conformance with Redis jobboard backend, that already allows SSL settings. Change-Id: Ic2670c3c4caa93f69bc2d541c3701df2b6037d2e
* Merge "Update TaskFlow for networkx 2.x"3.8.0Zuul2019-11-1515-165/+73
|\
| * Update TaskFlow for networkx 2.xMichael Johnson2019-10-1815-165/+73
| | | | | | | | | | | | | | | | | | | | | | The networkx 2.x series has been out for two years now and supports python 3.6 and greater[1]. This patch updates TaskFlow to require a minimum of networkx 2.1. It also updates the code to support recent deprecation expiration introduced in the 2.4 release. [1] https://networkx.github.io/documentation/stable/news.html Change-Id: Ife31d353ba80824ebc63c8b21ee90943badc8da3
* | Fix python3.8 hmac compatibilityEric Harney2019-09-031-1/+2
|/ | | | | | | | | The "digestmod" parameter must be passed in for Python 3.8. md5 was chosen here because this is what was implicitly used for Python < 3.8. Change-Id: I17eca0692c4a89ff90cfa9d4ef3947d2c9ee23e7
* Use mysql LONGTEXT for atomdetails resultstrain-em3.7.1Ann Taraday2019-08-091-0/+43
| | | | | | | | | | | | | | | With a lot of possible retries using mysql sa.Text for result field atomdetails table is not enough. Updated 'results' column type to LONGTEXT in mysql backend case. PosgreSQL seems not to be affected as postgres text type is not having length limitation. Co-Authored-By: Michael Johnson <johnsomor@gmail.com> Change-Id: I4e44e3288ae49d85c5b5ce8ff26ba83f3b9d2c7b Closes-bug: #1838015
* Handle collections.abc deprecationsSean McGinnis2019-05-091-1/+11
| | | | | | | | | | | | | | | The use of ABC classes directly from collections has been deprecated in 3.x versions of Python. The direction is to use the classes defined in collections.abc. Python 2.7 does not have this, but Python 3.8 will be dropping the backwards compatibility to use the old location. Six also does not have support for this yet, so in the mean time to make sure we don't run into issues as folks try to move to 3.8, and to get rid of deprecation warnings in logs, this handles importing from the preferred location and falls back if it not available. Change-Id: I15554bf3c109045ebdc237ce7cb40299f5d1b298 Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
* Remove the duplicated wordmelissaml2018-08-211-1/+1
| | | | Change-Id: Ia9964fd613efc49479e763b6a7095d63115179ab
* Create KazooClient with taskflow loggerTheodoros Tsioutsias2018-08-161-0/+5
| | | | | | | | | KazooClient should use the taskflow logger and not the default one. Using the default logger means that the user cannot control its log level Change-Id: Id9549028522955ece0ae9f675c5ecc1c2fd816de Closes-Bug: #1785036
* Update various links in docsVu Cong Tuan2018-07-173-13/+13
| | | | | | Avoid redirects, s/http/https/. Change-Id: Id03bd49f250b48109967a0e04b92b75925f08932
* Fix code to support networkx > 1.0Michal Arbet2018-07-116-16/+172
| | | | | | | | | | | | | With the release of NetworkX 2.0 the reporting API was moved to view/iterator model. Many methods were moved from reporting lists or dicts to iterating over the information. Methods that used to return containers now return views and methods that returned iterators have been removed in networkx. Because of this change in NetworkX 2.0 , taskflow code have to be changed also to support networkx > 2.0 Change-Id: I23c226f37bd85c1e38039fbcb302a2d0de49f333 Closes-Bug: #1778115
* Merge "Trivial: update pypi url to new url"Zuul2018-05-171-1/+1
|\
| * Trivial: update pypi url to new urlXiaojueGuan2018-05-131-1/+1
| | | | | | | | Change-Id: I172077bc2b4cf5e02691c98b07860f853f9f4da9
* | Fix doc buildBen Nemec2018-05-092-5/+11
|/ | | | | | | | | | | | | | | | | | | | Most notably, taskflow is hitting the sphinx issue https://github.com/sphinx-doc/sphinx/issues/2549 which causes a spurious warning that breaks the build with -W. There is a workaround posted in https://stackoverflow.com/questions/31784830/sphinx-ivar-tag-goes-looking-for-cross-references to move :ivar: docstrings to inline comments on the member variable itself. This is not ideal because it causes the docs to render differently from :ivar:, but until the sphinx bug is fixed it will allow us to keep documenting the problematic variables. There was also a problem with one of the doctests because the output had changed. That is now fixed. I also noticed a typo in one of the parameter descriptions so that is fixed too. Change-Id: Ib44621f6c3ba2c5476ec430218a0449f9f45d18f
* set default python to python3Doug Hellmann2018-04-173-3/+6
| | | | | | | | | | | | | | | | | | Set the default python to python3 except for the py27 environment. We have to set that explicitly to override the new default. Have py27 and py3* jobs run same commands by removing the commands entry for py27. The pep8 rules under python 3 are different, so we need to fix a couple of issues with blank space. The E721 rule is ignored entirely because we are not yet using a version of flake8 that supports ignoring individual rules on specific lines, although the instructions for that are added so that when we upgrade flake8 to a newer version we can remove the global rule. Change-Id: Id236b0632a83a188958130d04f1e9c476c5ace65 Signed-off-by: Doug Hellmann <doug@doughellmann.com>
* Fix invalid json unit testBen Nemec2018-02-081-1/+5
| | | | | | | | | | Recent versions of oslo.serialization have made it possible to dump exceptions to JSON, which broke a unit test in taskflow that assumed exceptions were unserializable. This change switches to an explicitly unserializable class for that test. Change-Id: If6d19bc9fcf1f1813cb087d42dc7ba6a61c71b3d Closes-Bug: 1748241
* Merge "Avoid log warning when closing is underway (on purpose)"3.0.1Zuul2017-12-121-1/+8
|\
| * Avoid log warning when closing is underway (on purpose)Joshua Harlow2017-08-011-1/+8
| | | | | | | | | | | | Related-Bug: #1557107 Change-Id: I8b2f327dadbf038cd050f05fbc46a428282a3d82
* | Remove class StopWatch from timingchenghuiyu2017-10-242-8/+2
| | | | | | | | | | | | | | | | In new release timeutils.StopWatch can be used from oslo.uitls, so it was deprecated for removal in timing.py Change-Id: I138b8d276998a4ea2537966767cbd14f96daf757 Closes-Bug: #1715043
* | Merge "Remove method blather in log adapter"2.17.0Zuul2017-10-241-7/+0
|\ \
| * | Remove method blather in log adapterchenghuiyu2017-09-041-7/+0
| |/ | | | | | | | | | | | | | | As method blather was using in new place, it was deprecated for removal in new release. Change-Id: Ic8d425e1774d6b7090ec105f2b37467a90c6e1e6 Closes-Bug: #1714873
* | Remove kwarg timeout in executor conductorchenghuiyu2017-09-011-3/+1
|/ | | | | | | The method stop returns immediately regardless of whether the conductor has been stopped, so kwarg timeout is no longer necessary, and it is deprecated for removal. Change-Id: I50e5cbb1a00eb9b6f358f150716d2a5d682a49c9
* Update URLs in documents according to document migrationChangBo Guo(gcb)2017-07-132-5/+5
| | | | Change-Id: I9ca92fdcec388e02462332e04fe7c1bf8b5f64b8
* Merge "Fix process based executor task proxying-back events"Jenkins2017-07-113-9/+87
|\
| * Fix process based executor task proxying-back eventsJoshua Harlow2017-07-113-9/+87
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Let's dive into what the problem is here. First a description of what happens to a task that is to be executed in a external (but local) process via the process executor mechanism. When a task is about to be sent to execute in the external (but local) process its first cloned, this is mainly done so that its notification callbacks can be altered in a safe manner (ie not altering the original task object to do this) and that clone has its notifier emptied out. What replaces the clone's notifier callbacks though is a new object (that has a __call__ method so it looks like just another callback) that will send messages to the parent process (the one that has the engine in it) over a secure(ish) channel whenever the local task triggers its notifier notify() method. This allows for callbacks in the parent process to get triggered because once the messages recieved the original tasks notifier object has its notify() method called (therefore those callbacks do not really know the task they are getting messages from is executing out of process). The issue though is that if the ANY(*) event type is registered due to how it works in the notifier is that if the child/cloned notifier has the ANY event type registered and the cloned task calls notify() with a specific event this will cause the ANY callback (in the clone) to transmit a message *and* it will cause the *specific* event callback to also transmit a message back to the parent process. On the engine process side it will get 2 messages and trigger the callbacks 3 times (twice for the specific event callback because how the local notifier has the ANY callback registered and one more time when the local process also sends the same event based on its registration of the ANY event in the child process). This is not what is expected (the message rcved on the engine process should only trigger one callback to get triggered if the engine process task has no ANY callback registered or two engine process callbacks to get triggered if the engine process task has the ANY callback registered). Closes-Bug: #1537948 Change-Id: I271bf1f23ad73df6c177cf00fd902c4881ba44ae
* | turn on warning-is-error in doc buildDoug Hellmann2017-07-084-5/+7
|/ | | | | Change-Id: Id87a1b17c1986d0e30629fed13bdfbecef4a3603 Signed-off-by: Doug Hellmann <doug@doughellmann.com>
* Replace assertRaisesRegexp with assertRaisesRegexVu Cong Tuan2017-06-0313-102/+102
| | | | | | | | | | assertRaisesRegexp was renamed to assertRaisesRegex in Py3.2 For more details, please check: https://docs.python.org/3/library/ unittest.html#unittest.TestCase.assertRaisesRegex Change-Id: I89cce19e80b04074aab9f49a76c7652acace78b3 Closes-Bug: #1436957
* do not allow redis job reclaim by same ownerRick van de Loo2017-05-132-5/+26
| | | | | | | | | | | | | | | Running a nonblocking conductor or two conductors on the same host will re-execute the same job multiple times with the current implementation of 'claim' for the redis jobboard backend. This is different from the ZooKeeper jobboard backend, there the same owner of a job is not allowed to reclaim the job again (https://github.com/openstack/taskflow/blob/master/taskflow/jobs/backends/impl_zookeeper.py#L554). If the same owner is allowed to reclaim the job again there can be no concurrent execution on the same owner because all jobs will be re-claimed and re-executed by the same owner every pass as long as it's on the jobboard. To reproduce this behavior: - Use the redis jobboard backend - Create a flow with a task that sleeps 10 seconds in the execute method - Post that flow as a job - Run a nonblocking conductor It will claim and execute the same job multiple times in a loop until the first worker is finished and consumes the job. After this change it will not re-execute the same job multiple times. Change-Id: I4f6c364211500e510fc496f23b03ce056771417d
* Merge "Stop using oslotest.mockpatch"2.11.0Jenkins2017-05-051-5/+4
|\
| * Stop using oslotest.mockpatchChangBo Guo(gcb)2017-04-281-5/+4
| | | | | | | | | | | | This module has been deprecated in favor of native fixtures. Change-Id: I40ed7360091fea51a257807035e923c85602949f
* | Merge "python3.0 has deprecated LOG.warn"Jenkins2017-05-0316-82/+88
|\ \
| * | python3.0 has deprecated LOG.warnxhzhf2017-03-0716-82/+88
| |/ | | | | | | | | | | | | | | python3.0 has deprecated LOG.warn https://docs.python.org/3/library/logging.html#logging.warning Closes-Bug: #1650843 Change-Id: Ib8473cbdb84547f385e619fe08ed723b1ee7bcdb
* | Fix py35 test failureChangBo Guo(gcb)2017-05-031-1/+1
|/ | | | | | | | dict.keys() returns object dict_keys on Python 3, can't work well with kombu recently, convert it to list before futhur processing. Closes-Bug: #1687855 Change-Id: I9dcfadd8f16eccce06f4cde3b2a6669f28a3f8bc
* Protect storage better against external concurrent accessJoshua Harlow2017-01-092-44/+62
| | | | | | | | | Lock down the various state machine action handling functions so that they are ensured correct (and consistent) access to the storage layer when they are modifiying and/or reading it. Change-Id: Ie893a44aa963ab515f19e77f9904f49c843cb4e5
* Rename engine analyzer to be named selectorJoshua Harlow2017-01-035-67/+67
| | | | | | | | | | | | | This moves out the engine next to run (or revert) selection process to a single class that just does this and moves out the common functions the analyzer class provided to the runtime object (which all components can access). This makes it easier to adjust the selection algorithm in different ways. Change-Id: I091c69297a7bff60729791d3ca6c3fae14d6eea5
* Merge "Remove vim header from source files"Jenkins2016-11-182-4/+0
|\
| * Remove vim header from source fileshowardlee2016-11-092-4/+0
| | | | | | | | | | | | | | | | Trivial fix [H106] Don't put vim configuration in source files (off by default). Change-Id: I7e9e9930fd6f135074078637f5af6f403e75ff78
* | Use assertIs(Not)None to check for Nonehowardlee2016-11-072-6/+6
|/ | | | | | | | | | | | | [H203] Use assertIs(Not)None to check for None (off by default) Unit test assertions tend to give better messages for more specific assertions. As a result, assertIsNone(...) is preferred over assertEqual(None, ...) and assertIs(None, ...), and assertIsNotNone(...) is preferred over assertNotEqual(None, ...) and assertIsNot(None, ...). Off by default. Trivial fix Change-Id: I5b6ac7d99f0689843eb98cb3e9b9b10531322640
* Merge "Replace retrying with tenacity"Jenkins2016-11-051-9/+7
|\
View Lorry Controller Status