summaryrefslogtreecommitdiff
path: root/taskflow/persistence/backends/impl_sqlalchemy.py
Commit message (Collapse)AuthorAgeFilesLines
* Rework the sqlalchemy backendJoshua Harlow2015-02-101-247/+183
| | | | | | | | | | | | | | | We can just simplify the usage of an sqlalchemy if we just use (we already have our own ORM like objects anyway) sqlalchemy core in the first place and have a very tiny layer that converts back and forth from our very limited object model that we use in our persistence layer. This change makes that adjustment, which makes it easier to read and understand the actions the sqlalchemy backend is doing when saving, reading and updating data, and avoids yet another layer that isn't useful for our purposes anyway. Change-Id: I911c509f65e7845aee86fed1622eaa56970741f2
* Merge "Move the persistence base to the parent directory"Jenkins2015-01-211-1/+1
|\
| * Move the persistence base to the parent directoryJoshua Harlow2014-12-191-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | In order to match the directory/module layout of the other pluggable backends better move the persistence base module that defines the base abstract classes up into the parent directory. This makes it easier to look at the taskflow code-base and understand the common layout. Change-Id: I7887cb0241b8fe65cbdfee32c101c3df5f05d27c
* | Merge "Move implementation(s) to there own sections"Jenkins2015-01-211-4/+4
|\ \
| * | Move implementation(s) to there own sectionsJoshua Harlow2015-01-161-4/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Instead of putting implementation(s) under the interfaces section put the implementation(s) under there own section. This also includes some other tweaks to refer to those implementation(s) where appropriate. Change-Id: Iffdc0439c843e7f70cf873e5a75501feb51f96c7
* | | Merge "Add back a 'eventlet_utils' helper utility module"Jenkins2015-01-191-2/+2
|\ \ \ | |/ / |/| |
| * | Add back a 'eventlet_utils' helper utility moduleJoshua Harlow2015-01-161-2/+2
| |/ | | | | | | | | | | | | | | | | | | | | Recreate a very simple eventlet utility module that has only a few features; one function checks if eventlet is available and if not raise an exception; and a constant that can be used by calling code (such as tests or other optional functionality) to check if eventlet is useable before proceeding. Change-Id: I32df0702eeae7c7c78972c9796156dd824b2f123
* | Switch to using 'oslo_utils' vs 'oslo.utils'Joshua Harlow2015-01-141-1/+1
|/ | | | | | | | Prefer the non-deprecated 'oslo_utils' instead of the namespaced 'oslo.utils' wherever it was previously used. Change-Id: I9a78150ef5266e1ff22147278162fe3cfe1b2e3f
* Merge "Remove default setting of 'mysql_traditional_mode'"Jenkins2014-12-181-2/+0
|\
| * Remove default setting of 'mysql_traditional_mode'Joshua Harlow2014-11-251-2/+0
| | | | | | | | | | | | | | | | | | | | | | The sqlalchemy versions we are using/supporting (0.6+) have the detection of the mysql mode built-in so only activate the connect setting if we are somehow overriden by a user who knows what they are doing. Fixes bug 1396278 Change-Id: If2226d3e9f921a1c5f62a6727016fe86cd50a9b5
* | Add and use a new simple helper logging moduleJoshua Harlow2014-12-081-1/+1
|/ | | | | | | | | Add a new logging BLATHER level to easily allow its usage for messages that are below the normal DEBUG level such as compilation information and scope lookup info which can be very verbose in logs if always enabled. Change-Id: I828211403bd02bfd6777b10cdcfe58fb0637a52c
* Remove direct usage of the deprecated failure locationJoshua Harlow2014-10-211-1/+2
| | | | | | | | Internally we should be using the new location and not the deprecated location wherever possible. This avoids emitting warnings messages on our own code, which is a dirty habit. Change-Id: Idac5a772eca7529d92542ada3be1cea092880e25
* Add a futures type that can unify our future functionalityJoshua Harlow2014-10-181-2/+2
| | | | | | | | | | | | | | | | | Move the currently existing green future executor and associated code to a new futures types module so that it can be accessed from this new location (TODO: deprecate the old location and link the old to the new for one release so that we can remove the old link in N + 1 release). This unifies the API that the existing pool (thread or process) future executors and the green thread pool future executor, and the newly added synchronous executor (replacing the previous `make_completed_future` function) provide so there usage is as seamless as possible. Part of blueprint top-level-types Change-Id: Ie5500eaa7f4425edb604b2dd13a15f82909a673b
* Switch to using oslo.utils and oslo.serializationJoshua Harlow2014-09-201-1/+1
| | | | | | | | | | | | Instead of copying modules from the incubator into taskflow we can now directly use these same modules from supported libraries instead so this moves the usage of everything except uuidutils which wasn't moved over to using those newly published libraries. Part of blueprint integrate-and-use-oslo-utils-serialization Change-Id: I1183bda96e1ddb062d9cab91990186f0f56f0a0e
* Merge "Remove misc.as_bool as oslo provides an equivalent"Jenkins2014-06-181-5/+18
|\
| * Remove misc.as_bool as oslo provides an equivalentJoshua Harlow2014-06-111-5/+18
| | | | | | | | | | | | | | | | | | There isn't a need to have a misc.as_bool function anymore now that we have imported the oslo incubator strutils module since that module provides a function that does *nearly* the same thing. Change-Id: I7afe141d5a37c50b0c926144743f9af71db95bbf
* | Upgrade hacking version and fix some of the issuesJoshua Harlow2014-06-131-3/+15
|/ | | | | | | | | | | | | | | | | Update hacking to the new requirements version and fix about half of the new reported issues. The other hacking issues are for now ignored until fixed by adjusting our tox.ini file. This commit fixes the following new hacking errors: H405 - multi line docstring summary not separated with an empty line E265 - block comment should start with '# ' F402 - import 'endpoint' from line 21 shadowed by loop variable Change-Id: I6bae61591fb988cc17fa79e21cb5f1508d22781c
* Merge "Include the function name on internal errors"0.3.21Jenkins2014-06-101-2/+5
|\
| * Include the function name on internal errorsJoshua Harlow2014-06-041-2/+5
| | | | | | | | | | | | | | | | | | To make it easier to debug and reason about the exception that occurred include the function name that was called during the sessions activation in the exception message (and associated LOG output). Change-Id: I21f9310f78968d1e60d88b1f77be0dc629e75525
* | Merge "Allow the mysql mode to be more than just TRADITIONAL"Jenkins2014-06-101-8/+19
|\ \
| * | Allow the mysql mode to be more than just TRADITIONALJoshua Harlow2014-06-041-8/+19
| |/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | To make it easier to later move to oslo.db match there ability to set a different mysql mode (this will default, unless configured explicitly off/differently to TRADITIONAL). To ensure that we retain backwards compatibility the prior existence of 'mysql_traditional_mode' as a boolean option will by default set the mode to TRADITIONAL, and a new configuration option 'mysql_sql_mode' can be used to provide a secondary overriding mode that will be used instead (incase someone wants to use something other than TRADITIONAL). Closes-Bug: 1326568 Change-Id: Ide34c27b12c26030c8842f3f4b0fcca43ce783a7
* | Add in default transaction isolation levelsJoshua Harlow2014-06-041-5/+29
|/ | | | | | | | | | | Apply a default setting for transaction isolation levels for mysql and postgresql to help avoid consistency issues that happen when two transactions occur at the same time on the same set of records. Closes-Bug: 1326507 Change-Id: I1819722889d0d66d938641af6aa6f79fcfd2deb4
* Use correct exception in the timing listenerStanislav Kudriashev2014-04-011-2/+2
| | | | | | | Used `StorageFailure` exception instead of non-existing `StorageError` in the timing listener. Change-Id: I83035b737f7507b760799a5d44d4c7d097103ae5
* Revert "Move taskflow.utils.misc.Failure to its own module"Jeremy Stanley2014-03-311-4/+2
| | | | | | | | | | | | This reverts commit 42ca240e8157b840c298d14fbf478ae570376633 which was a breaking change in a library consumed by other OpenStack projects with no deprecation or backwards compatibility considerations. It was able to merge because openstack/taskflow is apparently not yet part of the integrated gate via the proposed I202f4809afd689155e2cc4a00fc704fd772a0e92 change. Change-Id: I96cf36dc317499df91e43502efc85221f8177395 Closes-Bug: #1300161
* Move taskflow.utils.misc.Failure to its own moduleIvan A. Melnikov2014-03-311-2/+4
| | | | | | | | | | Failure class is important part of TaskFlow API, so it should be more visible and accessible. Breaking change: any client that used taskflow.utils.misc.Failure should be updated. Change-Id: Ib30000c9246bbcb227b34dfb0aba4d0b950bf926
* Persistence cleanup part oneJoshua Harlow2014-03-261-83/+92
| | | | | | | | | | | | | | | | | | | | | | | | | | | - Convert the various functions that take a task detail into ones that take atom details (since this is now the generic type they should take). - Don't expose the detail type strings as part of the atom detail api, leave those as private hidden strings and provide conversion functions from string<->class instead. - Have the logbook objects contain the following new methods to reduce the dependence on persistence_utils to do the same. - to_dict() which converts the current object into a dict - from_dict() which converts the provided dict into a object - merge() which merges a incoming objects data with the current objects - Have the persistence backends + storage + action engine use these new methods instead of there current usage. - Don't compare to logbook.RETRY_DETAIL or logbook.TASK_DETAIL since python has the isinstance function just use it (ideally we should fix the code so that this isn't even needed, usage of isinstance means something is not designed/structured right). - In storage tests we can't assume that failures will be non-lossy since under certain backends when a failure is stored information about the internally held exc_info is lost, so take this into account when testing by using matches() where applicable. Change-Id: Ie8a274cfd4cb4e64e87c355dc99d466d74a4e82c
* Adjust the exception hierachyJoshua Harlow2014-03-241-8/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Group the exceptions into the following groups * Storage * Jobs * Execution * Other (wrapped failure here) This grouping makes it easier to understand where one type of exception should be used vs using another type of exception. Backwards incompatible changes: * StorageError -> StorageFailure * AlreadyExists -> Duplicate * WrappedFailure now inherits from Exception and not directly from TaskFlowException since it wraps arbitrary other exceptions and is not specific to taskflow. Cleanups: * JobNotFound -> NotFound * EmptyFlow -> Empty * JobAlreadyExists -> AlreadyExists * InvariantViolation (X) * ConnectionFailure (X) Change-Id: I0e1e81b513fbbc7adb8bfaa1244993e345ab70d3
* Merge "Adjust logging levels and usage to follow standards"Jenkins2014-03-221-2/+2
|\
| * Adjust logging levels and usage to follow standardsJoshua Harlow2014-03-141-2/+2
| | | | | | | | | | | | | | | | | | | | A few of our logging levels and messages should be updated to better reflect the oslo logging standards created at https://wiki.openstack.org/LoggingStandards so update some of them to not report error when its really warning level and not report warning when debug is better. Change-Id: I8abdb1bcfa2893c6b97eda7e85ac779e456966b8
* | Add atom intentions for tasks and retriesAnastasia Karpinska2014-03-181-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | Add atom intentions: REVERT, EXECUTE, RETRY and IGNORE. Intentions will be used by action engine to schedule tasks correctly. Add intention to task detail and extend storage to work with atom intentions. Add alembic migration to add intentions column to database. Change-Id: I79c9bb5f11861658dbfedfd64ef93eb92b29cb2d
* | Extend logbook and storage to work with retryAnastasia Karpinska2014-03-141-4/+13
|/ | | | | | | | - Store AtomType in TaskDetail (TASK_DETAIL or RETRY_DETAIL) - Extend storage to work with retries - Add alembic migration for database Change-Id: I07896d4b79ebe5de33f65e00a3bd32af543a09b4
* Unify usage of storage error exception typeJoshua Harlow2014-02-241-14/+8
| | | | | | | | | | | | - Instead of formatting the exception by creating a storage error instance we can just let the already included cause make this formatting occur. - Handle cases in impl_dir which were not catching and raising storage errors. - Unify usage of 'backend internal error' messaging. - Pass the cause to storage errors where applicable. Change-Id: Ieb14256b202ccbfa3a96f68fa35db7e40f92114d
* Merge "Use six.text_type() instead of str() in sqlalchemy backend"Jenkins2014-02-241-4/+5
|\
| * Use six.text_type() instead of str() in sqlalchemy backendJoshua Harlow2014-02-201-4/+5
| | | | | | | | Change-Id: Ie78d2890ce008f22c94e705dbb489838375d303d
* | Remove extraneous vim configuration commentsyangxurong2014-02-141-2/+0
|/ | | | | | | | | Remove line containing comment - # vim: tabstop=4 shiftwidth=4 softtabstop=4 Change-Id: I7581cc88b8de433d5609ed06c6570b0b45c13573 Closes-Bug:#1229324
* Allow the usage of a passed in sqlalchemy engineJoshua Harlow2014-02-041-54/+71
| | | | | | | | | | | | | | In certain use-cases a user of taskflow will likely have a pre-existing engine that they want to use with out SQLAlchemyBackend. To allow them to provide such an engine via the backends constructor for use-cases where this makes sense. At the same time move the validation function the connection class so that it can be called by users to verify their connection is valid. Change-Id: I8150194570a84e19712d7d4ca66b79b7161a76e7
* Doc strings and comments clean-upskudriashev2014-01-261-5/+5
| | | | | | | | * Added missing period for doc strings * Correct syntax errors * Remove H402 from flake8 ignore list Change-Id: Ia8592bf99378e3658d6cca2ceb148bf9eb0b5de8
* Add connection_proxy paramJoshua Harlow2014-01-031-1/+1
| | | | | | | It seems like connection_proxy is needed as a param so include it so that calling this function doesn't fail. Change-Id: I7c013b23cf7ae61bd1b42c4412419aa847714efa
* Merge "Raise type error instead of silencing it"Jenkins2014-01-031-8/+4
|\
| * Raise type error instead of silencing itJoshua Harlow2013-12-301-8/+4
| | | | | | | | | | | | | | | | | | | | Instead of skipping the type error when invalid configuration values are provided instead we should just let the user know about these errors and let said user handle them appropriatly (by not passing in invalid configuration). Change-Id: Id17258e5e06b47bcabd2bfd9d02b9d21181244be
* | Allow max_backoff and use count instead of attemptsJoshua Harlow2014-01-011-1/+1
|/ | | | Change-Id: I18f7e2a32b290dcc5b09c9c0c1fa098c47e477a7
* Ensure that mysql traditional mode is enabledJoshua Harlow2013-12-211-0/+13
| | | | | | | | | | | | | | To avoid *silent* truncation of columns in mysql it is much safer to turn on traditional mode which behaves in the normal manner. From the mysql docs: Make MySQL behave like a "traditional" SQL database system. A simple description of this mode is "give an error instead of a warning" when inserting an incorrect value into a column. Change-Id: Ic62af8514c377d2b8c934449116498855e03d7bd
* Fix sqlalchemy 0.8 issuesOlga Kopylova2013-12-171-1/+1
| | | | | | | | | | | Use import of exc instead exception. Mutability tracking is changed in SQLAlchemy 0.8, but it seems that sqlalchmy.types.MutableType is not needed in classes Json and Failure from models. Closes-bug: 1206669 Change-Id: Ib13c2575ead495b22ed5624e68e044e2f4927201
* Fix up python 3.3 incompatabilitiesJoshua Harlow2013-11-211-1/+4
| | | | | | | | | | | | Make the python 3.3 testing work by selectively disabling & including eventlet, switch to testtools and testrepository which has 2.6, 2.7, 3.2+ unified testing support so that we can correctly run our tests in all supported python versions. Closes-Bug: #1251660 Co-authored-by: Alexander Gorodnev <agorodnev@griddynamics.com> Change-Id: I23b6f04387cfd3bf6b5a044edffa446ca897ce3a
* Correctly save task versionsIvan A. Melnikov2013-10-051-15/+5
| | | | | | | When task is added to storage or task detail is saved to persistence backend, task version should be saved with it. Change-Id: I43e97cb621c3ffbbc27cafdc206ac39389336a24
* Wrapping and serializing failuresIvan A. Melnikov2013-10-041-8/+12
| | | | | | | | | | | | * save task failures to sqlalchemy backend and restore them from there; * for any wrapped exception use common WrappedFailre exception type, which makes it easier to handle particular exception types; * several minor improvements. Affects-db: this change adds db migration that looses exception information for saved task details. Change-Id: I575282002e6999646bbf51f492b82a7e3525787a
* Create database from models for SQLiteIvan A. Melnikov2013-09-301-2/+11
| | | | | | | | | | Alembic does not support SQLite, and we don't recommend to use SQLite in production deployments, so migrations are rarely needed for SQLite. So we don't bother about working around SQLite limitations, and create database from models when it is in use. Of course, we still use alembic to migrate schema on 'real' databases. Change-Id: I8237a29c7bf5b60136bb1b72fd029fe9ce1abe1f
* Remove weakref usageJoshua Harlow2013-09-221-2/+1
| | | | | | | | | | | | | | | | | | If a connection escapes the scope of the backend object, then python will actually garbage collect the backend object even though the connection object has a property that allows you to fetch the backend. If the property is used after the gc occurs a weakref failure occurs. Python is smart enough to correctly deallocate these types of object links by itself so lets just let it do that. Fixes: bug 1228968 Change-Id: I49d36b74f896bd1be6c7a1a373b07bdaded3ef4b
* Move toward using a backend+connection modelJoshua Harlow2013-09-121-0/+519
Instead of having a pretty restrictive module based api for saving logbook objects it is much more friendly and extensible to move toward a more ceilometer-influenced engine and connection based storage backend using stevedore to do the backend loading instead of a custom registration/fetching mechanism. This allows us to provide a base object oriented backend api that can be easily inherited from to allow for customized & pluggable backend storage modules. Implements blueprint stevedore-based-backends Implements blueprint ceilometer-influenced-backends Change-Id: Ib5868d3d9018b7aa1a3354858dcb90ca1a04055d
View Lorry Controller Status