summaryrefslogtreecommitdiff
path: root/lib
Commit message (Collapse)AuthorAgeFilesLines
* Also fix asyncio exception imports.fix_asyncio_run_py36Federico Caselli2021-01-241-10/+4
| | | | | | | | The python documentation imports everything directly from the asyncio package, and it seems that py < 3.8 does not have the asyncio.exception module Change-Id: I4719185f02f3c8d030ff24601e15f81767ba7d5a
* Merge "remove __allowlist__"mike bayer2021-01-231-4/+0
|\
| * remove __allowlist__Mike Bayer2021-01-231-4/+0
| | | | | | | | | | | | this testing element is not used Change-Id: I484c9a9f070122545fcdabe5a7f13b0bfca17023
* | Merge "Improve error message when await_ call errors"mike bayer2021-01-232-4/+15
|\ \ | |/ |/|
| * Improve error message when await_ call errorsFederico Caselli2021-01-212-4/+15
| | | | | | | | | | Fixes: #5832 Change-Id: Ia2ed8f1d1ec54e5f6e1a8f817a69446fdb3b7f6d
* | Merge "Removed some legacy terms in favor of modern equivalents. (D&I)"mike bayer2021-01-231-1/+1
|\ \
| * | Removed some legacy terms in favor of modern equivalents. (D&I)jonathan vanasco2021-01-211-1/+1
| |/ | | | | | | | | | | | | | | | | | | | | | | | | Migrated testing fixture: `TestBase.__whitelist__` -> `TestBase.__allowlist__` Migrated tox commands from deprecated to current: `whitelist_externals` > `allowlist_externals` Migrated test_session: `blacklist` -> `blocklist` Change-Id: I395d5ee977ff22fa703276b9b873cc96c59b9a35
* | Fix a couple of bugs in the asyncio implementationFederico Caselli2021-01-214-16/+56
|/ | | | | | | | | | | | | Log an informative message if a connection is not closed and the gc is reclaiming it when using an async dpapi, that does not support running IO at that stage. The ``AsyncAdaptedQueue`` used by default on async dpapis should instantiate a queue only when it's first used to avoid binding it to a possibly wrong event loop. Fixes: #5823 Change-Id: Ibfc50e209b1937ae3d6599ae7997f028c7a92c33
* Revert "Implement support for functions as FROM with columns clause support"Mike Bayer2021-01-218-527/+75
| | | | | | | This reverts commit 05a31f2708590161d4b3b4c7ff65196c99b4a22b. Atom has this little button called "push" and just pushes to master, I wasn't even *on* master. oops
* Implement support for functions as FROM with columns clause supportMike Bayer2021-01-208-75/+527
| | | | | | | WIP Fixes: #3566 Change-Id: I5b093b72533ef695293e737eb75850b9713e5e03
* chain joins from SelectState context, not SelectMike Bayer2021-01-201-1/+14
| | | | | | | | | | | | | Fixed issue in new :meth:`_sql.Select.join` method where chaining from the current JOIN wasn't looking at the right state, causing an expression like "FROM a JOIN b <onclause>, b JOIN c <onclause>" rather than "FROM a JOIN b <onclause> JOIN c <onclause>". Added :meth:`_sql.Select.outerjoin_from` method to complement :meth:`_sql.Select.join_from`. Fixes: #5858 Change-Id: I1346ebe0963bbd1e4bf868650e3ee1d6d3072f04
* Merge "Document Table/Column accessors"mike bayer2021-01-202-2/+112
|\
| * Document Table/Column accessorsMike Bayer2021-01-192-2/+112
| | | | | | | | | | | | | | | | | | | | | | | | | | As Sphinx will not allow us to add attributes to the .rst file while maintaining order, these have to be added as class-level attributes. Inlcude notes that "index" and "unique" parameters, while indicated by Column.index / Column.unique, do not actually indicate if the column is part of an index. Fixes: #5851 Change-Id: I18fbaf6c504c4b1005b4c51057f80397fb48b387
* | Merge "Disallow non-native psycopg2 Unicode in Python 3; update docs"mike bayer2021-01-191-56/+64
|\ \ | |/ |/|
| * Disallow non-native psycopg2 Unicode in Python 3; update docsFederico Caselli2021-01-191-56/+64
| | | | | | | | | | | | | | | | | | | | | | | | | | Fixed issue where the psycopg2 dialect would silently pass the ``use_native_unicode=False`` flag without actually having any effect under Python 3, as the psycopg2 DBAPI uses Unicode unconditionally under Python 3. This usage now raises an :class:`_exc.ArgumentError` when used under Python 3. Added test support for Python 2. Additionally, added documentation for client_encoding parameter that may be passed to libpq directly via psycopg2. Change-Id: I40ddf6382c157fa9399c21f0e01064197ea100f8
* | Merge "``Identity`` implies ``nullable=False``."mike bayer2021-01-174-21/+51
|\ \
| * | ``Identity`` implies ``nullable=False``.Federico Caselli2021-01-164-21/+51
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Altered the behavior of the :class:`_schema.Identity` construct such that when applied to a :class:`_schema.Column`, it will automatically imply that the value of :paramref:`_sql.Column.nullable` should default to ``False``, in a similar manner as when the :paramref:`_sql.Column.primary_key` parameter is set to ``True``. This matches the default behavior of all supporting databases where ``IDENTITY`` implies ``NOT NULL``. The PostgreSQL backend is the only one that supports adding ``NULL`` to an ``IDENTITY`` column, which is here supported by passing a ``True`` value for the :paramref:`_sql.Column.nullable` parameter at the same time. Fixes: #5775 Change-Id: I0516d506ff327cff35cda605e8897a27440e0373
* | | introduce generalized decorator to prevent invalid method callsMike Bayer2021-01-165-13/+72
|/ / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This introduces the ``_exclusive_against()`` utility decorator that can be used to prevent repeated invocations of methods that typically should only be called once. An informative error message is now raised for a selected set of DML methods (currently all part of :class:`_dml.Insert` constructs) if they are called a second time, which would implicitly cancel out the previous setting. The methods altered include: :class:`_sqlite.Insert.on_conflict_do_update`, :class:`_sqlite.Insert.on_conflict_do_nothing` (SQLite), :class:`_postgresql.Insert.on_conflict_do_update`, :class:`_postgresql.Insert.on_conflict_do_nothing` (PostgreSQL), :class:`_mysql.Insert.on_duplicate_key_update` (MySQL) Fixes: #5169 Change-Id: I9278fa87cd3470dcf296ff96bb0fb17a3236d49d
* | Merge "Create explicit GC ordering between ConnectionFairy/ConnectionRecord"mike bayer2021-01-163-9/+55
|\ \
| * | Create explicit GC ordering between ConnectionFairy/ConnectionRecordMike Bayer2021-01-154-17/+59
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixed issue where connection pool would not return connections to the pool or otherwise be finalized upon garbage collection under pypy if the checked out connection fell out of scope without being closed. This is a long standing issue due to pypy's difference in GC behavior that does not call weakref finalizers if they are relative to another object that is also being garbage collected. A strong reference to the related record is now maintained so that the weakref has a strong-referenced "base" to trigger off of. Fixes: #5842 Change-Id: Id5448fdacb6cceaac1ea40b2fbc851f052ed8e86
* | | Merge "Guard against re-entrant autobegin in Core, ORM"mike bayer2021-01-163-5/+12
|\ \ \
| * | | Guard against re-entrant autobegin in Core, ORMMike Bayer2021-01-153-5/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixed bug in "future" version of :class:`.Engine` where emitting SQL during the :meth:`.EngineEvents.do_begin` event hook would cause a re-entrant condition due to autobegin, including the recipe documented for SQLite to allow for savepoints and serializable isolation support. Fixed issue in new :class:`_orm.Session` similar to that of the :class:`_engine.Connection` where the new "autobegin" logic could be tripped into a re-entrant state if SQL were executed within the :meth:`.SessionEvents.after_transaction_create` event hook. Also repair the new "testing_engine" pytest fixture to set up for "future" engine appropriately, which wasn't working leading to the test_execute.py tests not using the future engine since recent f1e96cb0874927a475d0c11139. Fixes: #5845 Change-Id: Ib2432d8c8bd753e24be60720ec47affb2df15a4a
* | | | run handle error for commit/rollback fail and cancel transactionMike Bayer2021-01-152-6/+26
|/ / / | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixed bug in asyncpg dialect where a failure during a "commit" or less likely a "rollback" should cancel the entire transaction; it's no longer possible to emit rollback. Previously the connection would continue to await a rollback that could not succeed as asyncpg would reject it. Fixes: #5824 Change-Id: I5a4916740c269b410f4d1a78ed25191de344b9d0
* | | Merge "Add missing requirements to suite tests"mike bayer2021-01-151-2/+8
|\ \ \ | |_|/ |/| |
| * | Add missing requirements to suite testsGord Thompson2021-01-151-2/+8
| | | | | | | | | | | | | | | Fixes: #5841 Change-Id: I53e8405e6ca8c3fd6909744632b76db724d9eb1f
* | | Merge "update execute() arg formats in modules and tests"mike bayer2021-01-1510-28/+42
|\ \ \ | |/ / |/| |
| * | update execute() arg formats in modules and testsMike Bayer2021-01-1510-28/+42
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | continuing with producing a SQLAlchemy 1.4.0b2 that internally does not emit any of its own 2.0 deprecation warnings, migrate the *args and **kwargs passed to execute() methods that now must be a single list or dictionary. Alembic 1.5 is again waiting on this internal consistency to be present so that it can pass all tests with no 2.0 deprecation warnings. Change-Id: If6b792e57c8c5dff205419644ab68e631575a2fa
* | | Merge "Add mssql+turbodbc to external dialect list"mike bayer2021-01-151-0/+10
|\ \ \
| * | | Add mssql+turbodbc to external dialect listGord Thompson2021-01-131-0/+10
| | | | | | | | | | | | | | | | | | | | Fixes: #5834 Change-Id: I1f207b84751e7e3425aa9e8e393787eeb9b595b7
* | | | allow Executable to be accepted by Session.execute()Mike Bayer2021-01-142-2/+1
| |/ / |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixed an issue where the API to create a custom executable SQL construct using the ``sqlalchemy.ext.compiles`` extension according to the documentation that's been up for many years would no longer function if only ``Executable, ClauseElement`` were used as the base classes, additional classes were needed if wanting to use :meth:`_orm.Session.execute`. This has been resolved so that those extra classes aren't needed. Change-Id: I99b8acd88515c2a52842d62974199121e64c0381
* | | fix double is_none()Mike Bayer2021-01-141-8/+4
| |/ |/| | | | | | | | | | | | | I added an extra is_none() by mistake the other day and for some reason it sneaked past flake8. it's breaking all the builds so get it back in Change-Id: I17b311341169571efa856e062c6be7e8f362618f
* | Merge "Use UnsupportedCompilationError for no default compiler"mike bayer2021-01-142-9/+13
|\ \
| * | Use UnsupportedCompilationError for no default compilerMike Bayer2021-01-142-9/+13
| |/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixed issue where the stringification that is sometimes called when attempting to generate the "key" for the ``.c`` collection on a selectable would fail if the column were an unlabeled custom SQL construct using the ``sqlalchemy.ext.compiler`` extension, and did not provide a default compilation form; while this seems like an unusual case, it can get invoked for some ORM scenarios such as when the expression is used in an "order by" in combination with joined eager loading. The issue is that the lack of a default compiler function was raising :class:`.CompileError` and not :class:`.UnsupportedCompilationError`. Fixes: #5836 Change-Id: I5af243b2c70c7dcca4b212a3869c3017a50c132b
* | Merge "reinvent xdist hooks in terms of pytest fixtures"mike bayer2021-01-1426-342/+828
|\ \ | |/ |/|
| * reinvent xdist hooks in terms of pytest fixturesMike Bayer2021-01-1326-342/+829
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | To allow the "connection" pytest fixture and others work correctly in conjunction with setup/teardown that expects to be external to the transaction, remove and prevent any usage of "xdist" style names that are hardcoded by pytest to run inside of fixtures, even function level ones. Instead use pytest autouse fixtures to implement our own r"setup|teardown_test(?:_class)?" methods so that we can ensure function-scoped fixtures are run within them. A new more explicit flow is set up within plugin_base and pytestplugin such that the order of setup/teardown steps, which there are now many, is fully documented and controllable. New granularity has been added to the test teardown phase to distinguish between "end of the test" when lock-holding structures on connections should be released to allow for table drops, vs. "end of the test plus its teardown steps" when we can perform final cleanup on connections and run assertions that everything is closed out. From there we can remove most of the defensive "tear down everything" logic inside of engines which for many years would frequently dispose of pools over and over again, creating for a broken and expensive connection flow. A quick test shows that running test/sql/ against a single Postgresql engine with the new approach uses 75% fewer new connections, creating 42 new connections total, vs. 164 new connections total with the previous system. As part of this, the new fixtures metadata/connection/future_connection have been integrated such that they can be combined together effectively. The fixture_session(), provide_metadata() fixtures have been improved, including that fixture_session() now strongly references sessions which are explicitly torn down before table drops occur afer a test. Major changes have been made to the ConnectionKiller such that it now features different "scopes" for testing engines and will limit its cleanup to those testing engines corresponding to end of test, end of test class, or end of test session. The system by which it tracks DBAPI connections has been reworked, is ultimately somewhat similar to how it worked before but is organized more clearly along with the proxy-tracking logic. A "testing_engine" fixture is also added that works as a pytest fixture rather than a standalone function. The connection cleanup logic should now be very robust, as we now can use the same global connection pools for the whole suite without ever disposing them, while also running a query for PostgreSQL locks remaining after every test and assert there are no open transactions leaking between tests at all. Additional steps are added that also accommodate for asyncio connections not explicitly closed, as is the case for legacy sync-style tests as well as the async tests themselves. As always, hundreds of tests are further refined to use the new fixtures where problems with loose connections were identified, largely as a result of the new PostgreSQL assertions, many more tests have moved from legacy patterns into the newest. An unfortunate discovery during the creation of this system is that autouse fixtures (as well as if they are set up by @pytest.mark.usefixtures) are not usable at our current scale with pytest 4.6.11 running under Python 2. It's unclear if this is due to the older version of pytest or how it implements itself for Python 2, as well as if the issue is CPU slowness or just large memory use, but collecting the full span of tests takes over a minute for a single process when any autouse fixtures are in place and on CI the jobs just time out after ten minutes. So at the moment this patch also reinvents a small version of "autouse" fixtures when py2k is running, which skips generating the real fixture and instead uses two global pytest fixtures (which don't seem to impact performance) to invoke the "autouse" fixtures ourselves outside of pytest. This will limit our ability to do more with fixtures until we can remove py2k support. py.test is still observed to be much slower in collection in the 4.6.11 version compared to modern 6.2 versions, so add support for new TOX_POSTGRESQL_PY2K and TOX_MYSQL_PY2K environment variables that will run the suite for fewer backends under Python 2. For Python 3 pin pytest to modern 6.2 versions where performance for collection has been improved greatly. Includes the following improvements: Fixed bug in asyncio connection pool where ``asyncio.TimeoutError`` would be raised rather than :class:`.exc.TimeoutError`. Also repaired the :paramref:`_sa.create_engine.pool_timeout` parameter set to zero when using the async engine, which previously would ignore the timeout and block rather than timing out immediately as is the behavior with regular :class:`.QueuePool`. For asyncio the connection pool will now also not interact at all with an asyncio connection whose ConnectionFairy is being garbage collected; a warning that the connection was not properly closed is emitted and the connection is discarded. Within the test suite the ConnectionKiller is now maintaining strong references to all DBAPI connections and ensuring they are released when tests end, including those whose ConnectionFairy proxies are GCed. Identified cx_Oracle.stmtcachesize as a major factor in Oracle test scalability issues, this can be reset on a per-test basis rather than setting it to zero across the board. the addition of this flag has resolved the long-standing oracle "two task" error problem. For SQL Server, changed the temp table style used by the "suite" tests to be the double-pound-sign, i.e. global, variety, which is much easier to test generically. There are already reflection tests that are more finely tuned to both styles of temp table within the mssql test suite. Additionally, added an extra step to the "dropfirst" mechanism for SQL Server that will remove all foreign key constraints first as some issues were observed when using this flag when multiple schemas had not been torn down. Identified and fixed two subtle failure modes in the engine, when commit/rollback fails in a begin() context manager, the connection is explicitly closed, and when "initialize()" fails on the first new connection of a dialect, the transactional state on that connection is still rolled back. Fixes: #5826 Fixes: #5827 Change-Id: Ib1d05cb8c7cf84f9a4bfd23df397dc23c9329bfe
* | Improve ``run_sync`` documentation in asyncio extFederico Caselli2021-01-072-0/+20
| | | | | | | | | | Change-Id: I202458ab6d81e29053118c9fb9c205b865c8d2ba Ref: #5817
* | Implement connection binding for AsyncSessionMike Bayer2021-01-075-6/+131
|/ | | | | | | | | | | | | | | | | | | | | | | | | | | Implemented "connection-binding" for :class:`.AsyncSession`, the ability to pass an :class:`.AsyncConnection` to create an :class:`.AsyncSession`. Previously, this use case was not implemented and would use the associated engine when the connection were passed. This fixes the issue where the "join a session to an external transaction" use case would not work correctly for the :class:`.AsyncSession`. Additionally, added methods :meth:`.AsyncConnection.in_transaction`, :meth:`.AsyncConnection.in_nested_transaction`, :meth:`.AsyncConnection.get_transaction`. The :class:`.AsyncEngine`, :class:`.AsyncConnection` and :class:`.AsyncTransaction` objects may be compared using Python ``==`` or ``!=``, which will compare the two given objects based on the "sync" object they are proxying towards. This is useful as there are cases particularly for :class:`.AsyncTransaction` where multiple instances of :class:`.AsyncTransaction` can be proxying towards the same sync :class:`_engine.Transaction`, and are actually equivalent. The :meth:`.AsyncConnection.get_transaction` method will currently return a new proxying :class:`.AsyncTransaction` each time as the :class:`.AsyncTransaction` is not otherwise statefully associated with its originating :class:`.AsyncConnection`. Fixes: #5811 Change-Id: I5a3a6b2f088541eee7b0e0f393510e61bc9f986b
* Merge "Remove special rule for TypeDecorator of TypeDecorator"mike bayer2021-01-071-5/+1
|\
| * Remove special rule for TypeDecorator of TypeDecoratorMike Bayer2021-01-051-5/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Removing this check for "TypeDecorator" in impl seems to not break anything and allows TypeDecorator.with_variant() to work correctly. The line has been traced back to 2007 and does not appear to have relevance today. Fixed bug where making use of the :meth:`.TypeEngine.with_variant` method on a :class:`.TypeDecorator` type would fail to take into account the dialect-specific mappings in use, due to a rule in :class:`.TypeDecorator` that was instead attempting to check for chains of :class:`.TypeDecorator` instances. Fixes: #5816 Change-Id: Ic86d9d985810e3050f15972b4841108acca2fa3e
* | Update connect args for pymysql 1.0.0; aiomysql fixesMike Bayer2021-01-073-8/+17
| | | | | | | | | | | | | | | | | | | | | | | | | | Fixed deprecation warnings that arose as a result of the release of PyMySQL 1.0, including deprecation warnings for the "db" and "passwd" parameters now replaced with "database" and "password". For the 1.4 version of this patch, we are also changing tox.ini to refer to a local branch of aiomysql that fixes pymysql compatibility issues. Fixes: #5821 Change-Id: I93876b52b2d96b52308f22aeb4f244ac5766a82f
* | Merge "remove more bound metadata"mike bayer2021-01-064-13/+19
|\ \
| * | remove more bound metadataMike Bayer2021-01-054-13/+19
| |/ | | | | | | | | | | | | | | | | | | | | | | | | | | in Iae6ab95938a7e92b6d42086aec534af27b5577d3 I missed that the "bind" was being stuck onto the MetaData in TablesTest, which led thousands of ORM tests to still use bound metadata. Keep looking for bound metadata. standardize all ORM tests on a single means of getting a Session when the Session API isn't the thing we are directly testing, using a new function fixture_session() that replaces create_session() and uses modern defaults. Change-Id: Iaf71206e9ee568151496d8bc213a069504bf65ef
* | Check for column expr in Oracle RETURNING checkMike Bayer2021-01-041-0/+1
|/ | | | | | | | | | Fixed regression in Oracle dialect introduced by :ticket:`4894` in SQLAlchemy 1.3.11 where use of a SQL expression in RETURNING for an UPDATE would fail to compile, due to a check for "server_default" when an arbitrary SQL expression is not a column. Fixes: #5813 Change-Id: I1977bb49bc971399195015ae45e761f774f4008d
* happy new yearMike Bayer2021-01-04188-190/+190
| | | | Change-Id: Ic5bb19ca8be3cb47c95a0d3315d84cb484bac47c
* remove metadata.bind use from test suiteMike Bayer2021-01-0311-613/+637
| | | | | | | | | | | | | | importantly this means we can remove bound metadata from the fixtures that are used by Alembic's test suite. hopefully this is the last one that has to happen to allow Alembic to be fully 1.4/2.0. Start moving from @testing.provide_metadata to a pytest metadata fixture. This does not seem to have any negative effects even though TablesTest uses a "self.metadata" attribute. Change-Id: Iae6ab95938a7e92b6d42086aec534af27b5577d3
* Cache asyngpg prepared statementsMike Bayer2021-01-021-21/+157
| | | | | | | | | | | | | | | | | Enhanced the performance of the asyncpg dialect by caching the asyncpg PreparedStatement objects on a per-connection basis. For a test case that makes use of the same statement on a set of pooled connections this appears to grant a 10-20% speed improvement. The cache size is adjustable and may also be disabled. Unfortunately the caching gets more complicated when there are schema changes present. An invalidation scheme is now also added to accommodate for prepared statements as well as asyncpg cached OIDs. However, the exception raises cannot be prevented if DDL has changed database structures that were cached for a particular asyncpg connection. Logic is added to clear the caches when these errors occur. Change-Id: Icf02aa4871eb192f245690f28be4e9f9c35656c6
* Merge "Repair async test refactor"mike bayer2021-01-037-18/+46
|\
| * Repair async test refactorMike Bayer2021-01-027-18/+46
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | in I4940d184a4dc790782fcddfb9873af3cca844398 we reworked how async tests run but apparently the async tests in test/ext/asyncio are reporting success without being run. This patch pushes pytestplugin further so that it won't instrument any test or function overall that declares itself async. This removes the need for the __async_wrap__ flag and also allows us to use a more strict "run_async_test" function that always runs the asyncio event loop from the top. Also start working asyncio into main testing suite. Change-Id: If7144e951a9db67eb7ea73b377f81c4440d39819
* | Merge "Further attempts to repair SQL server temp table issue"mike bayer2021-01-022-7/+14
|\ \
| * | Further attempts to repair SQL server temp table issueMike Bayer2021-01-022-7/+14
| |/ | | | | | | | | | | | | Still having non-reproducible failures where "user_tmp" cannot be dropped. try isolating the table name around config.ident Change-Id: I17e0a9674b22d246f0d52943b850e8f6de223305
View Lorry Controller Status