summaryrefslogtreecommitdiff
path: root/lib/sqlalchemy/orm/session.py
Commit message (Collapse)AuthorAgeFilesLines
* Dont return outer transaction for _subtrans flagMike Bayer2021-04-091-3/+11
| | | | | | | | | | | | | | | | | | | Fixed critical regression where the :class:`_orm.Session` could fail to "autobegin" a new transaction when a flush occurred without an existing transaction in place, implicitly placing the :class:`_orm.Session` into legacy autocommit mode which commit the transaction. The :class:`_orm.Session` now has a check that will prevent this condition from occurring, in addition to repairing the flush issue. Additionally, scaled back part of the change made as part of :ticket:`5226` which can run autoflush during an unexpire operation, to not actually do this in the case of a :class:`_orm.Session` using legacy :paramref:`_orm.Session.autocommit` mode, as this incurs a commit within a refresh operation. Fixes: #6233 Change-Id: Ia980e62a090e39e3e2a7fb77c95832ae784cc9a5
* Disallow AliasedReturnsRows from executionMike Bayer2021-04-051-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | Executing a :class:`_sql.Subquery` using :meth:`_engine.Connection.execute` is deprecated and will emit a deprecation warning; this use case was an oversight that should have been removed from 1.4. The operation will now execute the underlying :class:`_sql.Select` object directly for backwards compatibility. Similarly, the :class:`_sql.CTE` class is also not appropriate for execution. In 1.3, attempting to execute a CTE would result in an invalid "blank" SQL statement being executed; since this use case was not working it now raises :class:`_exc.ObjectNotExecutableError`. Previously, 1.4 was attempting to execute the CTE as a statement however it was working only erratically. The change also breaks out StatementRole from ReturnsRowsRole, as these roles should not be in the same lineage (some statements don't return rows, the whole class of ReturnsRows that are from clauses are not statements). Consolidate StatementRole and CoerceTextStatementRole as there's no usage difference between these. Simplify some old tests that were trying to make sure that "execution options" didn't transmit from a cte/subquery out to a select; as cte/subuqery() aren't executable in any case the options are removed. Fixes: #6204 Change-Id: I62613b7ab418afdd22c409eae75659e3f52fb65f
* fix __all__Mike Bayer2021-03-201-1/+10
| | | | | | | sqlalchemy.engine had an oddly restrictive __all__ for some reason. Add missing symbols to session.__all__ Change-Id: I017fa1c2a93f559f2ccc366f88660266c50e9ca6
* Fix typo in Session.identity_keyMike Bayer2021-03-161-1/+1
| | | | | | | | | Fixed regression in :meth:`_orm.Session.identity_key`, including that the method and related methods were not covered by any unit test as well as that the method contained a typo preventing it from functioning correctly. Fixes: #6067 Change-Id: I1a84f9ed095c4226d57eef1c46996601dc2f1eaa
* orm/session.py - comment grammar fixMaciej Brencz2021-02-041-1/+1
|
* Merge "Fix many spell glitches in docstrings and comments"mike bayer2021-01-261-2/+2
|\
| * Fix many spell glitches in docstrings and commentsLele Gaifax2021-01-241-2/+2
| | | | | | | | | | | | | | | | | | | | These were revealed by running `pylint --disable all --enable spelling --spelling-dict en_US` over all sources. Closes: #5868 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5868 Pull-request-sha: bb249195d92e3b806e81ecf1192d5a1b3cd5db48 Change-Id: I96080ec93a9fbd20ce21e9e16265b3c77f22bb14
* | Replace with_labels() and apply_labels() in ORM/CoreGord Thompson2021-01-261-4/+7
|/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Replace :meth:`_orm.Query.with_labels` and :meth:`_sql.GenerativeSelect.apply_labels` with explicit getters and setters ``get_label_style`` and ``set_label_style`` to accommodate the three supported label styles: ``LABEL_STYLE_DISAMBIGUATE_ONLY`` (default), ``LABEL_STYLE_TABLENAME_PLUS_COL``, and ``LABEL_STYLE_NONE``. In addition, for Core and "future style" ORM queries, ``LABEL_STYLE_DISAMBIGUATE_ONLY`` is now the default label style. This style differs from the existing "no labels" style in that labeling is applied in the case of column name conflicts; with ``LABEL_STYLE_NONE``, a duplicate column name is not accessible via name in any case. For legacy ORM queries using :class:`_query.Query`, the table-plus-column names labeling style applied by ``LABEL_STYLE_TABLENAME_PLUS_COL`` continues to be used so that existing test suites and logging facilities see no change in behavior by default, however this style of labeling is no longer required for SQLAlchemy queries to function, as result sets are commonly matched to columns using a positional approach since SQLAlchemy 1.0. Within test suites, all use of apply_labels() / use_labels now uses the new methods. New tests added to test/sql/test_deprecations.py nad test/orm/test_deprecations.py to cover just the old apply_labels() method call. Tests in ORM that made explicit use apply_labels()/ etc. where it isn't needed for the ORM to work correctly use default label style now. Co-authored-by: Mike Bayer <mike_mp@zzzcomputing.com> Fixes: #4757 Change-Id: I5fdcd2ed4ae8c7fe62f8be2b6d0e8f66409b6a54
* Guard against re-entrant autobegin in Core, ORMMike Bayer2021-01-151-3/+9
| | | | | | | | | | | | | | | | | | | | Fixed bug in "future" version of :class:`.Engine` where emitting SQL during the :meth:`.EngineEvents.do_begin` event hook would cause a re-entrant condition due to autobegin, including the recipe documented for SQLite to allow for savepoints and serializable isolation support. Fixed issue in new :class:`_orm.Session` similar to that of the :class:`_engine.Connection` where the new "autobegin" logic could be tripped into a re-entrant state if SQL were executed within the :meth:`.SessionEvents.after_transaction_create` event hook. Also repair the new "testing_engine" pytest fixture to set up for "future" engine appropriately, which wasn't working leading to the test_execute.py tests not using the future engine since recent f1e96cb0874927a475d0c11139. Fixes: #5845 Change-Id: Ib2432d8c8bd753e24be60720ec47affb2df15a4a
* happy new yearMike Bayer2021-01-041-1/+1
| | | | Change-Id: Ic5bb19ca8be3cb47c95a0d3315d84cb484bac47c
* implement sessionmaker.begin(), scalar() for async sessionMike Bayer2020-12-261-4/+9
| | | | | | | | | | | Added :meth:`_asyncio.AsyncSession.scalar` as well as support for :meth:`_orm.sessionmaker.begin` to work as an async context manager with :class:`_asyncio.AsyncSession`. Also added :meth:`_asyncio.AsyncSession.in_transaction` accessor. Fixes: #5796 Fixes: #5797 Change-Id: Id3d431421df0f8c38f356469a50a946ba9c38513
* Add ORMExecuteState mapper accessorsMike Bayer2020-12-231-0/+65
| | | | | | | | | | Added :attr:`_orm.ORMExecuteState.bind_mapper` and :attr:`_orm.ORMExecuteState.all_mappers` accessors to :class:`_orm.ORMExecuteState` event object, so that handlers can respond to the target mapper and/or mapped class or classes involved in an ORM statement execution. Change-Id: I2cfe3d422ce5df2559105d53a51135a583359bd9
* Fixes for lambda expressions and relationship loadersMike Bayer2020-12-111-1/+1
| | | | | | | | | | | | | | | | | | Fixed bug in lambda SQL feature, used by ORM :meth:`_orm.with_loader_criteria` as well as available generally in the SQL expression language, where assigning a boolean value True/False to a variable would cause the query-time expression calculation to fail, as it would produce a SQL expression not compatible with a bound value. Fixed issue where the :attr:`_orm.ORMExecuteState.is_relationship_load` parameter would not be set correctly for many lazy loads, all selectinloads, etc. The flag is essential in order to test if options should be added to statements or if they would already have been propagated via relationship loads. Fixes: #5763 Fixes: #5764 Change-Id: I66aafbef193f892ff75ede0670698647b7475482
* Revise attribute refresh for with_loader_criteria, relatedMike Bayer2020-12-111-0/+41
| | | | | | | | | | | | | | | | | | | | | | | | | | Added new attribute :attr:`_orm.ORMExecuteState.is_column_load` to indicate that a :meth:`_orm.SessionEvents.do_orm_execute` handler that a particular operation is a primary-key-directed column attribute load, such as from an expiration or a deferred attribute, and that WHERE criteria or additional loader options should not be added to the query. This has been added to the examples which illustrate the :func:`_orm.with_loader_criteria` option. The :func:`_orm.with_loader_criteria` option has been modified so that it will never apply its criteria to the SELECT statement for an ORM refresh operation, such as that invoked by :meth:`_orm.Session.refresh` or whenever an expired attribute is loaded. These queries are only against the primary key row of the object that is already present in memory so there should not be additional criteria added. Added doc caveats for using lambdas. Added test coverage for most ORMExecuteState flags and fixed a few basic access issues. Change-Id: I6707e4cf0dc95cdfb8ce93e5ca22ead86074baa7 References: #5760 Fixes: #5761 Fixes: #5762
* Typo in sessionmaker example: s/wih/withBrendan McCollam2020-11-261-1/+1
|
* Don't emit warnings on descriptor accessMike Bayer2020-11-201-0/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This commit is revising 5162f2bc5fc0ac239f26a76fc9f0c2, which when I did it felt a little rushed but I couldn't find anything wrong. Well here we are :). Fixed issue where a :class:`.RemovedIn20Warning` would erroneously emit when the ``.bind`` attribute were accessed internally on objects, particularly when stringifying a SQL construct. Alter the deprecated() decorator so that we can use it just to add docstring warnings but not actually warn when the function is accessed, adding new argument enable_warnings that can be set to False. Added a safety feature to deprecated_20() that will disallow an ":attr:" from proceeding if enable_warnings=False isn't present, unless there's an extra flag warn_on_attribute_access, since we want Session.transaction to emit a deprecation warning. This is a little hacky but it's essentially modifying the decorator to require a positive assertion that a deprecation decorator on a descriptor should actually warn on access. Remove the warning filter for session.transaction and get tests to pass to ensure this is not also being called internally. Added tests to ensure that common places .bind can be passed as a parameter definitely warn as I was not able to find this otherwise. Fixes: #5717 Change-Id: Ia586b4f9ee6b212f3a71104b1caf40b5edd399e2
* tutorial 2.0 WIPreview/mike_bayer/tutorial20Mike Bayer2020-10-311-16/+5
| | | | | | | | | | | | | | Add SelectBase.exists() method as it seems strange this is not available already. The Exists construct itself does not provide full SELECT-building capabilities so it makes sense this should be used more like a scalar_subquery. Make sure stream_results is getting set up when yield_per is used, for 2.0 style statements as well. this was hardcoded inside of Query.yield_per() and is now moved to take place within QueryContext. Change-Id: Icafcd4fd9b708772343d56edf40995c9e8f835d6
* Clarify bulk operations caveatsMike Bayer2020-10-291-9/+12
| | | | | Fixes: #5678 Change-Id: I186b26d1265c106bb87fb9e680229cad1eb8d7ea
* added docstring about expire_on_commit for #5243jonathan vanasco2020-10-261-1/+8
| | | | | | | | | Fixes: #5243 Closes: #5328 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5328 Pull-request-sha: e760ed4ef1749aadfb8d63d776c764d98b93b1e5 Change-Id: I8e7bc3429dc279d447cc66400c76b4d2a04626aa
* generalize scoped_session proxying and apply to asyncio elementsMike Bayer2020-10-101-30/+8
| | | | | | | | | | | | | | | | | | | | Reworked the proxy creation used by scoped_session() to be based on fully copied code with augmented docstrings and moved it into langhelpers. asyncio session, engine, connection can now take advantage of it so that all non-async methods are availble. Overall implementation of most important accessors / methods on AsyncConnection, etc. , including awaitable versions of invalidate, execution_options, etc. In order to support an event dispatcher on the async classes while still allowing them to hold __slots__, make some adjustments to the event system to allow that to be present, at least rudimentally. Fixes: #5628 Change-Id: I5eb6929fc1e4fdac99e4b767dcfd49672d56e2b2
* upgrade to black 20.8b1Mike Bayer2020-09-281-12/+28
| | | | | | | It's better, the majority of these changes look more readable to me. also found some docstrings that had formatting / quoting issues. Change-Id: I582a45fde3a5648b2f36bab96bad56881321899b
* Repair erroneous "future" symbolMike Bayer2020-09-271-7/+2
| | | | | | | | | | | the change in 1e800285508ecd869c6874fed failed to fully remove the "future" symbol which then got confused against the import of the "future" package itself, which is also not needed. remove it entirely. pin pytest < 6.1 to see if new error condition may be avoided Change-Id: Ibe45040120d3772c39faef4a61ed26f90b60c79e
* new docs WIPMike Bayer2020-09-241-0/+12
| | | | | | | | | | | | | | | This WIP is part of the final push for 1.4's docs to fully "2.0-ize" what we can, and have it all ready. So far this includes a rewrite of the 2.0 migration, set up for the 1.4 /2.0 docs style, and a total redesign of the index page using a new flex layout in zzzeeksphinx. It also reworks some of the API reference sections to have more subheaders. zzzeeksphinx is also enhanced to provide automatic summaries for all api doc section. Change-Id: I01d360cb9c8749520246b96ee6496143c6037918
* Update session.execute() and related documentationMike Bayer2020-09-131-91/+27
| | | | | | | | | | The docs here were completely out of date and referred to behaviors that are no longer true, behaviors that are deprecated, etc. For the moment, take out all the verbiage so that nothing incorrect is present. New ORM documentation will need to be constructed to support this statement. Change-Id: I4782aebb6443ceb68752c3b52b574fd30658ebc9
* Emit deprecation warning for **kw passed to session.execute()Mike Bayer2020-09-111-8/+19
| | | | | | | | | | Passing keyword arguments to methods such as :meth:`_orm.Session.execute` to be passed into the :meth:`_orm.Session.get_bind` method is deprecated; the new :paramref:`_orm.Session.execute.bind_arguments` dictionary should be passed instead. Fixes: #5573 Change-Id: I555bda84384dbf6d12ba4483c486f9488be0fa25
* Add more docs for populate_existing(); link with_for_updateMike Bayer2020-09-101-0/+6
| | | | | | | | | | | | | The populate_existing() method is actually changing to be an execution option, however it has almost no mention in the narrative docs so add docs in terms of the 1.x version first, including that we mention you almost definitely want to use this method if you are also using with_for_update(). Fixes: #5572 Fixes: #4774 Change-Id: Ieca916400622c1ffc1ae81204132a02a0983594c
* Update connection docs for migrating off of nestingMike Bayer2020-08-291-4/+4
| | | | Change-Id: I3a81140f00a4a9945121bfb8ec4c0e3953b4085f
* See if the future is hereMike Bayer2020-08-281-20/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | The docs are going to talk a lot about session.execute(select()) for ORM queries, and additionally it's much easier to help users with queries and such if we can use this new syntax. I'm hoping to see how hard it is to get a unified tutorial started that switches to new syntax. Basically, new syntax is much easier to explain and less buggy. But, if we are starting to present new syntax with the explicit goal of being easier to explain for less experienced programmers, the "future" thing is going to just be an impediment to that. See if we can remove "future" from session.execute(), so that ORM-enabled select() statements return ORM results at that level. This does not change the presence of the "future" flag for the Session's construction and for its transactional behaviors. The only perceptible change of the future flag for session.execute() is that session.execute(select()) where the statement has ORM entities in it now returns ORM new style tuples rather than old style tuples. Like mutating a URL, it's hopefully not very common that people are doing this. Change-Id: I0aa10322bb787d554d32772e3bc60548f1bf6206
* Documentation updates for 1.4Mike Bayer2020-08-051-15/+46
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * major additions to 1.4 migration doc; removed additional verbosity regarding caching methodology and reorganized the doc to present itself more as a "what's changed" guide * as we now have a path for asyncio, update that doc so that we aren't spreading obsolete information * updates to the 2.0 migration guide with latest info, however this is still an architecture doc and not a migration guide yet, will need further rework. * start really talking about 1.x vs. 2.0 style everywhere. Querying is most of the docs so this is going to be a prominent theme, start getting it to fit in * Add introductory documentation for ORM example sections as these are too sparse * new documentation for do_orm_execute(), many separate sections, adding deprecation notes to before_compile() and similar * new example suites to illustrate do_orm_execute(), with_loader_criteria() * modernized horizontal sharding examples and added a separate example to distinguish between multiple databases and single database w/ multiple tables use case * introducing DEEP ALCHEMY, will use zzzeeksphinx 1.1.6 * no name for the alchemist yet however the dragon's name is Flambé Change-Id: Id6b5c03b1ce9ddb7b280f66792212a0ef0a1c541
* Implement relationship AND criteria; global loader criteriaMike Bayer2020-08-051-55/+46
| | | | | | | | | | | | | | | | | | | Added the ability to add arbitrary criteria to the ON clause generated by a relationship attribute in a query, which applies to methods such as :meth:`_query.Query.join` as well as loader options like :func:`_orm.joinedload`. Additionally, a "global" version of the option allows limiting criteria to be applied to particular entities in a query globally. Documentation is minimal at this point, new examples will be coming in a subsequent commit. Some adjustments to execution options in how they are represented in the ORMExecuteState as well as well as a few ORM tests that forgot to get merged in a preceding commit. Fixes: #4472 Change-Id: I2b8fc57092dedf35ebd16f6343ad0f0d7d332beb
* Convert lazy loader, selectinload, load_on_ident to lambda statementsMike Bayer2020-08-051-5/+14
| | | | | | | | | Building on newly robust lambdas in I29a513c98917b1d503abfdd61e6b6e8800851aa8, convert key loading off of the "baked" system so that baked is no longer used by the ORM. Change-Id: I3abfb45dd6e50f84f29d39434caa0b550ce27864
* Establish future behavior for Session cascade backrefs, bindMike Bayer2020-08-011-4/+23
| | | | | | | | | | | | | | | | | | | | The behavior of the :paramref:`_orm.relationship.cascade_backrefs` flag will be reversed in 2.0 and set to ``False`` unconditionally, such that backrefs don't cascade save-update operations from a forwards-assignment to a backwards assignment. A 2.0 deprecation warning is emitted when the parameter is left at its default of ``True`` at the point at which such a cascade operation actually takes place. The new behavior can be established as always by setting the flag to ``False`` on a specific :func:`_orm.relationship`, or more generally can be set up across the board by setting the the :paramref:`_orm.Session.future` flag to True. Additionally in the interests of expediency, this commit will also move Session away from making use of bound metadata if the future=True flag is set. An application that sets future=True should ideally have to change as little else as possible for full 2.0 behavior. Fixes: #5150 Change-Id: I490d1d61f09c62ffc2de983208aeed25dfe48aec
* Further corrections to session docsMike Bayer2020-07-111-79/+54
| | | | | | | | | | | many more to come. One issue is that references to documentation sections inside of decorator-configured deprecation warnings will also generate a meaningless token when the runtime warning is emitted. It would be nice to improve upon this somehow. Change-Id: I16b214b3d310850bbfb0d9ade70235f5a9735eba
* Convert remaining ORM APIs to support 2.0 styleMike Bayer2020-07-111-237/+494
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is kind of a mixed bag of all kinds to help get us to 1.4 betas. The documentation stuff is a work in progress. Lots of other relatively small changes to APIs and things. More commits will follow to continue improving the documentation and transitioning to the 1.4/2.0 hybrid documentation. In particular some refinements to Session usage models so that it can match Engine's scoping / transactional patterns, and a decision to start moving away from "subtransactions" completely. * add select().from_statement() to produce FromStatement in an ORM context * begin referring to select() that has "plugins" for the few edge cases where select() will have ORM-only behaviors * convert dynamic.AppenderQuery to its own object that can use select(), though at the moment it uses Query to support legacy join calling forms. * custom query classes for AppenderQuery are replaced by do_orm_execute() hooks for custom actions, a separate gerrit will document this * add Session.get() to replace query.get() * Deprecate session.begin->subtransaction. propose within the test suite a hypothetical recipe for apps that rely on this pattern * introduce Session construction level context manager, sessionmaker context manager, rewrite the whole top of the session_transaction.rst documentation. Establish context manager patterns for Session that are identical to engine * ensure same begin_nested() / commit() behavior as engine * devise all new "join into an external transaction" recipe, add test support for it, add rules into Session so it just works, write new docs. need to ensure this doesn't break anything * vastly reduce the verbosity of lots of session docs as I dont think people read this stuff and it's difficult to keep current in any case * constructs like case(), with_only_columns() really need to move to *columns, add a coercion rule to just change these. * docs need changes everywhere I look. in_() is not in the Core tutorial? how do people even know about it? Remove tons of cruft from Select docs, etc. * build a system for common ORM options like populate_existing and autoflush to populate from execution options. * others? Change-Id: Ia4bea0f804250e54d90b3884cf8aab8b66b82ecf
* Add future=True to create_engine/Session; unify select()Mike Bayer2020-07-081-2/+41
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Several weeks of using the future_select() construct has led to the proposal there be just one select() construct again which features the new join() method, and otherwise accepts both the 1.x and 2.x argument styles. This would make migration simpler and reduce confusion. However, confusion may be increased by the fact that select().join() is different Current thinking is we may be better off with a few hard behavioral changes to old and relatively unknown APIs rather than trying to play both sides within two extremely similar but subtly different APIs. At the moment, the .join() thing seems to be the only behavioral change that occurs without the user taking any explicit steps. Session.execute() will still behave the old way as we are adding a future flag. This change also adds the "future" flag to Session() and session.execute(), so that interpretation of the incoming statement, as well as that the new style result is returned, does not occur for existing applications unless they add the use of this flag. The change in general is moving the "removed in 2.0" system further along where we want the test suite to fully pass even if the SQLALCHEMY_WARN_20 flag is set. Get many tests to pass when SQLALCHEMY_WARN_20 is set; this should be ongoing after this patch merges. Improve the RemovedIn20 warning; these are all deprecated "since" 1.4, so ensure that's what the messages read. Make sure the inforamtion link is on all warnings. Add deprecation warnings for parameters present and add warnings to all FromClause.select() types of methods. Fixes: #5379 Fixes: #5284 Change-Id: I765a0b912b3dcd0e995426427d8bb7997cbffd51 References: #5159
* Fix a wide variety of typos and broken linksaplatkouski2020-06-251-14/+14
| | | | | | | | | | | | Note the PR has a few remaining doc linking issues listed in the comment that must be addressed separately. Signed-off-by: aplatkouski <5857672+aplatkouski@users.noreply.github.com> Closes: #5371 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5371 Pull-request-sha: 7e7d233cf3a0c66980c27db0fcdb3c7d93bc2510 Change-Id: I9c36e8d8804483950db4b42c38ee456e384c59e3
* Propose using RETURNING for bulk updates, deletesMike Bayer2020-06-231-20/+52
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch makes several improvements in the area of bulk updates and deletes as well as the new session mechanics. RETURNING is now used for an UPDATE or DELETE statement emitted for a diaelct that supports "full returning" in order to satisfy the "fetch" strategy; this currently includes PostgreSQL and SQL Server. The Oracle dialect does not support RETURNING for more than one row, so a new dialect capability "full_returning" is added in addition to the existing "implicit_returning", indicating this dialect supports RETURNING for zero or more rows, not just a single identity row. The "fetch" strategy will gracefully degrade to the previous SELECT mechanics for dialects that do not support RETURNING. Additionally, the "fetch" strategy will attempt to use evaluation for the VALUES that were UPDATEd, rather than just expiring the updated attributes. Values should be evalutable in all cases where the value is not a SQL expression. The new approach also incurs some changes in the session.execute mechanics, where do_orm_execute() event handlers can now be chained to each return results; this is in turn used by the handler to detect on a per-bind basis if the fetch strategy needs to do a SELECT or if it can do RETURNING. A test suite is added to test_horizontal_shard that breaks up a single UPDATE or DELETE operation among multiple backends where some are SQLite and don't support RETURNING and others are PostgreSQL and do. The session event mechanics are corrected in terms of the "orm pre execute" hook, which now receives a flag "is_reentrant" so that the two ORM implementations for this can skip on their work if they are being called inside of ORMExecuteState.invoke(), where previously bulk update/delete were calling its SELECT a second time. In order for "fetch" to get the correct identity when called as pre-execute, it also requests the identity_token for each mapped instance which is now added as an optional capability of a SELECT for ORM columns. the identity_token that's placed by horizontal_sharding is now made available within each result row, so that even when fetching a merged result of plain rows we can tell which row belongs to which identity token. The evaluator that takes place within the ORM bulk update and delete for synchronize_session="evaluate" now supports the IN and NOT IN operators. Tuple IN is also supported. Fixes: #1653 Change-Id: I2292b56ae004b997cef0ba4d3fc350ae1dd5efc1
* Convert bulk update/delete to new execution modelMike Bayer2020-06-061-24/+90
| | | | | | | | | | | | | | | This reorganizes the BulkUD model in sqlalchemy.orm.persistence to be based on the CompileState concept and to allow plain update() / delete() to be passed to session.execute() where the ORM synchronize session logic will take place. Also gets "synchronize_session='fetch'" working with horizontal sharding. Adding a few more result.scalar_one() types of methods as scalar_one() seems like what is normally desired. Fixes: #5160 Change-Id: I8001ebdad089da34119eb459709731ba6c0ba975
* Inline a few ORM arguments, othersMike Bayer2020-06-031-21/+25
| | | | | | small changes Change-Id: Id89a0651196c431d0aaf6935f5a4e7b12dd70c6c
* callcount reductions and refinement for cached queriesMike Bayer2020-05-281-29/+77
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This commit includes that we've removed the "_orm_query" attribute from compile state as well as query context. The attribute created reference cycles and also added method call overhead. As part of this change, the interface for ORMExecuteState changes a bit, as well as the interface for the horizontal sharding extension which now deprecates the "query_chooser" callable in favor of "execute_chooser", which receives the contextual object. This will also work more nicely when we implement the new execution path for bulk updates and deletes. Pre-merge execution options for statement, connection, arguments all up front in Connection. that way they can be passed to the before_execute / after_execute events, and the ExecutionContext doesn't have to merge as second time. Core execute is pretty close to 1.3 now. baked wasn't using the new one()/first()/one_or_none() methods, fixed that. Convert non-buffered cursor strategy to be a stateless singleton. inline all the paths by which the strategy gets chosen, oracle and SQL Server dialects make use of the already-invoked post_exec() hook to establish the alternate strategies, and this is actually much nicer than it was before. Add caching to mapper instance processor for getters. Identified a reference cycle per query that was showing up as a lot of gc cleanup, fixed that. After all that, performance not budging much. Even test_baked_query now runs with significantly fewer function calls than 1.3, still 40% slower. Basically something about the new patterns just makes this slower and while I've walked a whole bunch of them back, it hardly makes a dent. that said, the performance issues are relatively small, in the 20-40% time increase range, and the new caching feature does provide for regular ORM and Core queries that are cached, and they are faster than non-cached. Change-Id: I7b0b0d8ca550c05f79e82f75cd8eff0bbfade053
* Convert execution to move through SessionMike Bayer2020-05-251-51/+268
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch replaces the ORM execution flow with a single pathway through Session.execute() for all queries, including Core and ORM. Currently included is full support for ORM Query, Query.from_statement(), select(), as well as the baked query and horizontal shard systems. Initial changes have also been made to the dogpile caching example, which like baked query makes use of a new ORM-specific execution hook that replaces the use of both QueryEvents.before_compile() as well as Query._execute_and_instances() as the central ORM interception hooks. select() and Query() constructs alike can be passed to Session.execute() where they will return ORM results in a Results object. This API is currently used internally by Query. Full support for Session.execute()->results to behave in a fully 2.0 fashion will be in later changesets. bulk update/delete with ORM support will also be delivered via the update() and delete() constructs, however these have not yet been adapted to the new system and may follow in a subsequent update. Performance is also beginning to lag as of this commit and some previous ones. It is hoped that a few central functions such as the coercions functions can be rewritten in C to re-gain performance. Additionally, query caching is now available and some subsequent patches will attempt to cache more of the per-execution work from the ORM layer, e.g. column getters and adapters. This patch also contains initial "turn on" of the caching system enginewide via the query_cache_size parameter to create_engine(). Still defaulting at zero for "no caching". The caching system still needs adjustments in order to gain adequate performance. Change-Id: I047a7ebb26aa85dc01f6789fac2bff561dcd555d
* Unify Query and select() , move all processing to compile phaseMike Bayer2020-05-241-3/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | Convert Query to do virtually all compile state computation in the _compile_context() phase, and organize it all such that a plain select() construct may also be used as the source of information in order to generate ORM query state. This makes it such that Query is not needed except for its additional methods like from_self() which are all to be deprecated. The construction of ORM state will occur beyond the caching boundary when the new execution model is integrated. future select() gains a working join() and filter_by() method. as we continue to rebase and merge each commit in the steps, callcounts continue to bump around. will have to look at the final result when it's all in. References: #5159 References: #4705 References: #4639 References: #4871 References: #5010 Change-Id: I19e05b3424b07114cce6c439b05198ac47f7ac10
* Update transaction / connection handlingMike Bayer2020-05-171-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | step one, do away with __connection attribute and using awkward AttributeError logic step two, move all management of "connection._transaction" into the transaction objects themselves where it's easier to follow. build MarkerTransaction that takes the role of "do-nothing block" new connection datamodel is: connection._transaction, always a root, connection._nested_transaction, always a nested. nested transactions still chain to each other as this is still sort of necessary but they consider the root transaction separately, and the marker transactions not at all. introduce new InvalidRequestError subclass PendingRollbackError. Apply to connection and session for all cases where a transaction needs to be rolled back before continuing. Within Connection, both PendingRollbackError as well as ResourceClosedError are now raised directly without being handled by handle_dbapi_error(); this removes these two exception cases from the handle_error event handler as well as from StatementError wrapping, as these two exceptions are not statement oriented and are instead programmatic issues, that the application is failing to handle database errors properly. Revise savepoints so that when a release fails, they set themselves as inactive so that their rollback() method does not throw another exception. Give savepoints another go on MySQL, can't get release working however get support for basic round trip going Fixes: #5327 Change-Id: Ia3cbbf56d4882fcc7980f90519412f1711fae74d
* Documentation updates for ResultProxy -> ResultMike Bayer2020-05-011-3/+3
| | | | | | | | | This is based off of I8091919d45421e3f53029b8660427f844fee0228 and includes all documentation-only changes as a separate merge, once the parent is merged. Change-Id: I711adea23df0f9f0b1fe7c76210bd2de6d31842d
* Create initial 2.0 engine implementationMike Bayer2020-04-161-5/+14
| | | | | | | | | | | | | | | | | | | Implemented the SQLAlchemy 2 :func:`.future.create_engine` function which is used for forwards compatibility with SQLAlchemy 2. This engine features always-transactional behavior with autobegin. Allow execution options per statement execution. This includes that the before_execute() and after_execute() events now accept an additional dictionary with these options, empty if not passed; a legacy event decorator is added for backwards compatibility which now also emits a deprecation warning. Add some basic tests for execution, transactions, and the new result object. Build out on a new testing fixture that swaps in the future engine completely to start with. Change-Id: I70e7338bb3f0ce22d2f702537d94bb249bd9fb0a Fixes: #4644
* Set up absolute references for create_engine and relatedMike Bayer2020-04-141-3/+5
| | | | | | | includes more replacements for create_engine(), Connection, disambiguation of Result from future/baked Change-Id: Icb60a79ee7a6c45ea9056c211ffd1be110da3b5e
* Run search and replace of symbolic module namesMike Bayer2020-04-141-75/+106
| | | | | | | | Replaces a wide array of Sphinx-relative doc references with an abbreviated absolute form now supported by zzzeeksphinx. Change-Id: I94bffcc3f37885ffdde6238767224296339698a2
* Remove code deprecated before version 1.1Federico Caselli2020-04-091-82/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Remove deprecated method ``get_primary_keys` in the :class:`.Dialect` and :class:`.Inspector` classes. - Remove deprecated event ``dbapi_error`` and the method ``ConnectionEvents.dbapi_error`. - Remove support for deprecated engine URLs of the form ``postgres://``. - Remove deprecated dialect ``mysql+gaerdbms``. - Remove deprecated parameter ``quoting`` from :class:`.mysql.ENUM` and :class:`.mysql.SET` in the ``mysql`` dialect. - Remove deprecated function ``comparable_property``. and function ``comparable_using`` in the declarative extension. - Remove deprecated function ``compile_mappers``. - Remove deprecated method ``collection.linker``. - Remove deprecated method ``Session.prune`` and parameter ``Session.weak_identity_map``. This change also removes the class ``StrongInstanceDict``. - Remove deprecated parameter ``mapper.order_by``. - Remove deprecated parameter ``Session._enable_transaction_accounting`. - Remove deprecated parameter ``Session.is_modified.passive``. - Remove deprecated class ``Binary``. Please use :class:`.LargeBinary`. - Remove deprecated methods ``Compiled.compile``, ``ClauseElement.__and__`` and ``ClauseElement.__or__`` and attribute ``Over.func``. - Remove deprecated ``FromClause.count`` method. - Remove deprecated parameter ``Table.useexisting``. - Remove deprecated parameters ``text.bindparams`` and ``text.typemap``. - Remove boolean support for the ``passive`` parameter in ``get_history``. - Remove deprecated ``adapt_operator`` in ``UserDefinedType.Comparator``. Fixes: #4643 Change-Id: Idcd390c77bf7b0e9957907716993bdaa3f1a1763
* Run autoflush for column attribute load operationsMike Bayer2020-04-031-4/+2
| | | | | | | | | | | | | | | | | The "autoflush" behavior of :class:`.Query` will now trigger for nearly all ORM level attribute load operations, including when a deferred column is loaded as well as when an expired column is loaded. Previously, autoflush on load of expired or unloaded attributes was limited to relationship-bound attributes only. However, this led to the issue where column-based attributes that also depended on other rows, or even other columns in the same row, in order to express the correct value, would show an effectively stale value when accessed as there could be pending changes in the session left to be flushed. Autoflush is now disabled only in some cases where attributes are being unexpired in the context of a history operation. Fixes: #5226 Change-Id: Ibd965b30918cd273ae020411a704bf2bb1891f59
* Remove deprecated elements from selectable.py; remove lockmodeMike Bayer2020-03-231-14/+4
| | | | | | | | | | | | | Removed autocommit and legacy "for update" / "lockmode" elements from selectable.py / query.py. lockmode was removed from selectable in 693938dd6fb2f3ee3e031aed4c62355ac97f3ceb however was not removed from the ORM. Also removes the ignore_nonexistent_tables option on join(). Change-Id: I0cfcf9e6a8d4ef6432c9e25ef75173b3b3f5fd87 Partially-fixes: #4643
View Lorry Controller Status