summaryrefslogtreecommitdiff
path: root/lib/sqlalchemy/sql/coercions.py
Commit message (Collapse)AuthorAgeFilesLines
* happy new year 2022Mike Bayer2022-01-061-1/+1
| | | | Change-Id: I49abf2607e0eb0623650efdf0091b1fb3db737ea
* Remove all remaining removed_in_20 warnings slated for removalMike Bayer2022-01-051-14/+6
| | | | | | | | | | | | | | | | | | | | | | | | | Finalize all remaining removed-in-2.0 changes so that we can begin doing pep-484 typing without old things getting in the way (we will also have to do public_factory). note there are a few "moved_in_20()" and "became_legacy_in_20()" warnings still in place. The SQLALCHEMY_WARN_20 variable is now removed. Also removed here are the legacy "in place mutators" for Select statements, and some keyword-only argument signatures in Core have been added. Also in the big change department, the ORM mapper() function is removed entirely; the Mapper class is otherwise unchanged, just the public-facing API function. Mappers are now always given a registry in which to participate, however the argument signature of Mapper is not changed. ideally "registry" would be the first positional argument. Fixes: #7257 Change-Id: Ic70c57b9f1cf7eb996338af5183b11bdeb3e1623
* Update Black's target-version to py37Hugo van Kemenade2022-01-051-5/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | <!-- Provide a general summary of your proposed changes in the Title field above --> ### Description <!-- Describe your changes in detail --> Black's `target-version` was still set to `['py27', 'py36']`. Set it to `[py37]` instead. Also update Black and other pre-commit hooks and re-format with Black. ### Checklist <!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once) --> This pull request is: - [ ] A documentation / typographical error fix - Good to go, no issue or tests are needed - [ ] A short code fix - please include the issue number, and create an issue if none exists, which must include a complete example of the issue. one line code fixes without an issue and demonstration will not be accepted. - Please include: `Fixes: #<issue number>` in the commit message - please include tests. one line code fixes without tests will not be accepted. - [ ] A new feature implementation - please include the issue number, and create an issue if none exists, which must include a complete example of how the feature would look. - Please include: `Fixes: #<issue number>` in the commit message - please include tests. **Have a nice day!** Closes: #7536 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/7536 Pull-request-sha: b3aedf5570d7e0ba6c354e5989835260d0591b08 Change-Id: I8be85636fd2c9449b07a8626050c8bd35bd119d5
* remove 2.0-removed Query elementsMike Bayer2022-01-011-14/+5
| | | | | | | | | | | | | | * :meth:`_orm.Query.join` no longer accepts the "aliased" and "from_joinpoint" arguments * :meth:`_orm.Query.join` no longer accepts chains of multiple join targets in one method call. * ``Query.from_self()`` and ``Query.with_polymorphic()`` are removed. Change-Id: I534d04b53a538a4fc374966eb2bc8eb98a16497d References: #7257
* remove legacy select patternsMike Bayer2021-12-291-4/+21
| | | | | Change-Id: If6e521a1eb461e08748a0432943b938528a2619e References: #7257
* Replace raise_ with raise fromFederico Caselli2021-12-271-13/+10
| | | | | Change-Id: I7aaeb5bc130271624335b79cf586581d6c6c34c7 References: #4600
* Removals: strings for join(), loader_options().Mike Bayer2021-12-081-25/+2
| | | | | | | | | | | | | | | | * The :meth:`_orm.Query.join` method no longer accepts strings for relationship names; the long-documented approach of using ``Class.attrname`` for join targets is now standard. * Loader options no longer accept strings for attribute names. The long-documented approach of using ``Class.attrname`` for loader option targets is now standard. It is hoped that a subsequent commit can refactor loader options to no longer need "UnboundLoad" for most cases. Change-Id: If4629882c40523dccbf4459256bf540fb468b618 References: #6986
* Warn when caching is disabled / documentMike Bayer2021-12-061-0/+16
| | | | | | | | | | | | | | | | | | | | | | | | | | | This patch adds new warnings for all elements that don't indicate their caching behavior, including user-defined ClauseElement subclasses and third party dialects. it additionally adds new documentation to discuss an apparent performance degradation in 1.4 when caching is disabled as a result in the significant expense incurred by ORM lazy loaders, which in 1.3 used BakedQuery so were actually cached. As a result of adding the warnings, a fair degree of lesser used SQL expression objects identified that they did not define caching behavior so would have been producing ``[no key]``, including PostgreSQL constructs ``hstore`` and ``array``. These have been amended to use inherit cache where appropriate. "on conflict" constructs in PostgreSQL, MySQL, SQLite still explicitly don't generate a cache key at this time. The change also adds a test for all constructs via assert_compile() to assert they will not generate cache warnings. Fixes: #7394 Change-Id: I85958affbb99bfad0f5efa21bc8f2a95e7e46981
* Clean up most py3k compatFederico Caselli2021-11-241-8/+8
| | | | Change-Id: I8172fdcc3103ff92aa049827728484c8779af6b7
* Remove object in class definitionFederico Caselli2021-11-221-8/+8
| | | | | References: #4600 Change-Id: I2a62ddfe00bc562720f0eae700a497495d7a987a
* Warn when trying to execute a query object with a session.Federico Caselli2021-10-041-1/+22
| | | | | | | | Passing a :class:`.Query` object to :meth:`_orm.Session.execute` is not the intended use of this object, and will now raise a deprecation warning. Fixes: #6284 Change-Id: I30a406d5a72335f9405f10f0a2030a32ccda41b9
* dont warn for dictionary passed positionallyMike Bayer2021-07-211-5/+6
| | | | | | | | | | | | | Fixed issue where use of the :paramref:`_sql.case.whens` parameter passing a dictionary positionally and not as a keyword argument would emit a 2.0 deprecation warning, referring to the deprecation of passing a list positionally. The dictionary format of "whens", passed positionally, is still supported and was accidentally marked as deprecated. Removes warning filter for case statement. Fixes: #6786 Change-Id: I8efd1882563773bec89ae5e34f0dfede77fc4683
* limit None->null coercion to not occur with crudMike Bayer2021-07-151-1/+8
| | | | | | | | | | | | | | | | | | | Fixed issue where type-specific bound parameter handlers would not be called upon in the case of using the :meth:`_sql.Insert.values` method with the Python ``None`` value; in particular, this would be noticed when using the :class:`_types.JSON` datatype as well as related PostgreSQL specific types such as :class:`_postgresql.JSONB` which would fail to encode the Python ``None`` value into JSON null, however the issue was generalized to any bound parameter handler in conjunction with this specific method of :class:`_sql.Insert`. The issue with coercions forcing out ``null()`` may still impact SQL expression usage as well; the change here is limited to crud as the behavior there is relevant to some use cases, which may need to be evaluated separately. Fixes: #6770 Change-Id: If53edad811b37dada7578a89daf395628db058a6
* implement independent CTEsMike Bayer2021-07-121-0/+4
| | | | | | | | | | | | | | Added new method :meth:`_sql.HasCTE.add_cte` to each of the :func:`_sql.select`, :func:`_sql.insert`, :func:`_sql.update` and :func:`_sql.delete` constructs. This method will add the given :class:`_sql.CTE` as an "independent" CTE of the statement, meaning it renders in the WITH clause above the statement unconditionally even if it is not otherwise referenced in the primary statement. This is a popular use case on the PostgreSQL database where a CTE is used for a DML statement that runs against database rows independently of the primary statement. Fixes: #6752 Change-Id: Ibf635763e40269cbd10f4c17e208850d8e8d0188
* Replace all http:// links to https://Federico Caselli2021-07-041-1/+1
| | | | | | Also replace http://pypi.python.org/pypi with https://pypi.org/project Change-Id: I84b5005c39969a82140706472989f2a30b0c7685
* Ensure propagate_attrs available on PropComparatorMike Bayer2021-05-291-0/+3
| | | | | | | | | | | | | | | | | Fixed regression caused by just-released performance fix mentioned in #6550 where a query.join() to a relationship could produce an AttributeError if the query were made against non-ORM structures only, a fairly unusual calling pattern. In this fix, since we are no longer going through the production of ``__clause_element__()`` for Cls.some_relationship, I assumed we just throw this object away completely but I missed the one little bit where we might be getting ``_propagate_attrs`` from it. So we implement ``_propagate_attrs`` on ``PropComparator`` as well, since this is easy to define. Fixes: #6558 Change-Id: If781bf844e7e3d3b0841aff1c3668e9d6af9f097
* Fix adaption in AnnotatedLabel; repair needless expense in coercionMike Bayer2021-05-281-5/+21
| | | | | | | | | | | | | | | Fixed regression involving clause adaption of labeled ORM compound elements, such as single-table inheritance discriminator expressions with conditionals or CASE expressions, which could cause aliased expressions such as those used in ORM join / joinedload operations to not be adapted correctly, such as referring to the wrong table in the ON clause in a join. This change also improves a performance bump that was located within the process of invoking :meth:`_sql.Select.join` given an ORM attribute as a target. Fixes: #6550 Change-Id: I98906476f0cce6f41ea00b77c789baa818e9d167
* Remove pep484 type comments from the codeFederico Caselli2021-05-161-9/+7
| | | | | | | | | | | | | Current effort is around the stub package, and having typing in two places makes thing worse, since the types here are usually outdated compared to the version in the stubs. Once v2 gets under way we can start consolidating the types here. Fixes: #6461 Change-Id: I7132a444bd7138123074bf5bc664b4bb119a85ce
* set bindparam.expanding in coercion againMike Bayer2021-05-101-3/+4
| | | | | | | | | | | | | | | | | Adjusted the logic added as part of :ticket:`6397` in 1.4.12 so that internal mutation of the :class:`.BindParameter` object occurs within the clause construction phase as it did before, rather than in the compilation phase. In the latter case, the mutation still produced side effects against the incoming construct and additionally could potentially interfere with other internal mutation routines. In order to solve the issue of the correct operator being present on the BindParameter.expand_op, we necessarily have to expand the BinaryExpression._negate() routine to flip the operator on the BindParameter also. Fixes: #6460 Change-Id: I1e53a9aeee4de4fc11af51d7593431532731561b
* Use non-subquery form for empty INMike Bayer2021-04-291-8/+3
| | | | | | | | | | | | | | | | | | | | Revised the "EMPTY IN" expression to no longer rely upon using a subquery, as this was causing some compatibility and performance problems. The new approach for selected databases takes advantage of using a NULL-returning IN expression combined with the usual "1 != 1" or "1 = 1" expression appended by AND or OR. The expression is now the default for all backends other than SQLite, which still had some compatibility issues regarding tuple "IN" for older SQLite versions. Third party dialects can still override how the "empty set" expression renders by implementing a new compiler method ``def visit_empty_set_op_expr(self, type_, expand_op)``, which takes precedence over the existing ``def visit_empty_set_expr(self, element_types)`` which remains in place. Fixes: #6258 Fixes: #6397 Change-Id: I2df09eb00d2ad3b57039ae48128fdf94641b5e59
* dont assume ClauseElement in attributes, coercionsMike Bayer2021-04-231-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixed two distinct issues, each of which would come into play under certain circumstances, most likely however one which is a common mis-configuration in :class:`_hybrid.hybrid_property`, where the "expression" implementation would return a non :class:`_sql.ClauseElement` such as a boolean value. For both issues, 1.3's behavior was to silently ignore the mis-configuration and ultimately attempt to interpret the value as a SQL expression, which would lead to an incorrect query. * Fixed issue regarding interaction of the attribute system with hybrid_property, where if the ``__clause_element__()`` method of the attribute returned a non-:class:`_sql.ClauseElement` object, an internal ``AttributeError`` would lead the attribute to return the ``expression`` function on the hybrid_property itself, as the attribute error was against the name ``.expression`` which would invoke the ``__getattr__()`` method as a fallback. This now raises explicitly. In 1.3 the non-:class:`_sql.ClauseElement` was returned directly. * Fixed issue in SQL argument coercions system where passing the wrong kind of object to methods that expect column expressions would fail if the object were altogether not a SQLAlchemy object, such as a Python function, in cases where the object were not just coerced into a bound value. Again 1.3 did not have a comprehensive argument coercion system so this case would also pass silently. Fixes: #6350 Change-Id: I5bba0a6b27f45e5f8ebadfd6d511fa773388ef7c
* Disallow AliasedReturnsRows from executionMike Bayer2021-04-051-6/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | Executing a :class:`_sql.Subquery` using :meth:`_engine.Connection.execute` is deprecated and will emit a deprecation warning; this use case was an oversight that should have been removed from 1.4. The operation will now execute the underlying :class:`_sql.Select` object directly for backwards compatibility. Similarly, the :class:`_sql.CTE` class is also not appropriate for execution. In 1.3, attempting to execute a CTE would result in an invalid "blank" SQL statement being executed; since this use case was not working it now raises :class:`_exc.ObjectNotExecutableError`. Previously, 1.4 was attempting to execute the CTE as a statement however it was working only erratically. The change also breaks out StatementRole from ReturnsRowsRole, as these roles should not be in the same lineage (some statements don't return rows, the whole class of ReturnsRows that are from clauses are not statements). Consolidate StatementRole and CoerceTextStatementRole as there's no usage difference between these. Simplify some old tests that were trying to make sure that "execution options" didn't transmit from a cte/subquery out to a select; as cte/subuqery() aren't executable in any case the options are removed. Fixes: #6204 Change-Id: I62613b7ab418afdd22c409eae75659e3f52fb65f
* Correct for coercion from list args to positional for caseMike Bayer2021-03-191-1/+1
| | | | | | | | | Fixed regression in the :func:`_sql.case` construct, where the "dictionary" form of argument specification failed to work correctly if it were passed positionally, rather than as a "whens" keyword argument. Fixes: #6097 Change-Id: I4138f54309a08c8e4e63cfafc211176e0b9a76c7
* turn off eager configure_mappers() outside of Query, LoadMike Bayer2021-03-161-2/+10
| | | | | | | | | | | | | | | | | | | | | | | | Fixed regression where producing a Core expression construct such as :func:`_sql.select` using ORM entities would eagerly configure the mappers, in an effort to maintain compatibility with the :class:`_orm.Query` object which necessarily does this to support many backref-related legacy cases. However, core :func:`_sql.select` constructs are also used in mapper configurations and such, and to that degree this eager configuration is more of an inconvenience, so eager configure has been disabled for the :func:`_sql.select` and other Core constructs in the absence of ORM loading types of functions such as :class:`_orm.Load`. The change maintains the behavior of :class:`_orm.Query` so that backwards compatibility is maintained. However, when using a :func:`_sql.select` in conjunction with ORM entities, a "backref" that isn't explicitly placed on one of the classes until mapper configure time won't be available unless :func:`_orm.configure_mappers` or the newer :func:`_orm.registry.configure` has been called elsewhere. Prefer using :paramref:`_orm.relationship.back_populates` for more explicit relationship configuration which does not have the eager configure requirement. Fixes: #6066 Change-Id: I7a953ddcc189471fbac63c97c51ab8956f64012e
* Merge "Implement support for functions as FROM with columns clause support"mike bayer2021-02-031-0/+21
|\
| * Implement support for functions as FROM with columns clause supportMike Bayer2021-02-031-0/+21
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Implemented support for "table valued functions" along with additional syntaxes supported by PostgreSQL, one of the most commonly requested features. Table valued functions are SQL functions that return lists of values or rows, and are prevalent in PostgreSQL in the area of JSON functions, where the "table value" is commonly referred towards as the "record" datatype. Table valued functions are also supported by Oracle and SQL Server. Moved from I5b093b72533ef695293e737eb75850b9713e5e03 due to accidental push Fixes: #3566 Change-Id: Iea36d04c80a5ed3509dcdd9ebf0701687143fef5
* | Add coercions to literal()Mike Bayer2021-02-031-3/+20
|/ | | | | | | | | | To prevent literal() from receiving a ClauseElement which then leads to confusing results, add a new LiteralValueRole coercion that does an _is_literal() check and implement for literal(). Fixes: #5639 Change-Id: Ibd686544af2d7c013765278f984baf237de88caf
* Replace with_labels() and apply_labels() in ORM/CoreGord Thompson2021-01-261-1/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Replace :meth:`_orm.Query.with_labels` and :meth:`_sql.GenerativeSelect.apply_labels` with explicit getters and setters ``get_label_style`` and ``set_label_style`` to accommodate the three supported label styles: ``LABEL_STYLE_DISAMBIGUATE_ONLY`` (default), ``LABEL_STYLE_TABLENAME_PLUS_COL``, and ``LABEL_STYLE_NONE``. In addition, for Core and "future style" ORM queries, ``LABEL_STYLE_DISAMBIGUATE_ONLY`` is now the default label style. This style differs from the existing "no labels" style in that labeling is applied in the case of column name conflicts; with ``LABEL_STYLE_NONE``, a duplicate column name is not accessible via name in any case. For legacy ORM queries using :class:`_query.Query`, the table-plus-column names labeling style applied by ``LABEL_STYLE_TABLENAME_PLUS_COL`` continues to be used so that existing test suites and logging facilities see no change in behavior by default, however this style of labeling is no longer required for SQLAlchemy queries to function, as result sets are commonly matched to columns using a positional approach since SQLAlchemy 1.0. Within test suites, all use of apply_labels() / use_labels now uses the new methods. New tests added to test/sql/test_deprecations.py nad test/orm/test_deprecations.py to cover just the old apply_labels() method call. Tests in ORM that made explicit use apply_labels()/ etc. where it isn't needed for the ORM to work correctly use default label style now. Co-authored-by: Mike Bayer <mike_mp@zzzcomputing.com> Fixes: #4757 Change-Id: I5fdcd2ed4ae8c7fe62f8be2b6d0e8f66409b6a54
* Revert "Implement support for functions as FROM with columns clause support"Mike Bayer2021-01-211-21/+0
| | | | | | | This reverts commit 05a31f2708590161d4b3b4c7ff65196c99b4a22b. Atom has this little button called "push" and just pushes to master, I wasn't even *on* master. oops
* Implement support for functions as FROM with columns clause supportMike Bayer2021-01-201-0/+21
| | | | | | | WIP Fixes: #3566 Change-Id: I5b093b72533ef695293e737eb75850b9713e5e03
* happy new yearMike Bayer2021-01-041-1/+1
| | | | Change-Id: Ic5bb19ca8be3cb47c95a0d3315d84cb484bac47c
* Major revisals to lambdasMike Bayer2020-12-161-4/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 1. Improve coercions._deep_is_literal to check sequences for clause elements, thus allowing a phrase like lambda: col.in_([literal("x"), literal("y")]) to be handled 2. revise closure variable caching completely. All variables entering must be part of a closure cache key or rejected. only objects that can be resolved to HasCacheKey or FunctionType are accepted; all other types are rejected. This adds a high degree of strictness to lambdas and will make them a little more awkward to use in some cases, however prevents several classes of critical issues: a. previously, a lambda that had an expression derived from some kind of state, like "self.x", or "execution_context.session.foo" would produce a closure cache key from "self" or "execution_context", objects that can very well be per-execution and would therefore cause a AnalyzedFunction objects to overflow. (memory won't leak as it looks like an LRUCache is already used for these) b. a lambda, such as one used within DeferredLamdaElement, that produces different SQL expressions based on the arguments (which is in fact what it's supposed to do), however it would through the use of conditionals produce different bound parameter combinations, leading to literal parameters not tracked properly. These are now rejected as uncacheable whereas previously they would again be part of the closure cache key, causing an overflow of AnalyizedFunction objects. 3. Ensure non-mapped mixins are handled correctly by with_loader_criteria(). 4. Fixed bug in lambda SQL system where we are not supposed to allow a Python function to be embedded in the lambda, since we can't predict a bound value from it. While there was an error condition added for this, it was not tested and wasn't working; an informative error is now raised. 5. new docs for lambdas 6. consolidated changelog for all of these Fixes: #5760 Fixes: #5765 Fixes: #5766 Fixes: #5768 Fixes: #5770 Change-Id: Iedaa636c3225fad496df23b612c516c8ab247ab7
* Emit deprecation warnings for plain text under SessionMike Bayer2020-12-111-17/+10
| | | | | | | | | | | | | | | | Deprecation warnings are emitted under "SQLALCHEMY_WARN_20" mode when passing a plain string to :meth:`_orm.Session.execute`. It was also considered to have DDL string expressions to include this as well, however this leaves us with no backwards-compatible way of handling reflection of elemens, such as an Index() which reflects "postgresql_where='x > 5'", there's no place for a rule that will turn those into text() within the reflection process that would be separate from when the user passes postgresql_where to the Index. Not worth it right now. Fixes: #5754 Change-Id: I8673a79f0e87de0df576b655f39dad0351725ca8
* Deprecate strings indicating attribute namesMike Bayer2020-10-131-2/+36
| | | | | | | | | | | | | | | | | | | Using strings to represent relationship names in ORM operations such as :meth:`_orm.Query.join`, as well as strings for all ORM attribute names in loader options like :func:`_orm.selectinload` is deprecated and will be removed in SQLAlchemy 2.0. The class-bound attribute should be passed instead. This provides much better specificity to the given method, allows for modifiers such as ``of_type()``, and reduces internal complexity. Additionally, the ``aliased`` and ``from_joinpoint`` parameters to :meth:`_orm.Query.join` are also deprecated. The :func:`_orm.aliased` construct now provides for a great deal of flexibility and capability and should be used directly. Fixes: #4705 Fixes: #5202 Change-Id: I32f61663d68026154906932913c288f269991adc
* upgrade to black 20.8b1Mike Bayer2020-09-281-3/+7
| | | | | | | It's better, the majority of these changes look more readable to me. also found some docstrings that had formatting / quoting issues. Change-Id: I582a45fde3a5648b2f36bab96bad56881321899b
* Add deprecation warning for .join().alias()Gord Thompson2020-09-281-2/+4
| | | | | | | | | The :meth:`_sql.Join.alias` method is deprecated and will be removed in SQLAlchemy 2.0. An explicit select + subquery, or aliasing of the inner tables, should be used instead. Fixes: #5010 Change-Id: Ic913afc31f0d70b0605f9a7af2742a0de1f9ad19
* Complete deprecation of from_self()Mike Bayer2020-09-181-4/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | For most from_self() tests, move them into test/orm/test_deprecated.py and replace the existing test with one that uses aliased() plus a subquery. This then revealed a few more issues. Related items: * Added slice() method to GenerativeSelect, to match that of orm.Query and to make possible migration of one of the from_self() tests. moved the utility functions used for this from orm/util into sql/util. * repairs a caching issue related to subqueryload where information being derived from the cached path info was mixing up with query information based on the per-query state, specifically an AliasedClass that is per query. * for the above issue, it seemed like path_registry maybe had to change so that it represents AliasedClass objects as their cache key rather than on identity, but it wasn't needed. still seems like it would be more correct. * enhances the error message raised by coercions for a case such as when an AliasedClass holds onto a select() object and not a subquery(); will name the original and resolved object for clarity (although how is AliasedClass able to accept a Select() object in the first place?) * Added _set_propagate_attrs() to Query so that again if it's passed to AliasedClass, it doesn't raise an error during coercion, but again maybe that should also be rejected up front Fixes: #5368 Change-Id: I5912aa611d899acc87a75eb5ee9f95990592f210
* Create a real type for Tuple() and handle appropriately in compilerMike Bayer2020-08-171-5/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | Improved the :func:`_sql.tuple_` construct such that it behaves predictably when used in a columns-clause context. The SQL tuple is not supported as a "SELECT" columns clause element on most backends; on those that do (PostgreSQL, not surprisingly), the Python DBAPI does not have a "nested type" concept so there are still challenges in fetching rows for such an object. Use of :func:`_sql.tuple_` in a :func:`_sql.select` or :class:`_orm.Query` will now raise a :class:`_exc.CompileError` at the point at which the :func:`_sql.tuple_` object is seen as presenting itself for fetching rows (i.e., if the tuple is in the columns clause of a subquery, no error is raised). For ORM use,the :class:`_orm.Bundle` object is an explicit directive that a series of columns should be returned as a sub-tuple per row and is suggested by the error message. Additionally ,the tuple will now render with parenthesis in all contexts. Previously, the parenthesization would not render in a columns context leading to non-defined behavior. As part of this change, Tuple receives a dedicated datatype which appears to allow us the very desirable change of removing the bindparam._expanding_in_types attribute as well as ClauseList._tuple_values (which might already have not been needed due to #4645). Fixes: #5127 Change-Id: Iecafa0e0aac2f1f37ec8d0e1631d562611c90200
* Robustness for lambdas, lambda statementsMike Bayer2020-08-051-3/+38
| | | | | | | | | | | | | | | | | | | | in order to accommodate relationship loaders with lambda caching, a lot more is needed. This is a full refactor of the lambda system such that it now has two levels of caching; the first level caches what can be known from the __code__ element, then the next level of caching is against the lambda itself and the contents of __closure__. This allows for the elements inside the lambdas, like columns and entities, to change and then be part of the cache key. Lazy/selectinloads' use of baked queries had to add distinct cache key elements, which was attempted here but overall things needed to be more robust than that. This commit is broken out from the very long and sprawling commit at Id6b5c03b1ce9ddb7b280f66792212a0ef0a1c541 . Change-Id: I29a513c98917b1d503abfdd61e6b6e8800851aa8
* Convert remaining ORM APIs to support 2.0 styleMike Bayer2020-07-111-0/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is kind of a mixed bag of all kinds to help get us to 1.4 betas. The documentation stuff is a work in progress. Lots of other relatively small changes to APIs and things. More commits will follow to continue improving the documentation and transitioning to the 1.4/2.0 hybrid documentation. In particular some refinements to Session usage models so that it can match Engine's scoping / transactional patterns, and a decision to start moving away from "subtransactions" completely. * add select().from_statement() to produce FromStatement in an ORM context * begin referring to select() that has "plugins" for the few edge cases where select() will have ORM-only behaviors * convert dynamic.AppenderQuery to its own object that can use select(), though at the moment it uses Query to support legacy join calling forms. * custom query classes for AppenderQuery are replaced by do_orm_execute() hooks for custom actions, a separate gerrit will document this * add Session.get() to replace query.get() * Deprecate session.begin->subtransaction. propose within the test suite a hypothetical recipe for apps that rely on this pattern * introduce Session construction level context manager, sessionmaker context manager, rewrite the whole top of the session_transaction.rst documentation. Establish context manager patterns for Session that are identical to engine * ensure same begin_nested() / commit() behavior as engine * devise all new "join into an external transaction" recipe, add test support for it, add rules into Session so it just works, write new docs. need to ensure this doesn't break anything * vastly reduce the verbosity of lots of session docs as I dont think people read this stuff and it's difficult to keep current in any case * constructs like case(), with_only_columns() really need to move to *columns, add a coercion rule to just change these. * docs need changes everywhere I look. in_() is not in the Core tutorial? how do people even know about it? Remove tons of cruft from Select docs, etc. * build a system for common ORM options like populate_existing and autoflush to populate from execution options. * others? Change-Id: Ia4bea0f804250e54d90b3884cf8aab8b66b82ecf
* introduce deferred lambdasMike Bayer2020-07-031-2/+84
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The coercions system allows us to add in lambdas as arguments to Core and ORM elements without changing them at all. By allowing the lambda to produce a deterministic cache key where we can also cheat and yank out literal parameters means we can move towards having 90% of "baked" functionality in a clearer way right in Core / ORM. As a second step, we can have whole statements inside the lambda, and can then add generation with __add__(), so then we have 100% of "baked" functionality with full support of ad-hoc literal values. Adds some more short_selects tests for the moment for comparison. Other tweaks inside cache key generation as we're trying to approach a certain level of performance such that we can remove the use of "baked" from the loader strategies. As we have not yet closed #4639, however the caching feature has been fully integrated as of b0cfa7379cf8513a821a3dbe3028c4965d9f85bd, we will also add complete caching documentation here and close that issue as well. Closes: #4639 Fixes: #5380 Change-Id: If91f61527236fd4d7ae3cad1f24c38be921c90ba
* Convert bulk update/delete to new execution modelMike Bayer2020-06-061-0/+10
| | | | | | | | | | | | | | | This reorganizes the BulkUD model in sqlalchemy.orm.persistence to be based on the CompileState concept and to allow plain update() / delete() to be passed to session.execute() where the ORM synchronize session logic will take place. Also gets "synchronize_session='fetch'" working with horizontal sharding. Adding a few more result.scalar_one() types of methods as scalar_one() seems like what is normally desired. Fixes: #5160 Change-Id: I8001ebdad089da34119eb459709731ba6c0ba975
* Refine IN and scalar subquery coercionsMike Bayer2020-06-011-8/+16
| | | | | | | | | | | | | | | | | Ensure IN emits a warning when it coerces a FromClause into a select(), however that it continues to allow the scalar_subquery() coercion to be automatic, particularly since it's not clear that "col IN (select)" is necessarily "scalar" in the case of tuples. Convert the "scalar_subquery()" warning emitted in other cases to be a warning, rather than a deprecation warning. I can't imagine taking this coercion out as it is intuitive and is always going to happen; we just would like to note that an implicit coercion is occurring. Fixes: #5369 Change-Id: I748f01f40bc85c64e2776f9b88ef35641fa8fb5c
* callcount reductions and refinement for cached queriesMike Bayer2020-05-281-47/+44
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This commit includes that we've removed the "_orm_query" attribute from compile state as well as query context. The attribute created reference cycles and also added method call overhead. As part of this change, the interface for ORMExecuteState changes a bit, as well as the interface for the horizontal sharding extension which now deprecates the "query_chooser" callable in favor of "execute_chooser", which receives the contextual object. This will also work more nicely when we implement the new execution path for bulk updates and deletes. Pre-merge execution options for statement, connection, arguments all up front in Connection. that way they can be passed to the before_execute / after_execute events, and the ExecutionContext doesn't have to merge as second time. Core execute is pretty close to 1.3 now. baked wasn't using the new one()/first()/one_or_none() methods, fixed that. Convert non-buffered cursor strategy to be a stateless singleton. inline all the paths by which the strategy gets chosen, oracle and SQL Server dialects make use of the already-invoked post_exec() hook to establish the alternate strategies, and this is actually much nicer than it was before. Add caching to mapper instance processor for getters. Identified a reference cycle per query that was showing up as a lot of gc cleanup, fixed that. After all that, performance not budging much. Even test_baked_query now runs with significantly fewer function calls than 1.3, still 40% slower. Basically something about the new patterns just makes this slower and while I've walked a whole bunch of them back, it hardly makes a dent. that said, the performance issues are relatively small, in the 20-40% time increase range, and the new caching feature does provide for regular ORM and Core queries that are cached, and they are faster than non-cached. Change-Id: I7b0b0d8ca550c05f79e82f75cd8eff0bbfade053
* Convert execution to move through SessionMike Bayer2020-05-251-29/+28
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch replaces the ORM execution flow with a single pathway through Session.execute() for all queries, including Core and ORM. Currently included is full support for ORM Query, Query.from_statement(), select(), as well as the baked query and horizontal shard systems. Initial changes have also been made to the dogpile caching example, which like baked query makes use of a new ORM-specific execution hook that replaces the use of both QueryEvents.before_compile() as well as Query._execute_and_instances() as the central ORM interception hooks. select() and Query() constructs alike can be passed to Session.execute() where they will return ORM results in a Results object. This API is currently used internally by Query. Full support for Session.execute()->results to behave in a fully 2.0 fashion will be in later changesets. bulk update/delete with ORM support will also be delivered via the update() and delete() constructs, however these have not yet been adapted to the new system and may follow in a subsequent update. Performance is also beginning to lag as of this commit and some previous ones. It is hoped that a few central functions such as the coercions functions can be rewritten in C to re-gain performance. Additionally, query caching is now available and some subsequent patches will attempt to cache more of the per-execution work from the ORM layer, e.g. column getters and adapters. This patch also contains initial "turn on" of the caching system enginewide via the query_cache_size parameter to create_engine(). Still defaulting at zero for "no caching". The caching system still needs adjustments in order to gain adequate performance. Change-Id: I047a7ebb26aa85dc01f6789fac2bff561dcd555d
* Unify Query and select() , move all processing to compile phaseMike Bayer2020-05-241-5/+77
| | | | | | | | | | | | | | | | | | | | | | | | | | Convert Query to do virtually all compile state computation in the _compile_context() phase, and organize it all such that a plain select() construct may also be used as the source of information in order to generate ORM query state. This makes it such that Query is not needed except for its additional methods like from_self() which are all to be deprecated. The construction of ORM state will occur beyond the caching boundary when the new execution model is integrated. future select() gains a working join() and filter_by() method. as we continue to rebase and merge each commit in the steps, callcounts continue to bump around. will have to look at the final result when it's all in. References: #5159 References: #4705 References: #4639 References: #4871 References: #5010 Change-Id: I19e05b3424b07114cce6c439b05198ac47f7ac10
* Structural / performance refinementsMike Bayer2020-05-221-76/+103
| | | | | | | | | | | | * state connection schema_translate_map entirely in terms of execution options, support for per-execution options as well * use slots for role impls, remove superclass of the roles themselves as this is not needed * tighten loop in resolve, might become a C function Change-Id: Ib98ac9b65022fbf976e49c6060e4c37573528c5f
* Remove code deprecated before version 1.1Federico Caselli2020-04-091-3/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Remove deprecated method ``get_primary_keys` in the :class:`.Dialect` and :class:`.Inspector` classes. - Remove deprecated event ``dbapi_error`` and the method ``ConnectionEvents.dbapi_error`. - Remove support for deprecated engine URLs of the form ``postgres://``. - Remove deprecated dialect ``mysql+gaerdbms``. - Remove deprecated parameter ``quoting`` from :class:`.mysql.ENUM` and :class:`.mysql.SET` in the ``mysql`` dialect. - Remove deprecated function ``comparable_property``. and function ``comparable_using`` in the declarative extension. - Remove deprecated function ``compile_mappers``. - Remove deprecated method ``collection.linker``. - Remove deprecated method ``Session.prune`` and parameter ``Session.weak_identity_map``. This change also removes the class ``StrongInstanceDict``. - Remove deprecated parameter ``mapper.order_by``. - Remove deprecated parameter ``Session._enable_transaction_accounting`. - Remove deprecated parameter ``Session.is_modified.passive``. - Remove deprecated class ``Binary``. Please use :class:`.LargeBinary`. - Remove deprecated methods ``Compiled.compile``, ``ClauseElement.__and__`` and ``ClauseElement.__or__`` and attribute ``Over.func``. - Remove deprecated ``FromClause.count`` method. - Remove deprecated parameter ``Table.useexisting``. - Remove deprecated parameters ``text.bindparams`` and ``text.typemap``. - Remove boolean support for the ``passive`` parameter in ``get_history``. - Remove deprecated ``adapt_operator`` in ``UserDefinedType.Comparator``. Fixes: #4643 Change-Id: Idcd390c77bf7b0e9957907716993bdaa3f1a1763
* Try to measure new style caching in the ORM, take twoMike Bayer2020-04-011-5/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Supercedes: If78fbb557c6f2cae637799c3fec2cbc5ac248aaf Trying to see if by making the cache key memoized, we still can have the older "identity" form of caching which is the cheapest of all, at the same time as the newer "cache key each time" version that is not nearly as cheap; but still much cheaper than no caching at all. Also needed is a per-execution update of _keymap when we invoke from a cached select, so that Column objects that are anonymous or otherwise adapted will match up. this is analogous to the adaption of bound parameters from the cache key. Adds test coverage for the keymap / construct_params() changes related to caching. Also hones performance to a large extent for statement construction and cache key generation. Also includes a new memoized attribute approach that vastly simplifies the previous approach of "group_expirable_memoized_property" and finally integrates cleanly with _clone(), _generate(), etc. no more hardcoding of attributes is needed, as well as that most _reset_memoization() calls are no longer needed as the reset is inherent in a _generate() call; this also has dramatic performance improvements. Change-Id: I95c560ffcbfa30b26644999412fb6a385125f663
* Decouple compiler state from DML objects; make cacheableMike Bayer2020-03-061-4/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Targeting select / insert / update / delete, the goal is to minimize overhead of construction and generative methods so that only the raw arguments passed are handled. An interim stage that converts the raw state into more compiler-ready state is added, which is analogous to the ORM QueryContext which will also be rolled in to be a similar concept, as is currently being prototyped in I19e05b3424b07114cce6c439b05198ac47f7ac10. the ORM update/delete BulkUD concept is also going to be rolled onto this idea. So while the compiler-ready state object, here called DMLState, looks a little thin, it's the base of a bigger pattern that will allow for ORM functionality to embed itself directly into the compiler, execution context, and result set objects. This change targets the DML objects, primarily focused on the values() method which is the most complex process. The work done by values() is minimized as much as possible while still being able to create a cache key. Additional computation is then offloaded to a new object ValuesState that is handled by the compiler. Architecturally, a big change here is that insert.values() and update.values() will generate BindParameter objects for the values now, which are then carefully received by crud.py so that they generate the expected names. This is so that the values() portion of these constructs is cacheable. for the "multi-values" version of Insert, this is all skipped and the plan right now is that a multi-values insert is not worth caching (can always be revisited). Using the coercions system in values() also gets us nicer validation for free, we can remove the NotAClauseElement thing from schema, and we also now require scalar_subquery() is called for an insert/update that uses a SELECT as a column value, 1.x deprecation path is added. The traversal system is then applied to the DML objects including tests so that they have traversal, cloning, and cache key support. cloning is not a use case for DML however having it present allows better validation of the structure within the tests. Special per-dialect DML is explicitly not cacheable at the moment, more as a proof of concept that third party DML constructs can exist as gracefully not-cacheable rather than producing an incomplete cache key. A few selected performance improvements have been added as well, simplifying the immutabledict.union() method and adding a new SQLCompiler function that can generate delimeter-separated clauses like WHERE and ORDER BY without having to build a ClauseList object at all. The use of ClauseList will be removed from Select in an upcoming commit. Overall, ClaustList is unnecessary for internal use and only adds overhead to statement construction and will likely be removed as much as possible except for explcit use of conjunctions like and_() and or_(). Change-Id: I408e0b8be91fddd77cf279da97f55020871f75a9
View Lorry Controller Status