summaryrefslogtreecommitdiff
path: root/lib/sqlalchemy/sql/util.py
Commit message (Collapse)AuthorAgeFilesLines
* Merge "Add future=True to create_engine/Session; unify select()"mike bayer2020-07-081-22/+0
|\
| * Add future=True to create_engine/Session; unify select()Mike Bayer2020-07-081-22/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Several weeks of using the future_select() construct has led to the proposal there be just one select() construct again which features the new join() method, and otherwise accepts both the 1.x and 2.x argument styles. This would make migration simpler and reduce confusion. However, confusion may be increased by the fact that select().join() is different Current thinking is we may be better off with a few hard behavioral changes to old and relatively unknown APIs rather than trying to play both sides within two extremely similar but subtly different APIs. At the moment, the .join() thing seems to be the only behavioral change that occurs without the user taking any explicit steps. Session.execute() will still behave the old way as we are adding a future flag. This change also adds the "future" flag to Session() and session.execute(), so that interpretation of the incoming statement, as well as that the new style result is returned, does not occur for existing applications unless they add the use of this flag. The change in general is moving the "removed in 2.0" system further along where we want the test suite to fully pass even if the SQLALCHEMY_WARN_20 flag is set. Get many tests to pass when SQLALCHEMY_WARN_20 is set; this should be ongoing after this patch merges. Improve the RemovedIn20 warning; these are all deprecated "since" 1.4, so ensure that's what the messages read. Make sure the inforamtion link is on all warnings. Add deprecation warnings for parameters present and add warnings to all FromClause.select() types of methods. Fixes: #5379 Fixes: #5284 Change-Id: I765a0b912b3dcd0e995426427d8bb7997cbffd51 References: #5159
* | ensure we unwrap desc() /label() all the way w/ order byMike Bayer2020-07-071-1/+19
|/ | | | | | | | | | | | | The deprecated logic to move order_by expressions up into the columns clause needed adjustment to accommodate for a more deeply-wrapped structure when desc() + label() are combined in an order by column. This structure now comes from coercions in 1.4. it's not clear to me at the moment why it's different from 1.3 but this shouldn't really matter. Fixes: #5443 Change-Id: If909a86f715992318d7aa283603197f7711f1d3b
* Improve rendering of core statements w/ ORM elementsMike Bayer2020-05-311-1/+6
| | | | | | | | | | | | | | | | | | | | This patch contains a variety of ORM and expression layer tweaks to support ORM constructs in select() statements, without the 1.3.x requiremnt in Query that a full _compile_context() + new select() is needed in order to get a working statement object. Includes such tweaks as the ability to implement aliased class of an aliased class, as we are looking to fully support ACs against subqueries, as well as the ability to access anonymously-labeled ColumnProperty expressions within subqueries by naming the ".key" of the label after the property key. Some tuning to query.join() as well as ORMJoin internals to allow things to work more smoothly. Change-Id: Id810f485c5f7ed971529489b84694e02a3356d6d
* Unify Query and select() , move all processing to compile phaseMike Bayer2020-05-241-0/+31
| | | | | | | | | | | | | | | | | | | | | | | | | | Convert Query to do virtually all compile state computation in the _compile_context() phase, and organize it all such that a plain select() construct may also be used as the source of information in order to generate ORM query state. This makes it such that Query is not needed except for its additional methods like from_self() which are all to be deprecated. The construction of ORM state will occur beyond the caching boundary when the new execution model is integrated. future select() gains a working join() and filter_by() method. as we continue to rebase and merge each commit in the steps, callcounts continue to bump around. will have to look at the final result when it's all in. References: #5159 References: #4705 References: #4639 References: #4871 References: #5010 Change-Id: I19e05b3424b07114cce6c439b05198ac47f7ac10
* Streamline visitors.iterateMike Bayer2020-05-181-1/+1
| | | | | | | | | | | | | | | | | | | | This method might be used more significantly in the ORM refactor, so further refine it. * all get_children() methods now work entirely based on iterators. Basically only select() was sensitive to this anymore and it now chains the iterators together * remove all kinds of flags like column_collections, schema_visitor that apparently aren't used anymore. * remove the "depthfirst" visitors as these don't seem to be used either. * make sure select() yields its columns first as these will be used to determine the current mapper. Change-Id: I05273a2d5306a57c2d1b0979050748cf3ac964bf
* Repair find tablesMike Bayer2020-04-091-1/+3
| | | | | | | | The recipe given out on the ML for improving ABC fails in 1.4 due to find_tables() missing out on subqueries, which is due to the refactor of Alias into multiple sub-types. Change-Id: Idd2803b26a84b28db28e0b8c328b8f24550577e4
* Try to measure new style caching in the ORM, take twoMike Bayer2020-04-011-1/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Supercedes: If78fbb557c6f2cae637799c3fec2cbc5ac248aaf Trying to see if by making the cache key memoized, we still can have the older "identity" form of caching which is the cheapest of all, at the same time as the newer "cache key each time" version that is not nearly as cheap; but still much cheaper than no caching at all. Also needed is a per-execution update of _keymap when we invoke from a cached select, so that Column objects that are anonymous or otherwise adapted will match up. this is analogous to the adaption of bound parameters from the cache key. Adds test coverage for the keymap / construct_params() changes related to caching. Also hones performance to a large extent for statement construction and cache key generation. Also includes a new memoized attribute approach that vastly simplifies the previous approach of "group_expirable_memoized_property" and finally integrates cleanly with _clone(), _generate(), etc. no more hardcoding of attributes is needed, as well as that most _reset_memoization() calls are no longer needed as the reset is inherent in a _generate() call; this also has dramatic performance improvements. Change-Id: I95c560ffcbfa30b26644999412fb6a385125f663
* happy new yearMike Bayer2020-01-011-1/+1
| | | | Change-Id: I08440dc25e40ea1ccea1778f6ee9e28a00808235
* Test for short term reference cycles and resolve as many as possibleMike Bayer2019-12-301-3/+4
| | | | | | | | Added test support and repaired a wide variety of unnecessary reference cycles created for short-lived objects, mostly in the area of ORM queries. Fixes: #5056 Change-Id: Ifd93856eba550483f95f9ae63d49f36ab068b85a
* Add anonymizing context to cache keys, comparison; convert traversalMike Bayer2019-11-041-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Created new visitor system called "internal traversal" that applies a data driven approach to the concept of a class that defines its own traversal steps, in contrast to the existing style of traversal now known as "external traversal" where the visitor class defines the traversal, i.e. the SQLCompiler. The internal traversal system now implements get_children(), _copy_internals(), compare() and _cache_key() for most Core elements. Core elements with special needs like Select still implement some of these methods directly however most of these methods are no longer explicitly implemented. The data-driven system is also applied to ORM elements that take part in SQL expressions so that these objects, like mappers, aliasedclass, query options, etc. can all participate in the cache key process. Still not considered is that this approach to defining traversibility will be used to create some kind of generic introspection system that works across Core / ORM. It's also not clear if real statement caching using the _cache_key() method is feasible, if it is shown that running _cache_key() is nearly as expensive as compiling in any case. Because it is data driven, it is more straightforward to optimize using inlined code, as is the case now, as well as potentially using C code to speed it up. In addition, the caching sytem now accommodates for anonymous name labels, which is essential so that constructs which have anonymous labels can be cacheable, that is, their position within a statement in relation to other anonymous names causes them to generate an integer counter relative to that construct which will be the same every time. Gathering of bound parameters from any cache key generation is also now required as there is no use case for a cache key that does not extract bound parameter values. Applies-to: #4639 Change-Id: I0660584def8627cad566719ee98d3be045db4b8d
* pass executemany context to _repr_paramsMike Bayer2019-10-111-11/+9
| | | | | | | | | | | | | | | | | | Fixed bug where parameter repr as used in logging and error reporting needs additional context in order to distinguish between a list of parameters for a single statement and a list of parameter lists, as the "list of lists" structure could also indicate a single parameter list where the first parameter itself is a list, such as for an array parameter. The engine/connection now passes in an additional boolean indicating how the parameters should be considered. The only SQLAlchemy backend that expects arrays as parameters is that of psycopg2 which uses pyformat parameters, so this issue has not been too apparent, however as other drivers that use positional gain more features it is important that this be supported. It also eliminates the need for the parameter repr function to guess based on the parameter structure passed. Fixes: #4902 Change-Id: I086246ee0eb51484adbefd83e07295fa56576c5f
* Deprecate textual column matching in RowMike Bayer2019-10-041-3/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Deprecate query.instances() without a context Deprecate string alias with contains_eager() Deprecated the behavior by which a :class:`.Column` can be used as the key in a result set row lookup, when that :class:`.Column` is not part of the SQL selectable that is being selected; that is, it is only matched on name. A deprecation warning is now emitted for this case. Various ORM use cases, such as those involving :func:`.text` constructs, have been improved so that this fallback logic is avoided in most cases. Calling the :meth:`.Query.instances` method without passing a :class:`.QueryContext` is deprecated. The original use case for this was that a :class:`.Query` could yield ORM objects when given only the entities to be selected as well as a DBAPI cursor object. However, for this to work correctly there is essential metadata that is passed from a SQLAlchemy :class:`.ResultProxy` that is derived from the mapped column expressions, which comes originally from the :class:`.QueryContext`. To retrieve ORM results from arbitrary SELECT statements, the :meth:`.Query.from_statement` method should be used. Note there is a small bump in test_zoomark because the column._label is being calculated for each of those columns within baseline_3_properties, as it is now part of the result map. This label can't be calculated when the column is attached to the table because it needs to have all the columns present to do this correctly. Another approach here would be to pre-load the _label before the test runs however the zoomark tests don't have an easy place for this to happen and it's not really worth it. Fixes: #4877 Fixes: #4719 Change-Id: I9bd29e72e6dce7c855651d69ba68d7383469acbc
* Cancel polymorphic loading in optimized getMike Bayer2019-10-011-0/+2
| | | | | | | | | | | | | Since optimized_get for inheriting mappers writes a simple SELECT, we need to cancel out any with_polymorphic selectables that interfere with simple column lookup. While adaptation is another option, just removing the with_polymorphic is much simpler. The issue is not noticeable unless the ResultProxy is not allowing "key fallback" column lookups, which will be the case when this behavior is deprecated. Fixes: #4718 Change-Id: I8fa2f5c0434b6a681813a92ac71fe12712f5d634
* Simplify _ColumnEntity, relatedMike Bayer2019-09-301-10/+6
| | | | | | | | | | | | | | | In the interests of making Query much more lightweight up front, rework the calculations done at the top when the entities are constructed to be much less inolved. Use the new coercion system for _ColumnEntity and stop accepting plain strings, this will need to emit a deprecation warning in 1.3.x. Use annotations and other techniques to reduce the decisionmaking and complexity of Query. For the use case of subquery(), .statement, etc. we would like to do minimal work in order to get the columns clause. Change-Id: I7e459bbd3bb10ec71235f75ef4f3b0a969bec590
* Adjustments to _copy_internals()Mike Bayer2019-09-061-4/+19
| | | | | | | | | | | | We are looking to build a generalization of copy_internals(), so move out any special logic from these methods. Re-implement and clarify rationale for the Alias doesnt copy a TableClause rule as part of the adaption traversal, establish that we forgot to build out comparison and cache key for CTE, remove incomplete _copy_internals() from GenerativeSelect (it doesn't handle the order_by_clause or group_by_clause, so is incomplete) Change-Id: I95039f042503171aade4ba0fabc9b1598e3c49cf
* SelectBase no longer a FromClauseMike Bayer2019-07-061-1/+1
| | | | | | | | | | | | | | | | | | | | As part of the SQLAlchemy 2.0 migration project, a conceptual change has been made to the role of the :class:`.SelectBase` class hierarchy, which is the root of all "SELECT" statement constructs, in that they no longer serve directly as FROM clauses, that is, they no longer subclass :class:`.FromClause`. For end users, the change mostly means that any placement of a :func:`.select` construct in the FROM clause of another :func:`.select` requires first that it be wrapped in a subquery first, which historically is through the use of the :meth:`.SelectBase.alias` method, and is now also available through the use of :meth:`.SelectBase.subquery`. This was usually a requirement in any case since several databases don't accept unnamed SELECT subqueries in their FROM clause in any case. See the documentation in this change for lots more detail. Fixes: #4617 Change-Id: I0f6174ee24b9a1a4529168e52e855e12abd60667
* Black updatesMike Bayer2019-04-111-3/+1
| | | | | | Updating for a few black adjustments Change-Id: I9c8abadd1ccd6173e6d68dd62f14cec208cc304a
* Fold entities into existing joins when resolving FROM ambiguityMike Bayer2019-04-021-0/+11
| | | | | | | | | | | | | | Fixed 1.3 regression in new "ambiguous FROMs" query logic introduced in :ref:`change_4365` where a :class:`.Query` that explicitly places an entity in the FROM clause with :meth:`.Query.select_from` and also joins to it using :meth:`.Query.join` would later cause an "ambiguous FROM" error if that entity were used in additional joins, as the entity appears twice in the "from" list of the :class:`.Query`. The fix resolves this ambiguity by folding the standalone entity into the join that it's already a part of in the same way that ultimately happens when the SELECT statement is rendered. Fixes: #4584 Change-Id: Ic62ca09c6d329695b21ed4f1be8621edcbb18e19
* Expand joins when calculating PostgreSQL "WITH FOR UPDATE OF"raylu2019-03-211-0/+16
| | | | | | | | | | | | | | | | Modified the :paramref:`.Select.with_for_update.of` parameter so that if a join or other composed selectable is passed, the individual :class:`.Table` objects will be filtered from it, allowing one to pass a join() object to the parameter, as occurs normally when using joined table inheritance with the ORM. Pull request courtesy Raymond Lu. Fixes: #4550 Co-authored-by: Mike Bayer <mike_mp@zzzcomputing.com> Closes: #4551 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4551 Pull-request-sha: 452da77d154a4087d530456db1c9af207d65cef4 Change-Id: If4b7c231f7b71190d7245543959fb5c3351125a1
* Fix many spell glitchesLele Gaifax2019-01-251-1/+1
| | | | | | | | | | | | This affects mostly docstrings, except in orm/events.py::dispose_collection() where one parameter gets renamed: given that the method is empty, it seemed reasonable to me to fix that too. Closes: #4440 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4440 Pull-request-sha: 779ed75acb6142e1f1daac467b5b14134529bb4b Change-Id: Ic0553fe97853054b09c2453af76d96363de6eb0e
* Cleanup with query aliasingMike Bayer2019-01-181-3/+0
| | | | | | | | | | | | | | | | Try to simplify some of the "adapter" stuff in query: 1. identify that join(.., aliased=True) doesn't work if the right side has no mapper. The adaption of the right side is done via the mapper with aliased(), so that doesn't effect a selectable only. raise an error, so we can simplify the code. 2. build fewer adapter objects. these are confusing to follow and we should try to figure out exactly what purpose which one serves where and make that clear. Change-Id: I18dfcd01e6ad533aa0b8d557fc637ee2766ed050
* happy new yearMike Bayer2019-01-111-1/+1
| | | | Change-Id: I6a71f4924d046cf306961c58dffccf21e9c03911
* Post black reformattingMike Bayer2019-01-061-27/+25
| | | | | | | | | | | | | Applied on top of a pure run of black -l 79 in I7eda77fed3d8e73df84b3651fd6cfcfe858d4dc9, this set of changes resolves all remaining flake8 conditions for those codes we have enabled in setup.cfg. Included are resolutions for all remaining flake8 issues including shadowed builtins, long lines, import order, unused imports, duplicate imports, and docstring issues. Change-Id: I4f72d3ba1380dd601610ff80b8fb06a2aff8b0fe
* Run black -l 79 against all source filesMike Bayer2019-01-061-131/+196
| | | | | | | | | | | | | | This is a straight reformat run using black as is, with no edits applied at all. The black run will format code consistently, however in some cases that are prevalent in SQLAlchemy code it produces too-long lines. The too-long lines will be resolved in the following commit that will resolve all remaining flake8 issues including shadowed builtins, long lines, import order, unused imports, duplicate imports, and docstring issues. Change-Id: I7eda77fed3d8e73df84b3651fd6cfcfe858d4dc9
* Allow join() to pick the best candidate from multiple froms/entitiesMike Bayer2018-11-141-2/+87
| | | | | | | | | | | | | | Refactored :meth:`.Query.join` to further clarify the individual components of structuring the join. This refactor adds the ability for :meth:`.Query.join` to determine the most appropriate "left" side of the join when there is more than one element in the FROM list or the query is against multiple entities. In particular this targets the regression we saw in :ticket:`4363` but is also of general use. The codepaths within :meth:`.Query.join` are now easier to follow and the error cases are decided more specifically at an earlier point in the operation. Fixes: #4365 Change-Id: I403f451243904a020ceab4c3f94bead550c7b2d5
* happy new yearMike Bayer2018-01-121-1/+1
| | | | Change-Id: I3ef36bfd0cb0ba62b3123c8cf92370a43156cf8f
* Don't include SelectBase when searching for surface column elementsMike Bayer2017-10-051-3/+7
| | | | | | | | | | Fixed bug where correlated select used against single-table inheritance entity would fail to render correctly in the outer query, due to adjustment for single inheritance discriminator criteria inappropriately re-applying the criteria to the outer query. Change-Id: I38df21f1392af1843e10119682fa0635d346e2a8 Fixes: #4103
* Support python3.6Mike Bayer2017-01-131-1/+1
| | | | | | | | | | | Corrects some warnings and adds tox config. Adds DeprecationWarning to the error category. Large sweep for string literals w/ backslashes as this is common in docstrings Co-authored-by: Andrii Soldatenko Fixes: #3886 Change-Id: Ia7c838dfbbe70b262622ed0803d581edc736e085 Pull-request: https://github.com/zzzeek/sqlalchemy/pull/337
* update for 2017 copyrightMike Bayer2017-01-041-1/+1
| | | | Change-Id: I4e8c2aa8fe817bb2af8707410fa0201f938781de
* Fix label referencing in SQL Server OFFSET logicMike Bayer2016-05-101-0/+10
| | | | | | | | | | Fixed bug where by ROW_NUMBER OVER clause applied for OFFSET selects in SQL Server would inappropriately substitute a plain column from the local statement that overlaps with a label name used by the ORDER BY criteria of the statement. Change-Id: Ic2500c886cbfc83a1ad5a2681783f008b9f23838 Fixes: #3711
* - handle parameter sets that aren't correctly formed, so thatMike Bayer2016-02-171-1/+1
| | | | | for example an exception object made within a test suite can still repr (error seen in Keystone)
* - do the trailing comma logic of tuple repr() exactlyMike Bayer2016-02-171-4/+7
|
* - All string formatting of bound parameter sets and result rows forMike Bayer2016-02-171-10/+107
| | | | | | | | | logging, exception, and ``repr()`` purposes now truncate very large scalar values within each collection, including an "N characters truncated" notation, similar to how the display for large multiple-parameter sets are themselves truncated. fixes #2837
* - A refinement to the logic which adds columns to the resulting SQL whenMike Bayer2016-02-091-0/+22
| | | | | | | | | | | | | :meth:`.Query.distinct` is combined with :meth:`.Query.order_by` such that columns which are already present will not be added a second time, even if they are labeled with a different name. Regardless of this change, the extra columns added to the SQL have never been returned in the final result, so this change only impacts the string form of the statement as well as its behavior when used in a Core execution context. Additionally, columns are no longer added when the DISTINCT ON format is used, provided the query is not wrapped inside a subquery due to joined eager loading. fixes #3641
* - happy new yearMike Bayer2016-01-291-1/+1
|
* - Added support for parameter-ordered SET clauses in an UPDATEMike Bayer2015-11-281-1/+0
| | | | | | | | | | statement. This feature is available by passing the :paramref:`~.sqlalchemy.sql.expression.update.preserve_parameter_order` flag either to the core :class:`.Update` construct or alternatively adding it to the :paramref:`.Query.update.update_args` dictionary at the ORM-level, also passing the parameters themselves as a list of 2-tuples. Thanks to Gorka Eguileor for implementation and tests. adapted from pullreq github:200
* - correct the commit from ref #3582 to refer to the correct sub-elementMike Bayer2015-11-111-1/+1
|
* - Fixed bug where the "single table inheritance" criteria would beMike Bayer2015-11-111-0/+15
| | | | | | | added onto the end of a query in some inappropriate situations, such as when querying from an exists() of a single-inheritance subclass. fixes #3582
* - Added support for "set-aggregate" functions of the formticket_3516Mike Bayer2015-08-261-2/+5
| | | | | | | | | | | ``<function> WITHIN GROUP (ORDER BY <criteria>)``, using the method :class:`.FunctionElement.within_group`. A series of common set-aggregate functions with return types derived from the set have been added. This includes functions like :class:`.percentile_cont`, :class:`.dense_rank` and others. fixes #1370 - make sure we use func.name for all _literal_as_binds in functions.py so we get consistent naming behavior for parameters.
* - altered part of the use contract first set up in #2992; weMike Bayer2015-04-271-1/+4
| | | | | | | | now skip textual label references when copying ORDER BY elements to the joined-eager-load subquery, as we can't know that these expressions are compatible with this placement; either because they are meant for text(), or because they refer to label names already stated and aren't bound to a table. fixes #3392
* - copyright 2015Mike Bayer2015-03-101-1/+1
|
* - rework the previous "order by" system in terms of the new one,Mike Bayer2014-09-081-1/+3
| | | | | | | | unify everything. - create a new layer of separation between the "from order bys" and "column order bys", so that an OVER doesn't ORDER BY a label in the same columns clause - identify another issue with polymorphic for ref #3148, match on label keys rather than the objects
* - rework ColumnAdapter and ORMAdapter to only provide the featuresticket_3148Mike Bayer2014-09-071-76/+66
| | | | | | | | | | | we're now using; rework them fully so that their behavioral contract is consistent regarding adapter.traverse() vs. adapter.columns[], add a full suite of tests including advanced wrapping scenarios previously only covered by test/orm/test_froms.py and test/orm/inheritance/test_relationships.py - identify several cases where label._order_by_label_clause would be corrupted, e.g. due to adaption or annotation separately - add full tests for #3148
* - enhance ClauseAdapter / ColumnAdapter to have new behaviors with labels.Mike Bayer2014-09-071-10/+18
| | | | | | | | | | | | | | | | | | | | | The "anonymize label" logic is now generalized to ClauseAdapter, and takes place when the anonymize_labels flag is sent, taking effect for all .columns lookups as well as within traverse() calls against the label directly. - traverse() will also memoize what it gets in columns, so that calling upon traverse() / .columns against the same Label will produce the same anonymized label. This is so that AliasedClass produces the same anonymized label when it is accessed per-column (e.g. SomeAlias.some_column) as well as when it is applied to a Query, and within column loader strategies (e.g. query(SomeAlias)); the former uses traverse() while the latter uses .columns - AliasedClass now calls onto ColumnAdapter - Query also makes sure to use that same ColumnAdapter from the AliasedClass in all cases - update the logic from 0.9 in #1068 to make use of the same _label_resolve_dict we use for #2992, simplifying how that works and adding support for new scenarios that were pretty broken (see #3148, #3188)
* wip for #3148Mike Bayer2014-09-061-1/+5
|
* - apply pep8 formatting to sqlalchemy/sql, sqlalchemy/util, sqlalchemy/dialects,Brian Jarrett2014-07-201-56/+60
| | | | sqlalchemy/orm, sqlalchemy/event, sqlalchemy/testing
* - break up the <authors> copyright comment as part of a passMike Bayer2014-07-091-1/+2
| | | | to get all flake8 passing
* Fix many typos throughout the codebasepr/85Alex Gaynor2014-04-261-2/+2
| | | | Found using: https://github.com/intgr/topy
* - happy new yearMike Bayer2014-01-051-1/+1
|
View Lorry Controller Status