summaryrefslogtreecommitdiff
path: root/lib/sqlalchemy/sql
Commit message (Collapse)AuthorAgeFilesLines
...
* happy new yearMike Bayer2021-01-0423-23/+23
| | | | Change-Id: Ic5bb19ca8be3cb47c95a0d3315d84cb484bac47c
* Support TypeDecorator.get_dbapi_type() for setinpusizesMike Bayer2020-12-301-3/+8
| | | | | | | | Adjusted the "setinputsizes" logic relied upon by the cx_Oracle, asyncpg and pg8000 dialects to support a :class:`.TypeDecorator` that includes an override the :meth:`.TypeDecorator.get_dbapi_type()` method. Change-Id: I5aa70abf0d9a9e2ca43309f2dd80b3fcd83881b9
* Fix issues with JSON and float/numericGord Thompson2020-12-201-1/+19
| | | | | | | | | | | | | | | | | | | Decimal accuracy and behavior has been improved when extracting floating point and/or decimal values from JSON strings using the :meth:`_sql.sqltypes.JSON.Comparator.as_float` method, when the numeric value inside of the JSON string has many significant digits; previously, MySQL backends would truncate values with many significant digits and SQL Server backends would raise an exception due to a DECIMAL cast with insufficient significant digits. Both backends now use a FLOAT-compatible approach that does not hardcode significant digits for floating point values. For precision numerics, a new method :meth:`_sql.sqltypes.JSON.Comparator.as_numeric` has been added which accepts arguments for precision and scale, and will return values as Python ``Decimal`` objects with no floating point conversion assuming the DBAPI supports it (all but pysqlite). Fixes: #5788 Change-Id: I6eb51fe172a389548dd6e3c65efec9f1f538012e
* Improve type detection for Values / TupleMike Bayer2020-12-184-6/+32
| | | | | | | | | | | | | | | | | Fixed issue in new :class:`_sql.Values` construct where passing tuples of objects would fall back to per-value type detection rather than making use of the :class:`_schema.Column` objects passed directly to :class:`_sql.Values` that tells SQLAlchemy what the expected type is. This would lead to issues for objects such as enumerations and numpy strings that are not actually necessary since the expected type is given. note this changes NullType() to raise CompileError for literal_processor; NullType() does not imply the actual value NULL as much as it does "unknown type" so this should make failure modes more clear. Fixes: #5785 Change-Id: Ifbf5e78373102380b301098f30e15011efa98b5e
* Major revisals to lambdasMike Bayer2020-12-163-149/+322
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 1. Improve coercions._deep_is_literal to check sequences for clause elements, thus allowing a phrase like lambda: col.in_([literal("x"), literal("y")]) to be handled 2. revise closure variable caching completely. All variables entering must be part of a closure cache key or rejected. only objects that can be resolved to HasCacheKey or FunctionType are accepted; all other types are rejected. This adds a high degree of strictness to lambdas and will make them a little more awkward to use in some cases, however prevents several classes of critical issues: a. previously, a lambda that had an expression derived from some kind of state, like "self.x", or "execution_context.session.foo" would produce a closure cache key from "self" or "execution_context", objects that can very well be per-execution and would therefore cause a AnalyzedFunction objects to overflow. (memory won't leak as it looks like an LRUCache is already used for these) b. a lambda, such as one used within DeferredLamdaElement, that produces different SQL expressions based on the arguments (which is in fact what it's supposed to do), however it would through the use of conditionals produce different bound parameter combinations, leading to literal parameters not tracked properly. These are now rejected as uncacheable whereas previously they would again be part of the closure cache key, causing an overflow of AnalyizedFunction objects. 3. Ensure non-mapped mixins are handled correctly by with_loader_criteria(). 4. Fixed bug in lambda SQL system where we are not supposed to allow a Python function to be embedded in the lambda, since we can't predict a bound value from it. While there was an error condition added for this, it was not tested and wasn't working; an informative error is now raised. 5. new docs for lambdas 6. consolidated changelog for all of these Fixes: #5760 Fixes: #5765 Fixes: #5766 Fixes: #5768 Fixes: #5770 Change-Id: Iedaa636c3225fad496df23b612c516c8ab247ab7
* Merge "Emit deprecation warnings for plain text under Session"mike bayer2020-12-141-17/+10
|\
| * Emit deprecation warnings for plain text under SessionMike Bayer2020-12-111-17/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Deprecation warnings are emitted under "SQLALCHEMY_WARN_20" mode when passing a plain string to :meth:`_orm.Session.execute`. It was also considered to have DDL string expressions to include this as well, however this leaves us with no backwards-compatible way of handling reflection of elemens, such as an Index() which reflects "postgresql_where='x > 5'", there's no place for a rule that will turn those into text() within the reflection process that would be separate from when the user passes postgresql_where to the Index. Not worth it right now. Fixes: #5754 Change-Id: I8673a79f0e87de0df576b655f39dad0351725ca8
* | Emit deprecation warning for general DDLElement.bind argumentMike Bayer2020-12-141-7/+54
| | | | | | | | Change-Id: I8ab1fd98340cb30aa43075508b3a0b9feffa290c
* | Merge "Support IF EXISTS/IF NOT EXISTS for DDL constructs"mike bayer2020-12-142-9/+90
|\ \
| * | Support IF EXISTS/IF NOT EXISTS for DDL constructsRamonWill2020-12-142-9/+90
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Added parameters :paramref:`_ddl.CreateTable.if_not_exists`, :paramref:`_ddl.CreateIndex.if_not_exists`, :paramref:`_ddl.DropTable.if_exists` and :paramref:`_ddl.DropIndex.if_exists` to the :class:`_ddl.CreateTable`, :class:`_ddl.DropTable`, :class:`_ddl.CreateIndex` and :class:`_ddl.DropIndex` constructs which result in "IF NOT EXISTS" / "IF EXISTS" DDL being added to the CREATE/DROP. These phrases are not accepted by all databases and the operation will fail on a database that does not support it as there is no similarly compatible fallback within the scope of a single DDL statement. Pull request courtesy Ramon Williams. Fixes: #2843 Closes: #5663 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5663 Pull-request-sha: 748b8472345d96efb446e2a444fbe020b313669f Change-Id: I6a2b1f697993ed49c31584f0a31887fb0a868ed3
* | | Merge "Fixes for lambda expressions and relationship loaders"mike bayer2020-12-122-1/+26
|\ \ \
| * | | Fixes for lambda expressions and relationship loadersMike Bayer2020-12-112-1/+26
| | |/ | |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixed bug in lambda SQL feature, used by ORM :meth:`_orm.with_loader_criteria` as well as available generally in the SQL expression language, where assigning a boolean value True/False to a variable would cause the query-time expression calculation to fail, as it would produce a SQL expression not compatible with a bound value. Fixed issue where the :attr:`_orm.ORMExecuteState.is_relationship_load` parameter would not be set correctly for many lazy loads, all selectinloads, etc. The flag is essential in order to test if options should be added to statements or if they would already have been propagated via relationship loads. Fixes: #5763 Fixes: #5764 Change-Id: I66aafbef193f892ff75ede0670698647b7475482
* | | Merge "Send deterministic ordering into unit of work topological"mike bayer2020-12-121-2/+0
|\ \ \ | |/ / |/| |
| * | Send deterministic ordering into unit of work topologicalMike Bayer2020-12-111-2/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Improved the unit of work topological sorting system such that the toplogical sort is now deterministic based on the sorting of the input set, which itself is now sorted at the level of mappers, so that the same inputs of affected mappers should produce the same output every time, among mappers / tables that don't have any dependency on each other. This further reduces the chance of deadlocks as can be observed in a flush that UPDATEs among multiple, unrelated tables such that row locks are generated. topological.sort() has been made "deterministic" in all cases by using a separate list + set. Fixes: #5735 Change-Id: I073103df414dba549e46605b394f8ccae6e80d0e
* | | Merge "correct for "autocommit" deprecation warning"mike bayer2020-12-111-2/+2
|\ \ \
| * | | correct for "autocommit" deprecation warningMike Bayer2020-12-111-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Ensure no autocommit warnings occur internally or within tests. Also includes fixes for SQL Server full text tests which apparently have not been working at all for a long time, as it used long removed APIs. CI has not had fulltext running for some years and is now installed. Change-Id: Id806e1856c9da9f0a9eac88cebc7a94ecc95eb96
* | | | Implement `TypeEngine.as_generic`Gord Thompson2020-12-083-31/+85
|/ / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Added :meth:`_types.TypeEngine.as_generic` to map dialect-specific types, such as :class:`sqlalchemy.dialects.mysql.INTEGER`, with the "best match" generic SQLAlchemy type, in this case :class:`_types.Integer`. Pull request courtesy Andrew Hannigan. Abstract away how we check for "overridden methods" so it is more clear what the intent is and that the methodology can be independently tested. Fixes: #5659 Closes: #5714 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5714 Pull-request-sha: 91afb9a0ba3bfa81a1ded80c025989213cf6e4eb Change-Id: Ic54d6690ecc10dc69e6e72856d5620036cea472a
* | | Properly render ``cycle=False`` and ``order=False``Federico Caselli2020-12-011-3/+3
| | | | | | | | | | | | | | | | | | | | | | | | These get rendered as ``NO CYCLE`` and ``NO ORDER`` in :class:`_sql.Sequence` and :class:`_sql.Identity` objects. Fixes: #5738 Change-Id: Ia9ccb5481a104cb32d3b517e99efd5e730c84946
* | | Merge "improve cross-linking between Core /ORM for schema arg"mike bayer2020-12-011-41/+3
|\ \ \ | |/ / |/| |
| * | improve cross-linking between Core /ORM for schema argMike Bayer2020-11-301-41/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | this should be backported to 1.3 as well to as much a degree as possible. Includes a new recipe to set the default schema name on connect. this will only work on 1.4, but also requires that we fix #5708 for it to work fully. Change-Id: I882edd5bbe06ee5b4d0a9c148854a57b2bcd4741
* | | Don't discard leftovers from surface_selectablesMike Bayer2020-11-261-1/+6
|/ / | | | | | | | | | | | | | | | | | | | | Fixed regression introduced in 1.3.2 for the PostgreSQL dialect, also copied out to the MySQL dialect's feature in 1.3.18, where usage of a non :class:`_schema.Table` construct such as :func:`_sql.text` as the argument to :paramref:`_sql.Select.with_for_update.of` would fail to be accommodated correctly within the PostgreSQL or MySQL compilers. Fixes: #5729 Change-Id: I265bcc171f0eb865ac3910ee805b162f3b70e2c1
* | Don't emit warnings on descriptor accessMike Bayer2020-11-202-0/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This commit is revising 5162f2bc5fc0ac239f26a76fc9f0c2, which when I did it felt a little rushed but I couldn't find anything wrong. Well here we are :). Fixed issue where a :class:`.RemovedIn20Warning` would erroneously emit when the ``.bind`` attribute were accessed internally on objects, particularly when stringifying a SQL construct. Alter the deprecated() decorator so that we can use it just to add docstring warnings but not actually warn when the function is accessed, adding new argument enable_warnings that can be set to False. Added a safety feature to deprecated_20() that will disallow an ":attr:" from proceeding if enable_warnings=False isn't present, unless there's an extra flag warn_on_attribute_access, since we want Session.transaction to emit a deprecation warning. This is a little hacky but it's essentially modifying the decorator to require a positive assertion that a deprecation decorator on a descriptor should actually warn on access. Remove the warning filter for session.transaction and get tests to pass to ensure this is not also being called internally. Added tests to ensure that common places .bind can be passed as a parameter definitely warn as I was not able to find this otherwise. Fixes: #5717 Change-Id: Ia586b4f9ee6b212f3a71104b1caf40b5edd399e2
* | Allow MetaData as the target for column_reflect eventMike Bayer2020-11-182-13/+43
| | | | | | | | | | | | | | | | | | The :meth:`_event.DDLEvents.column_reflect` event may now be applied to a :class:`_schema.MetaData` object where it will take effect for the :class:`_schema.Table` objects local to that collection. Fixes: #5712 Change-Id: I6044baa72d096ebd1fd99128270119747d1461b9
* | improve column_reflect docsMike Bayer2020-11-181-40/+44
| | | | | | | | | | Change-Id: I41d947e3cccb68c2d1ca811b5b2ae4b3f66409fb References: #5711
* | Data type is supported only on sequences, not indentityFederico Caselli2020-11-171-7/+4
| | | | | | | | Change-Id: I16ed79c008ccbb25778426a261e87695e99964c3
* | Ensure the "orm" plugin is used unconditionally for bundlesMike Bayer2020-11-161-1/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | This also introduces that the "orm" plugin may be used when the plugin_subject is None. Fixed regression where the :paramref:`.Bundle.single_entity` flag would take effect for a :class:`.Bundle` even though it were not set. Additionally, this flag is legacy as it only makes sense for the :class:`_orm.Query` object and not 2.0 style execution. a deprecation warning is emitted when used with new-style execution. Fixes: #5702 Change-Id: If9f43365b9d966cb398890aeba2efa555658a7e5
* | Some small improvements on the tutorial 2.0 documentsFederico Caselli2020-11-132-2/+2
| | | | | | | | Change-Id: I7fb37d45c29307b2213bebd0ef280d73804ac473
* | Allow multiple returning() callsMike Bayer2020-11-111-29/+46
| | | | | | | | | | | | | | | | Multiple calls to "returning", e.g. :meth:`_sql.Insert.returning`, may now be chained to add new columns to the RETURNING clause. Fixes: #5695 Change-Id: Ie2dac4162f686c730e000e31dccfb38f9ce9c96e
* | Warn / raise for returning() / return_defaults() combinationsMike Bayer2020-11-111-5/+20
| | | | | | | | | | | | | | | | | | | | | | | | | | A warning is emmitted if a returning() method such as :meth:`_sql.Insert.returning` is called multiple times, as this does not yet support additive operation. Version 1.4 will support additive operation for this. Additionally, any combination of the :meth:`_sql.Insert.returning` and :meth:`_sql.Insert.return_defaults` methods now raises an error as these methods are mutually exclusive; previously the operation would fail silently. Fixes: #5691 Change-Id: Id95e0f9da48bba0b59439cb26564f0daa684c8e3
* | Remove misleading correlation examplesMike Bayer2020-11-111-42/+36
|/ | | | | | | add links to tutorial docs Fixes: #5694 Change-Id: I10a8e3f46a115945ded36d4ee59165c056c10f7a
* Implement upsert for SQLiteRamonWill2020-11-081-4/+5
| | | | | | | | | | | | Implemented INSERT... ON CONFLICT clause for SQLite. Pull request courtesy Ramon Williams. Fixes: #4010 Closes: #5580 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5580 Pull-request-sha: fb422e0749fac442a455cbce539ef662d9512bc0 Change-Id: Ibeea44f4c2cee8dab5dc22b7ec3ae1ab95c12b65
* Merge "Allow dialect-specific stringification"mike bayer2020-11-083-2/+18
|\
| * Allow dialect-specific stringificationMike Bayer2020-11-083-2/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Dialect-specific constructs such as :meth:`_postgresql.Insert.on_conflict_do_update` can now stringify in-place without the need to specify an explicit dialect object. The constructs, when called upon for ``str()``, ``print()``, etc. now have internal direction to call upon their appropriate dialect rather than the "default"dialect which doesn't know how to stringify these. The approach is also adapted to generic schema-level create/drop such as :class:`_schema.AddConstraint`, which will adapt its stringify dialect to one indicated by the element within it, such as the :class:`_postgresql.ExcludeConstraint` object. mostly towards being able to provide doctest-style examples for "on conflict" constructs using print statements. Change-Id: I4b855516fe6dee2df77744c1bb21a373d7fbab93
* | Convert to autoload_with internallyMike Bayer2020-11-071-1/+3
|/ | | | | | | | | Fixed bug where the now-deprecated ``autoload`` parameter was being called internally within the reflection routines when a related table were reflected. Fixes: #5684 Change-Id: I6ab439a2f49ff1ae2d3c7a15b531cbafbc3cf594
* Reduce import time overheadMike Bayer2020-11-034-33/+26
| | | | | | | | | | | | | | | | | | * Fix subclass traversals to not run classes multiple times * switch compiler visitor to use an attrgetter, to avoid an eval() at startup time * don't pre-generate traversal functions, there's lots of these which are expensive to generate at once and most applications won't use them all; have it generate them on first use instead * Some ideas about removing asyncio imports, they don't seem to be too signficant, apply some more simplicity to the overall "greenlet fallback" situation Fixes: #5681 Change-Id: Ib564ddaddb374787ce3e11ff48026e99ed570933
* Merge "tutorial 2.0 WIP"mike bayer2020-10-315-81/+329
|\
| * tutorial 2.0 WIPreview/mike_bayer/tutorial20Mike Bayer2020-10-315-81/+329
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Add SelectBase.exists() method as it seems strange this is not available already. The Exists construct itself does not provide full SELECT-building capabilities so it makes sense this should be used more like a scalar_subquery. Make sure stream_results is getting set up when yield_per is used, for 2.0 style statements as well. this was hardcoded inside of Query.yield_per() and is now moved to take place within QueryContext. Change-Id: Icafcd4fd9b708772343d56edf40995c9e8f835d6
* | Merge "while working on #5435, I found some misses from the previous PR for ↵mike bayer2020-10-311-2/+2
|\ \ | | | | | | | | | #5429"
| * | while working on #5435, I found some misses from the previous PR for #5429jonathan vanasco2020-10-301-2/+2
| |/ | | | | | | Change-Id: I0be15f6234c74302734672450a3275add762bdb8
* | Merge "Deprecate bind args, execute() methods that were missed"mike bayer2020-10-316-7/+52
|\ \ | |/ |/|
| * Deprecate bind args, execute() methods that were missedMike Bayer2020-10-306-7/+52
| | | | | | | | | | | | in particular text(bind), DDL.execute(). Change-Id: Ie85ae9f61219182f5649f68e5f52b4923843199c
* | Apply underscore naming to several more operatorsjonathan vanasco2020-10-306-106/+205
|/ | | | | | | | | | | | | | | | | | | | | | | | | | The operator changes are: * `isfalse` is now `is_false` * `isnot_distinct_from` is now `is_not_distinct_from` * `istrue` is now `is_true` * `notbetween` is now `not_between` * `notcontains` is now `not_contains` * `notendswith` is now `not_endswith` * `notilike` is now `not_ilike` * `notlike` is now `not_like` * `notmatch` is now `not_match` * `notstartswith` is now `not_startswith` * `nullsfirst` is now `nulls_first` * `nullslast` is now `nulls_last` Because these are core operators, the internal migration strategy for this change is to support legacy terms for an extended period of time -- if not indefinitely -- but update all documentation, tutorials, and internal usage to the new terms. The new terms are used to define the functions, and the legacy terms have been deprecated into aliases of the new terms. Fixes: #5435 Change-Id: Ifbd7cb1cdda5981990243c4fc4b4ff467dc132ac
* Ensure no compiler visit method tries to access .statementMike Bayer2020-10-191-1/+42
| | | | | | | | | | | | Fixed structural compiler issue where some constructs such as MySQL / PostgreSQL "on conflict / on duplicate key" would rely upon the state of the :class:`_sql.Compiler` object being fixed against their statement as the top level statement, which would fail in cases where those statements are branched from a different context, such as a DDL construct linked to a SQL statement. Fixes: #5656 Change-Id: I568bf40adc7edcf72ea6c7fd6eb9d07790de189e
* Ensure escaping of percent signs in columns, parametersMike Bayer2020-10-173-32/+77
| | | | | | | | | | | | | | | | | | | | | | | | Improved support for column names that contain percent signs in the string, including repaired issues involving anoymous labels that also embedded a column name with a percent sign in it, as well as re-established support for bound parameter names with percent signs embedded on the psycopg2 dialect, using a late-escaping process similar to that used by the cx_Oracle dialect. * Added new constructor for _anonymous_label() that ensures incoming string tokens based on column or table names will have percent signs escaped; abstracts away the format of the label. * generalized cx_Oracle's quoted_bind_names facility into the compiler itself, and leveraged this for the psycopg2 dialect's issue with percent signs in names as well. the parameter substitution is now integrated with compiler.construct_parameters() as well as the recently reworked set_input_sizes(), reducing verbosity in the cx_Oracle dialect. Fixes: #5653 Change-Id: Ia2ad13ea68b4b0558d410026e5a33f5cb3fbab2c
* Merge "Add deprecation for base Executable.bind"mike bayer2020-10-172-0/+27
|\
| * Add deprecation for base Executable.bindMike Bayer2020-10-162-0/+27
| | | | | | | | | | | | | | | | | | | | These attributes will be removed in SQLAlchemy 2.0. Also alters the deprecation message to qualify the type of object correctly. this in turn requires changes in the warnings filter and deprecation tests. Change-Id: I5779d9813e88f42e5db0c7b5e3ffff1d1535c203
* | Genericize setinputsizes and support pyodbcMike Bayer2020-10-161-4/+4
|/ | | | | | | | | | | | Reworked the "setinputsizes()" set of dialect hooks to be correctly extensible for any arbirary DBAPI, by allowing dialects individual hooks that may invoke cursor.setinputsizes() in the appropriate style for that DBAPI. In particular this is intended to support pyodbc's style of usage which is fundamentally different from that of cx_Oracle. Added support for pyodbc. Fixes: #5649 Change-Id: I9f1794f8368bf3663a286932cfe3992dae244a10
* Merge "Deprecate strings indicating attribute names"mike bayer2020-10-141-2/+36
|\
| * Deprecate strings indicating attribute namesMike Bayer2020-10-131-2/+36
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Using strings to represent relationship names in ORM operations such as :meth:`_orm.Query.join`, as well as strings for all ORM attribute names in loader options like :func:`_orm.selectinload` is deprecated and will be removed in SQLAlchemy 2.0. The class-bound attribute should be passed instead. This provides much better specificity to the given method, allows for modifiers such as ``of_type()``, and reduces internal complexity. Additionally, the ``aliased`` and ``from_joinpoint`` parameters to :meth:`_orm.Query.join` are also deprecated. The :func:`_orm.aliased` construct now provides for a great deal of flexibility and capability and should be used directly. Fixes: #4705 Fixes: #5202 Change-Id: I32f61663d68026154906932913c288f269991adc
* | Merge "Deprecate bound metadata"mike bayer2020-10-122-2/+36
|\ \
View Lorry Controller Status