summaryrefslogtreecommitdiff
path: root/lib/sqlalchemy/sql/crud.py
Commit message (Collapse)AuthorAgeFilesLines
* ORM executemany returningMike Bayer2020-06-271-0/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Build on #5401 to allow the ORM to take advanage of executemany INSERT + RETURNING. Implemented the feature updated tests to support INSERT DEFAULT VALUES, needed to come up with a new syntax for compiler INSERT INTO table (anycol) VALUES (DEFAULT) which can then be iterated out for executemany. Added graceful degrade to plain executemany for PostgreSQL <= 8.2 Renamed EXECUTEMANY_DEFAULT to EXECUTEMANY_PLAIN Fix issue where unicode identifiers or parameter names wouldn't work with execute_values() under Py2K, because we have to encode the statement and therefore have to encode the insert_single_values_expr too. Correct issue from #5401 to support executemany + return_defaults for a PK that is explicitly pre-generated, meaning we aren't actually getting RETURNING but need to return it from compiled_parameters. Fixes: #5263 Change-Id: Id68e5c158c4f9ebc33b61c06a448907921c2a657
* Default psycopg2 executemany mode to "values_only"Mike Bayer2020-06-251-1/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The psycopg2 dialect now defaults to using the very performant ``execute_values()`` psycopg2 extension for compiled INSERT statements, and also impements RETURNING support when this extension is used. This allows INSERT statements that even include an autoincremented SERIAL or IDENTITY value to run very fast while still being able to return the newly generated primary key values. The ORM will then integrate this new feature in a separate change. Implements RETURNING for insert with executemany Adds support to return_defaults() mode and inserted_primary_key to support mutiple INSERTed rows, via return_defauls_rows and inserted_primary_key_rows accessors. within default execution context, new cached compiler getters are used to fetch primary keys from rows inserted_primary_key now returns a plain tuple. this is not yet a row-like object however this can be added. Adds distinct "values_only" and "batch" modes, as "values" has a lot of benefits but "batch" breaks cursor.rowcount psycopg2 minimum version 2.7 so we can remove the large number of checks for very old versions of psycopg2 simplify tests to no longer distinguish between native and non-native json Fixes: #5401 Change-Id: Ic08fd3423d4c5d16ca50994460c0c234868bd61c
* Documentation updates for ResultProxy -> ResultMike Bayer2020-05-011-1/+1
| | | | | | | | | This is based off of I8091919d45421e3f53029b8660427f844fee0228 and includes all documentation-only changes as a separate merge, once the parent is merged. Change-Id: I711adea23df0f9f0b1fe7c76210bd2de6d31842d
* Run search and replace of symbolic module namesMike Bayer2020-04-141-3/+3
| | | | | | | | Replaces a wide array of Sphinx-relative doc references with an abbreviated absolute form now supported by zzzeeksphinx. Change-Id: I94bffcc3f37885ffdde6238767224296339698a2
* Try to measure new style caching in the ORM, take twoMike Bayer2020-04-011-5/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Supercedes: If78fbb557c6f2cae637799c3fec2cbc5ac248aaf Trying to see if by making the cache key memoized, we still can have the older "identity" form of caching which is the cheapest of all, at the same time as the newer "cache key each time" version that is not nearly as cheap; but still much cheaper than no caching at all. Also needed is a per-execution update of _keymap when we invoke from a cached select, so that Column objects that are anonymous or otherwise adapted will match up. this is analogous to the adaption of bound parameters from the cache key. Adds test coverage for the keymap / construct_params() changes related to caching. Also hones performance to a large extent for statement construction and cache key generation. Also includes a new memoized attribute approach that vastly simplifies the previous approach of "group_expirable_memoized_property" and finally integrates cleanly with _clone(), _generate(), etc. no more hardcoding of attributes is needed, as well as that most _reset_memoization() calls are no longer needed as the reset is inherent in a _generate() call; this also has dramatic performance improvements. Change-Id: I95c560ffcbfa30b26644999412fb6a385125f663
* Decouple compiler state from DML objects; make cacheableMike Bayer2020-03-061-78/+103
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Targeting select / insert / update / delete, the goal is to minimize overhead of construction and generative methods so that only the raw arguments passed are handled. An interim stage that converts the raw state into more compiler-ready state is added, which is analogous to the ORM QueryContext which will also be rolled in to be a similar concept, as is currently being prototyped in I19e05b3424b07114cce6c439b05198ac47f7ac10. the ORM update/delete BulkUD concept is also going to be rolled onto this idea. So while the compiler-ready state object, here called DMLState, looks a little thin, it's the base of a bigger pattern that will allow for ORM functionality to embed itself directly into the compiler, execution context, and result set objects. This change targets the DML objects, primarily focused on the values() method which is the most complex process. The work done by values() is minimized as much as possible while still being able to create a cache key. Additional computation is then offloaded to a new object ValuesState that is handled by the compiler. Architecturally, a big change here is that insert.values() and update.values() will generate BindParameter objects for the values now, which are then carefully received by crud.py so that they generate the expected names. This is so that the values() portion of these constructs is cacheable. for the "multi-values" version of Insert, this is all skipped and the plan right now is that a multi-values insert is not worth caching (can always be revisited). Using the coercions system in values() also gets us nicer validation for free, we can remove the NotAClauseElement thing from schema, and we also now require scalar_subquery() is called for an insert/update that uses a SELECT as a column value, 1.x deprecation path is added. The traversal system is then applied to the DML objects including tests so that they have traversal, cloning, and cache key support. cloning is not a use case for DML however having it present allows better validation of the structure within the tests. Special per-dialect DML is explicitly not cacheable at the moment, more as a proof of concept that third party DML constructs can exist as gracefully not-cacheable rather than producing an incomplete cache key. A few selected performance improvements have been added as well, simplifying the immutabledict.union() method and adding a new SQLCompiler function that can generate delimeter-separated clauses like WHERE and ORDER BY without having to build a ClauseList object at all. The use of ClauseList will be removed from Select in an upcoming commit. Overall, ClaustList is unnecessary for internal use and only adds overhead to statement construction and will likely be removed as much as possible except for explcit use of conjunctions like and_() and or_(). Change-Id: I408e0b8be91fddd77cf279da97f55020871f75a9
* Restore crud flags if visiting_cte is setMike Bayer2020-03-031-2/+4
| | | | | | | | | | Fixed bug where a CTE of an INSERT/UPDATE/DELETE that also uses RETURNING could then not be SELECTed from directly, as the internal state of the compiler would try to treat the outer SELECT as a DELETE statement itself and access nonexistent state. Fixes: #5181 Change-Id: Icba76f2148c8344baa1c04bac4ab6c6d24f23072
* happy new yearMike Bayer2020-01-011-1/+1
| | | | Change-Id: I08440dc25e40ea1ccea1778f6ee9e28a00808235
* Add sequence support for MariaDB 10.3+.Gord Thompson2019-11-261-2/+10
| | | | | | | | | | | | | | | | | | | | Added support for use of the :class:`.Sequence` construct with MariaDB 10.3 and greater, as this is now supported by this database. The construct integrates with the :class:`.Table` object in the same way that it does for other databases like PostrgreSQL and Oracle; if is present on the integer primary key "autoincrement" column, it is used to generate defaults. For backwards compatibility, to support a :class:`.Table` that has a :class:`.Sequence` on it to support sequence only databases like Oracle, while still not having the sequence fire off for MariaDB, the optional=True flag should be set, which indicates the sequence should only be used to generate the primary key if the target database offers no other option. Fixes: #4976 Closes: #4996 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/4996 Pull-request-sha: cb2e1426ea0b6bc6c93dbe8f033a11df9d8c4915 Change-Id: I507bc405eee6cae2c5991345d0eac53a37fe7512
* Implement new ClauseElement role and coercion systemMike Bayer2019-05-181-10/+17
| | | | | | | | | | | | | | | | | | | | A major refactoring of all the functions handle all detection of Core argument types as well as perform coercions into a new class hierarchy based on "roles", each of which identify a syntactical location within a SQL statement. In contrast to the ClauseElement hierarchy that identifies "what" each object is syntactically, the SQLRole hierarchy identifies the "where does it go" of each object syntactically. From this we define a consistent type checking and coercion system that establishes well defined behviors. This is a breakout of the patch that is reorganizing select() constructs to no longer be in the FromClause hierarchy. Also includes a rename of as_scalar() into scalar_subquery(); deprecates automatic coercion to scalar_subquery(). Partially-fixes: #4617 Change-Id: I26f1e78898693c6b99ef7ea2f4e7dfd0e8e1a1bd
* Reimplement .compare() in terms of a visitorMike Bayer2019-04-291-0/+6
| | | | | | | | | | | | Reworked the :meth:`.ClauseElement.compare` methods in terms of a new visitor-based approach, and additionally added test coverage ensuring that all :class:`.ClauseElement` subclasses can be accurately compared against each other in terms of structure. Structural comparison capability is used to a small degree within the ORM currently, however it also may form the basis for new caching features. Fixes: #4336 Change-Id: I581b667d8e1642a6c27165cc9f4aded1c66effc6
* Allow SQL expression for ORM primary keysMike Bayer2019-02-121-1/+6
| | | | | | | | | | | | | A SQL expression can now be assigned to a primary key attribute for an ORM flush in the same manner as ordinary attributes as described in :ref:`flush_embedded_sql_expressions` where the expression will be evaulated and then returned to the ORM using RETURNING, or in the case of pysqlite, works using the cursor.lastrowid attribute.Requires either a database that supports RETURNING (e.g. Postgresql, Oracle, SQL Server) or pysqlite. Fixes: #3133 Fixes: #4494 Change-Id: I83da8357354de002cb04fa4a553f2a2f90c5157d
* happy new yearMike Bayer2019-01-111-1/+1
| | | | Change-Id: I6a71f4924d046cf306961c58dffccf21e9c03911
* Post black reformattingMike Bayer2019-01-061-12/+22
| | | | | | | | | | | | | Applied on top of a pure run of black -l 79 in I7eda77fed3d8e73df84b3651fd6cfcfe858d4dc9, this set of changes resolves all remaining flake8 conditions for those codes we have enabled in setup.cfg. Included are resolutions for all remaining flake8 issues including shadowed builtins, long lines, import order, unused imports, duplicate imports, and docstring issues. Change-Id: I4f72d3ba1380dd601610ff80b8fb06a2aff8b0fe
* Run black -l 79 against all source filesMike Bayer2019-01-061-178/+262
| | | | | | | | | | | | | | This is a straight reformat run using black as is, with no edits applied at all. The black run will format code consistently, however in some cases that are prevalent in SQLAlchemy code it produces too-long lines. The too-long lines will be resolved in the following commit that will resolve all remaining flake8 issues including shadowed builtins, long lines, import order, unused imports, duplicate imports, and docstring issues. Change-Id: I7eda77fed3d8e73df84b3651fd6cfcfe858d4dc9
* happy new yearMike Bayer2018-01-121-1/+1
| | | | Change-Id: I3ef36bfd0cb0ba62b3123c8cf92370a43156cf8f
* `ValuesBase.values` inconsistency fixAubrey Stark-Toller2018-01-121-17/+22
| | | | | | | | | | Fixed bug in :meth:`.Insert.values` where using the "multi-values" format in combination with :class:`.Column` objects as keys rather than strings would fail. Pull request courtesy Aubrey Stark-Toller. Change-Id: I9d3b40b5950df8f5bfdc8b1d22f9c3afb277f17f Pull-request: https://github.com/zzzeek/sqlalchemy/pull/412 Fixes: #4162
* Add multivalued insert context for defaultsreview/mike_bayer/ticket_4075Mike Bayer2017-09-161-0/+3
| | | | | | | | | | | | | | Added a new method :class:`.DefaultExecutionContext.current_parameters` which is used within a function-based default value generator in order to retrieve the current parameters being passed to the statement. The new function differs from the ``.current_parameters`` attribute in that it also provides for optional grouping of parameters that correspond to a multi-valued "insert" construct. Previously it was not possible to identify the subset of parameters that were relevant to the function call. Change-Id: I6894c7b4a2bce3e83c3ade8af0e5b2f8df37b785 Fixes: #4075
* Pass **kw to bound params in multi valuesMike Bayer2017-01-091-1/+1
| | | | | | | | | Fixed bug where literal_binds compiler flag was not honored by the :class:`.Insert` construct for the "multiple values" feature; the subsequent values are now rendered as literals. Change-Id: I81ac358fd59995885d482e7571620090210865d2 Fixes: #3880
* update for 2017 copyrightMike Bayer2017-01-041-1/+1
| | | | Change-Id: I4e8c2aa8fe817bb2af8707410fa0201f938781de
* Call nextval() on sequence when doing INSERT from SELECTMike Bayer2016-12-211-1/+1
| | | | | | | | Fixed bug where an INSERT from SELECT where the source table contains an autoincrementing Sequence would fail to compile correctly. Change-Id: I41eb9f65789a4007712ae61ed5fa23a9839a5128 Fixes: #3877
* Count columns using PrimaryKeyConstraint.__len__ directlyMike Bayer2016-11-121-1/+1
| | | | | | | | | | PrimaryKeyConstraint is present on Table however on table() and others it's a ColumnSet. The warning here only needs len() and PrimaryKeyConstraint supports that directly in the same way as ColumnSet. Change-Id: I19c11a39110bfef48cdea49a471e7ab80b537538 Fixes: #3842
* Add a qualifying character to multi INSERT..VALUES parameter namesMike Bayer2016-10-171-3/+3
| | | | | | | | | | Changed the naming convention used when generating bound parameters for a multi-VALUES insert statement, so that the numbered parameter names don't conflict with the anonymized parameters of a WHERE clause, as is now common in a PostgreSQL ON CONFLICT construct. Change-Id: I3188d100fe4d322a47d344d6a63d3e40b915f228 Fixes: #3828
* Change autoincrement compileerror to a warningMike Bayer2016-10-071-5/+5
| | | | | | | | Users are complaining that IntegrityError is no longer raised. Change-Id: I0855d5b7a98d4338f0910501b6e6d404ba33634d Fixes: #3216
* Work w/ prefetch even for selects, if presentMike Bayer2016-07-121-12/+23
| | | | | | | | | | | | | | | | | | | Fixed bug in new CTE feature for update/insert/delete stated as a CTE inside of an enclosing statement (typically SELECT) whereby oninsert and onupdate values weren't called upon for the embedded statement. This is accomplished by consulting prefetch for all statements. The collection is also broken into separate insert/update collections so that we don't need to consult toplevel self.isinsert to determine if the prefetch is for an insert or an update. What we don't yet test for are CTE combinations that have both insert/update in one statement, though these should now work in theory provided the underlying database supports such a statement. Change-Id: I3b6a860e22c86743c91c56a7ec751ff706f66f64 Fixes: #3745
* Raise exception when we have multiple values with an unknown column. Fixes ↵Athena Yao2016-03-021-1/+1
| | | | #3666.
* - CTE functionality has been expanded to support all DML, allowingMike Bayer2016-02-111-5/+42
| | | | | | | INSERT, UPDATE, and DELETE statements to both specify their own WITH clause, as well as for these statements themselves to be CTE expressions when they include a RETURNING clause. fixes #2551
* fix passing literal_binds flag through for update and insertpr/232Tim Tate2016-02-021-3/+4
|
* - happy new yearMike Bayer2016-01-291-1/+1
|
* - Fixed bug in :meth:`.Update.return_defaults` which would cause allMike Bayer2015-12-141-0/+1
| | | | | | | | | | | | | | insert-default holding columns not otherwise included in the SET clause (such as primary key cols) to get rendered into the RETURNING even though this is an UPDATE. - Major fixes to the :paramref:`.Mapper.eager_defaults` flag, this flag would not be honored correctly in the case that multiple UPDATE statements were to be emitted, either as part of a flush or a bulk update operation. Additionally, RETURNING would be emitted unnecessarily within update statements. fixes #3609
* - Fixed issue within the :meth:`.Insert.from_select` construct wherebyMike Bayer2015-12-091-2/+3
| | | | | | | | | | | the :class:`.Select` construct would have its ``._raw_columns`` collection mutated in-place when compiling the :class:`.Insert` construct, when the target :class:`.Table` has Python-side defaults. The :class:`.Select` construct would compile standalone with the erroneous column present subsequent to compilation of the :class:`.Insert`, and the the :class:`.Insert` statement itself would fail on a second compile attempt due to duplicate bound parameters. fixes #3603
* - Added support for parameter-ordered SET clauses in an UPDATEMike Bayer2015-11-281-1/+12
| | | | | | | | | | statement. This feature is available by passing the :paramref:`~.sqlalchemy.sql.expression.update.preserve_parameter_order` flag either to the core :class:`.Update` construct or alternatively adding it to the :paramref:`.Query.update.update_args` dictionary at the ORM-level, also passing the parameters themselves as a list of 2-tuples. Thanks to Gorka Eguileor for implementation and tests. adapted from pullreq github:200
* - The system by which a :class:`.Column` considers itself to be anMike Bayer2015-10-071-9/+82
| | | | | | | | | | | | | | | | | "auto increment" column has been changed, such that autoincrement is no longer implicitly enabled for a :class:`.Table` that has a composite primary key. In order to accommodate being able to enable autoincrement for a composite PK member column while at the same time maintaining SQLAlchemy's long standing behavior of enabling implicit autoincrement for a single integer primary key, a third state has been added to the :paramref:`.Column.autoincrement` parameter ``"auto"``, which is now the default. fixes #3216 - The MySQL dialect no longer generates an extra "KEY" directive when generating CREATE TABLE DDL for a table using InnoDB with a composite primary key with AUTO_INCREMENT on a column that isn't the first column; to overcome InnoDB's limitation here, the PRIMARY KEY constraint is now generated with the AUTO_INCREMENT column placed first in the list of columns.
* - Fixed regression in 1.0-released default-processor for multi-VALUESMike Bayer2015-08-311-0/+1
| | | | | | | | insert statement, :ticket:`3288`, where the column type for the default-holding column would not be propagated to the compiled statement in the case where the default was being used, leading to bind-level type handlers not being invoked. fixes #3520
* - copyright 2015Mike Bayer2015-03-101-1/+1
|
* - refine the previous commit a bitMike Bayer2015-01-131-20/+27
|
* - The multi-values version of :meth:`.Insert.values` has beenMike Bayer2015-01-131-32/+51
| | | | | | | | | | | | repaired to work more usefully with tables that have Python- side default values and/or functions, as well as server-side defaults. The feature will now work with a dialect that uses "positional" parameters; a Python callable will also be invoked individually for each row just as is the case with an "executemany" style invocation; a server- side default column will no longer implicitly receive the value explicitly specified for the first row, instead refusing to invoke without an explicit value. fixes #3288
* - :meth:`.Insert.from_select` now includes Python and SQL-expressionMike Bayer2014-10-101-20/+77
| | | | | | | defaults if otherwise unspecified; the limitation where non- server column defaults aren't included in an INSERT FROM SELECT is now lifted and these expressions are rendered as constants into the SELECT statement.
* - cyclomatic complexity: _get_colparams() becomes sql.crud._get_crud_params,Mike Bayer2014-09-271-0/+473
CC goes from F to D
View Lorry Controller Status