summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* ORM executemany returningMike Bayer2020-06-2716-367/+911
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Build on #5401 to allow the ORM to take advanage of executemany INSERT + RETURNING. Implemented the feature updated tests to support INSERT DEFAULT VALUES, needed to come up with a new syntax for compiler INSERT INTO table (anycol) VALUES (DEFAULT) which can then be iterated out for executemany. Added graceful degrade to plain executemany for PostgreSQL <= 8.2 Renamed EXECUTEMANY_DEFAULT to EXECUTEMANY_PLAIN Fix issue where unicode identifiers or parameter names wouldn't work with execute_values() under Py2K, because we have to encode the statement and therefore have to encode the insert_single_values_expr too. Correct issue from #5401 to support executemany + return_defaults for a PK that is explicitly pre-generated, meaning we aren't actually getting RETURNING but need to return it from compiled_parameters. Fixes: #5263 Change-Id: Id68e5c158c4f9ebc33b61c06a448907921c2a657
* Merge "Default psycopg2 executemany mode to "values_only""mike bayer2020-06-2639-531/+881
|\
| * Default psycopg2 executemany mode to "values_only"Mike Bayer2020-06-2539-531/+881
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The psycopg2 dialect now defaults to using the very performant ``execute_values()`` psycopg2 extension for compiled INSERT statements, and also impements RETURNING support when this extension is used. This allows INSERT statements that even include an autoincremented SERIAL or IDENTITY value to run very fast while still being able to return the newly generated primary key values. The ORM will then integrate this new feature in a separate change. Implements RETURNING for insert with executemany Adds support to return_defaults() mode and inserted_primary_key to support mutiple INSERTed rows, via return_defauls_rows and inserted_primary_key_rows accessors. within default execution context, new cached compiler getters are used to fetch primary keys from rows inserted_primary_key now returns a plain tuple. this is not yet a row-like object however this can be added. Adds distinct "values_only" and "batch" modes, as "values" has a lot of benefits but "batch" breaks cursor.rowcount psycopg2 minimum version 2.7 so we can remove the large number of checks for very old versions of psycopg2 simplify tests to no longer distinguish between native and non-native json Fixes: #5401 Change-Id: Ic08fd3423d4c5d16ca50994460c0c234868bd61c
* | cherry-pick changelog update for 1.3.19Mike Bayer2020-06-251-0/+4
| |
* | cherry-pick changelog from 1.3.18Mike Bayer2020-06-2522-161/+182
| |
* | Merge "Use index name to determine if an index is for the PK"mike bayer2020-06-253-14/+104
|\ \
| * | Use index name to determine if an index is for the PKMike Bayer2020-06-253-14/+104
| |/ | | | | | | | | | | | | | | | | | | | | | | Fixed bug in Oracle dialect where indexes that contain the full set of primary key columns would be mistaken as the primary key index itself, which is omitted, even if there were multiples. The check has been refined to compare the name of the primary key constraint against the index name itself, rather than trying to guess based on the columns present in the index. Fixes: #5421 Change-Id: I47c2ccdd0b13977cfd9ef249d4de06371c4fb241
* | Use time.perf_counter() for cache time measurementMike Bayer2020-06-244-4/+5
|/ | | | | | See https://twitter.com/raymondh/status/1275937373080023040 Change-Id: Iaa0abb0c433ccedfbd88d00e3970120242ba379b
* Merge "Propose using RETURNING for bulk updates, deletes"mike bayer2020-06-2321-124/+698
|\
| * Propose using RETURNING for bulk updates, deletesMike Bayer2020-06-2321-124/+698
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch makes several improvements in the area of bulk updates and deletes as well as the new session mechanics. RETURNING is now used for an UPDATE or DELETE statement emitted for a diaelct that supports "full returning" in order to satisfy the "fetch" strategy; this currently includes PostgreSQL and SQL Server. The Oracle dialect does not support RETURNING for more than one row, so a new dialect capability "full_returning" is added in addition to the existing "implicit_returning", indicating this dialect supports RETURNING for zero or more rows, not just a single identity row. The "fetch" strategy will gracefully degrade to the previous SELECT mechanics for dialects that do not support RETURNING. Additionally, the "fetch" strategy will attempt to use evaluation for the VALUES that were UPDATEd, rather than just expiring the updated attributes. Values should be evalutable in all cases where the value is not a SQL expression. The new approach also incurs some changes in the session.execute mechanics, where do_orm_execute() event handlers can now be chained to each return results; this is in turn used by the handler to detect on a per-bind basis if the fetch strategy needs to do a SELECT or if it can do RETURNING. A test suite is added to test_horizontal_shard that breaks up a single UPDATE or DELETE operation among multiple backends where some are SQLite and don't support RETURNING and others are PostgreSQL and do. The session event mechanics are corrected in terms of the "orm pre execute" hook, which now receives a flag "is_reentrant" so that the two ORM implementations for this can skip on their work if they are being called inside of ORMExecuteState.invoke(), where previously bulk update/delete were calling its SELECT a second time. In order for "fetch" to get the correct identity when called as pre-execute, it also requests the identity_token for each mapped instance which is now added as an optional capability of a SELECT for ORM columns. the identity_token that's placed by horizontal_sharding is now made available within each result row, so that even when fetching a merged result of plain rows we can tell which row belongs to which identity token. The evaluator that takes place within the ORM bulk update and delete for synchronize_session="evaluate" now supports the IN and NOT IN operators. Tuple IN is also supported. Fixes: #1653 Change-Id: I2292b56ae004b997cef0ba4d3fc350ae1dd5efc1
* | Add pip list to workflow, update setuptoolsFederico Caselli2020-06-221-3/+10
|/ | | | Change-Id: I2e32ba315a79dcfc7c5f79ed6517ac0cee4f55e3
* Merge "Apply dialect_options copy fix"mike bayer2020-06-203-3/+115
|\
| * Apply dialect_options copy fixGord Thompson2020-06-183-3/+115
| | | | | | | | | | Fixes: #5276 Change-Id: Ic608310d4a85934fc9fa4d72daef66323c6e2525
* | Merge "Propose black py27 + py35 mode for the rest of Py2K"mike bayer2020-06-191-0/+1
|\ \
| * | Propose black py27 + py35 mode for the rest of Py2KMike Bayer2020-06-191-0/+1
| |/ | | | | | | | | | | | | | | | | | | | | | | | | | | py27 mode produces one failure for flake8 which is the space added between exec and parenthesis. however apparently we can add multiple versions to target-versions which allows the exec() calls to come out in python 3 style. The issue we want to improve is issues of trailing commas being added. I'm not really able to get black to consistently add or remove these trailing commas in any case no matter what I set target-version towards. Change-Id: Iceb9e9aeb5f5e0d317fb877ef06eca0290997fe4
* | Merge "Added reflection method :meth:`.Inspector.get_sequence_names`"mike bayer2020-06-199-85/+228
|\ \
| * | Added reflection method :meth:`.Inspector.get_sequence_names`Federico Caselli2020-06-039-85/+228
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Added new reflection method :meth:`.Inspector.get_sequence_names` which returns all the sequences defined. Support for this method has been added to the backend that support :class:`.Sequence`: PostgreSql, Oracle, MSSQL and MariaDB >= 10.3. Fixes: #2056 Change-Id: I0949696a39aa28c849edf2504779241f7443778a
* | | Merge "Move most setup options to setup.cfg"mike bayer2020-06-193-95/+79
|\ \ \
| * | | Move most setup options to setup.cfgMike Bayer2020-06-193-95/+79
| | |/ | |/| | | | | | | | | | | | | | | | | | | Installation has been modernized to use setup.cfg for most package metadata. Fixes: #5404 Change-Id: Id59c0c4e7f81eca89999d1f1f8b03c89835a09d1
* | | Merge "perf tweaks"mike bayer2020-06-197-52/+107
|\ \ \
| * | | perf tweaksMike Bayer2020-06-197-52/+107
| |/ / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - avoid abc checks in distill_20 - ColumnEntity subclasses are unique to their compile state and have no querycontext specific state. They can do a simple memoize of their fetch_column without using attributes, and they can memoize their _getter() too so that it goes into the cache, just like instance_processor() does. - unify ORMColumnEntity and RawColumnEntity for the row processor part, add some test coverage for the case where it is used in a from_statement - do a faster generate if there are no memoized entries - query._params is always immutabledict Change-Id: I1e2dfe607a1749b5b434fc11f9348ee631501dfa
* | | Add note indicating order of join() calls are importantMike Bayer2020-06-191-0/+13
|/ / | | | | | | | | Fixes: #5406 Change-Id: I186792b32bd156d5ddf256dcd018af32ad5b515e
* | Warn when transaction context manager ends on inactive transactionMike Bayer2020-06-123-5/+28
| | | | | | | | | | | | | | | | | | | | if .rollback() or .commit() is called inside the transaction context manager, the transaction object is deactivated. the context manager continues but will not be able to correctly fulfill it's closing state. Ensure a warning is emitted when this happens. Change-Id: I8fc3a73f7c21575dda5bcbd6fb74ddb679771630
* | Add version token to error URLMike Bayer2020-06-113-10/+35
| | | | | | | | | | | | | | | | | | | | | | | | | | the sqlalche.me redirector now supports the numerical version code in the URL, e.g. /13/, /14/, /20/, etc., so that we can redirect to the error codes for the appropriate version of SQLAlchemy in use without going through the catch-all "latest" link. If a particular version of the docs is no longer on the site, the redirect will revert to falling through the "latest" link (which ultimately lands on the current release version, /13/ at the time of this writing). Change-Id: I3bb463fd6fb6c8767c95a57f3699aba715a9a72d
* | Add 'exists' to SQLite's reserved keywordsThodoris Sotiropoulos2020-06-113-0/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | Added "exists" to the list of reserved words for SQLite so that this word will be quoted when used as a label or column name. Pull request courtesy Thodoris Sotiropoulos. Fixes: #5395 Closes: #5396 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5396 Pull-request-sha: 5608317a38db7b63115e3b467d182a3e5cc31580 Change-Id: Ia4769de2dec159dcf282eb4b30c11560da51a5c7
* | Merge "Add documentation regarding row constructo in PostgreSQL"mike bayer2020-06-111-1/+45
|\ \
| * | Add documentation regarding row constructo in PostgreSQLFederico Caselli2020-05-251-1/+45
| | | | | | | | | | | | | | | Fixes: #5331 Change-Id: Ia795a4d4d8ae4944d8a160d18f8b917177acf0de
* | | Merge "Improve ``Rendering Bound Parameters Inline`` documentation"mike bayer2020-06-111-6/+11
|\ \ \
| * | | Improve ``Rendering Bound Parameters Inline`` documentationFederico Caselli2020-06-041-6/+11
| | | | | | | | | | | | | | | | | | | | | | | | This is mainly to make this section more discoverable by users Change-Id: I58ba19e4a9ae85b227e5b5553ba5c30c01f3e005
* | | | Merge "Turn on caching everywhere, add logging"mike bayer2020-06-1055-832/+1921
|\ \ \ \
| * | | | Turn on caching everywhere, add loggingMike Bayer2020-06-1055-832/+1921
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | A variety of caching issues found by running all tests with statement caching turned on. The cache system now has a more conservative approach where any subclass of a SQL element will by default invalidate the cache key unless it adds the flag inherit_cache=True at the class level, or if it implements its own caching. Add working caching to a few elements that were omitted previously; fix some caching implementations to suit lesser used edge cases such as json casts and array slices. Refine the way BaseCursorResult and CursorMetaData interact with caching; to suit cases like Alembic modifying table structures, don't cache the cursor metadata if it were created against a cursor.description using non-positional matching, e.g. "select *". if a table re-ordered its columns or added/removed, now that data is obsolete. Additionally we have to adapt the cursor metadata _keymap regardless of if we just processed cursor.description, because if we ran against a cached SQLCompiler we won't have the right columns in _keymap. Other refinements to how and when we do this adaption as some weird cases were exposed in the Postgresql dialect, a text() construct that names just one column that is not actually in the statement. Fixed that also as it looks like a cut-and-paste artifact that doesn't actually affect anything. Various issues with re-use of compiled result maps and cursor metadata in conjunction with tables being changed, such as change in order of columns. mappers can be cleared but the class remains, meaning a mapper has to use itself as the cache key not the class. lots of bound parameter / literal issues, due to Alembic creating a straight subclass of bindparam that renders inline directly. While we can update Alembic to not do this, we have to assume other people might be doing this, so bindparam() implements the inherit_cache=True logic as well that was a bit involved. turn on cache stats in logging. Includes a fix to subqueryloader which moves all setup to the create_row_processor() phase and elminates any storage within the compiled context. This includes some changes to create_row_processor() signature and a revising of the technique used to determine if the loader can participate in polymorphic queries, which is also applied to selectinloading. DML update.values() and ordered_values() now coerces the keys as we have tests that pass an arbitrary class here which only includes __clause_element__(), so the key can't be cached unless it is coerced. this in turn changed how composite attributes support bulk update to use the standard approach of ClauseElement with annotations that are parsed in the ORM context. memory profiling successfully caught that the Session from Query was getting passed into _statement_20() so that was a big win for that test suite. Apparently Compiler had .execute() and .scalar() methods stuck on it, these date back to version 0.4 and there was a single test in the PostgreSQL dialect tests that exercised it for no apparent reason. Removed these methods as well as the concept of a Compiler holding onto a "bind". Fixes: #5386 Change-Id: I990b43aab96b42665af1b2187ad6020bee778784
* | | | | Merge "updated historical terms with modern equivalents"mike bayer2020-06-092-40/+40
|\ \ \ \ \
| * | | | | updated historical terms with modern equivalentsjonathan vanasco2020-06-052-40/+40
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | ### Description There were a few remnant uses of master/slave in the code and docs. The project previously made a decision to move away from them to use modern and inclusive terminology. This PR does not cover a bug or necessitate a documented entry into the changelog, so an issue ticket was not created. ### Checklist This pull request is: - [x] A documentation / typographical error fix - [x] A short code fix - [ ] A new feature implementation Closes: #5381 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5381 Pull-request-sha: 92597e83d0e1e18960dbb39b604b313e7a1cbb30 Change-Id: I1fb34fe5ab6c19fd7360568d7b51cdea9d271b3b
* | | | | | Fix capitalization typos in engines.rstAbraham Sangha2020-06-081-2/+2
| |/ / / / |/| | | |
* | | | | Convert bulk update/delete to new execution modelMike Bayer2020-06-0631-507/+1123
|/ / / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This reorganizes the BulkUD model in sqlalchemy.orm.persistence to be based on the CompileState concept and to allow plain update() / delete() to be passed to session.execute() where the ORM synchronize session logic will take place. Also gets "synchronize_session='fetch'" working with horizontal sharding. Adding a few more result.scalar_one() types of methods as scalar_one() seems like what is normally desired. Fixes: #5160 Change-Id: I8001ebdad089da34119eb459709731ba6c0ba975
* | | | Fix connection string escaping for mssql+pyodbcGord Thompson2020-06-043-4/+12
|/ / / | | | | | | | | | | | | Fixes: #5373 Change-Id: Ia41e8f1ef8644c54d23ebfdf3f909c785adf0fb0
* | | Document that type_coerce does not currently imply parenthesizationMike Bayer2020-06-041-11/+25
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We've had a few issues where the current solution is to use the self_group() method, so document that as the current approach for the parenthesization use case. Whether or not type_coerce() is changed later, this is how it works at the moment. Fixes: #5375 Change-Id: I97414762a87ec8f1fd1adc1b6be5a52e576814ca References: #5344
* | | Merge "Folds two identical exception handlers into a single one"mike bayer2020-06-032-8/+2
|\ \ \
| * | | Folds two identical exception handlers into a single oneElmer de Looff2020-06-032-8/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixes a `TODO` that searches for py2/3 compatible syntax to match multiple exception types. ### Description Merges the two exception clauses using the syntax that exists for both Python 2 and 3 as per the exception handling tutorials ([Python 2](https://docs.python.org/2/tutorial/errors.html#handling-exceptions), [Python 3](https://docs.python.org/3/tutorial/errors.html#handling-exceptions)) ### Checklist This pull request is: - [ ] A documentation / typographical error fix - Good to go, no issue or tests are needed - [x] A short code fix - please include the issue number, and create an issue if none exists, which must include a complete example of the issue. one line code fixes without an issue and demonstration will not be accepted. - Please include: `Fixes: #<issue number>` in the commit message - please include tests. one line code fixes without tests will not be accepted. - [ ] A new feature implementation - please include the issue number, and create an issue if none exists, which must include a complete example of how the feature would look. - Please include: `Fixes: #<issue number>` in the commit message - please include tests. **Have a nice day!** Closes: #5325 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5325 Pull-request-sha: d0835511e5509fb368a2193ebe4ecbbe3504b129 Change-Id: If3a8854e4de023e0ec68ae8649d24af9412f0717
* | | | Merge "Default create_constraint to False"mike bayer2020-06-038-46/+188
|\ \ \ \ | |_|_|/ |/| | |
| * | | Default create_constraint to FalseMike Bayer2020-06-038-46/+188
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The :paramref:`.Enum.create_constraint` and :paramref:`.Boolean.create_constraint` parameters now default to False, indicating when a so-called "non-native" version of these two datatypes is created, a CHECK constraint will not be generated by default. These CHECK constraints present schema-management maintenance complexities that should be opted in to, rather than being turned on by default. Fixes: #5367 Change-Id: I0a3fb608ce32143fa757546cc17ba2013e93272a
* | | | Inline a few ORM arguments, othersMike Bayer2020-06-036-48/+149
| |/ / |/| | | | | | | | | | | | | | small changes Change-Id: Id89a0651196c431d0aaf6935f5a4e7b12dd70c6c
* | | Merge "Add default expression to query_expression()"mike bayer2020-06-024-11/+71
|\ \ \ | |/ / |/| |
| * | Add default expression to query_expression()Haoyu Sun2020-06-014-11/+71
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Added a new parameter :paramref:`_orm.query_expression.default_expr` to the :func:`_orm.query_expression` construct, which will be appled to queries automatically if the :func:`_orm.with_expression` option is not used. Pull request courtesy Haoyu Sun. Fixes: #5198 Closes: #5354 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5354 Pull-request-sha: 57dd4922a3ae4e5fe56dcc541d85ce42256b38b9 Change-Id: I3400f2c00b58bf161f31c74c579feb9ac0f03356
* | | Merge "Refine IN and scalar subquery coercions"mike bayer2020-06-024-191/+325
|\ \ \
| * | | Refine IN and scalar subquery coercionsMike Bayer2020-06-014-191/+325
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Ensure IN emits a warning when it coerces a FromClause into a select(), however that it continues to allow the scalar_subquery() coercion to be automatic, particularly since it's not clear that "col IN (select)" is necessarily "scalar" in the case of tuples. Convert the "scalar_subquery()" warning emitted in other cases to be a warning, rather than a deprecation warning. I can't imagine taking this coercion out as it is intuitive and is always going to happen; we just would like to note that an implicit coercion is occurring. Fixes: #5369 Change-Id: I748f01f40bc85c64e2776f9b88ef35641fa8fb5c
* | | | Mark two tests as requiring boolean col expressionsMike Bayer2020-06-011-0/+2
|/ / / | | | | | | | | | | | | | | | fixes CI failures on mssql Change-Id: Ib9d3102bd7f8f4fb9ac5ac21abbb6a866e1f99cf
* | | Add a test / comment about test_orm_queryMike Bayer2020-06-011-0/+19
| | | | | | | | | | | | Change-Id: I8d78b3e75127141da177f711fd91216391a3c859
* | | Merge "Improve rendering of core statements w/ ORM elements"mike bayer2020-06-0135-927/+1495
|\ \ \
| * | | Improve rendering of core statements w/ ORM elementsMike Bayer2020-05-3135-927/+1495
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch contains a variety of ORM and expression layer tweaks to support ORM constructs in select() statements, without the 1.3.x requiremnt in Query that a full _compile_context() + new select() is needed in order to get a working statement object. Includes such tweaks as the ability to implement aliased class of an aliased class, as we are looking to fully support ACs against subqueries, as well as the ability to access anonymously-labeled ColumnProperty expressions within subqueries by naming the ".key" of the label after the property key. Some tuning to query.join() as well as ORMJoin internals to allow things to work more smoothly. Change-Id: Id810f485c5f7ed971529489b84694e02a3356d6d
View Lorry Controller Status