<feed xmlns='http://www.w3.org/2005/Atom'>
<title>delta/python-packages/sqlalchemy.git/test/perf, branch 2020_tutorial</title>
<subtitle>github.com: zzzeek/sqlalchemy.git
</subtitle>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/python-packages/sqlalchemy.git/'/>
<entry>
<title>Turn on caching everywhere, add logging</title>
<updated>2020-06-10T19:29:01+00:00</updated>
<author>
<name>Mike Bayer</name>
<email>mike_mp@zzzcomputing.com</email>
</author>
<published>2020-06-07T00:40:43+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/python-packages/sqlalchemy.git/commit/?id=b0cfa7379cf8513a821a3dbe3028c4965d9f85bd'/>
<id>b0cfa7379cf8513a821a3dbe3028c4965d9f85bd</id>
<content type='text'>
A variety of caching issues found by running
all tests with statement caching turned on.

The cache system now has a more conservative approach where
any subclass of a SQL element will by default invalidate
the cache key unless it adds the flag inherit_cache=True
at the class level, or if it implements its own caching.

Add working caching to a few elements that were
omitted previously; fix some caching implementations
to suit lesser used edge cases such as json casts
and array slices.

Refine the way BaseCursorResult and CursorMetaData
interact with caching; to suit cases like Alembic
modifying table structures, don't cache the
cursor metadata if it were created against a
cursor.description using non-positional matching,
e.g. "select *".   if a table re-ordered its columns
or added/removed, now that data is obsolete.

Additionally we have to adapt the cursor metadata
_keymap regardless of if we just processed
cursor.description, because if we ran against
a cached SQLCompiler we won't have the right
columns in _keymap.

Other refinements to how and when we do this
adaption as some weird cases
were exposed in the Postgresql dialect,
a text() construct that names just one column that
is not actually in the statement.   Fixed that
also as it looks like a cut-and-paste artifact
that doesn't actually affect anything.

Various issues with re-use of compiled result maps
and cursor metadata in conjunction with tables being
changed, such as change in order of columns.

mappers can be cleared but the class remains, meaning
a mapper has to use itself as the cache key not the class.

lots of bound parameter / literal issues, due to Alembic
creating a straight subclass of bindparam that renders
inline directly.   While we can update Alembic to not
do this, we have to assume other people might be doing
this, so bindparam() implements the inherit_cache=True
logic as well that was a bit involved.

turn on cache stats in logging.

Includes a fix to subqueryloader which moves all setup to
the create_row_processor() phase and elminates any storage
within the compiled context.   This includes some changes
to create_row_processor() signature and a revising of the
technique used to determine if the loader can participate
in polymorphic queries, which is also applied to
selectinloading.

DML update.values() and ordered_values() now coerces the
keys as we have tests that pass an arbitrary class here
which only includes __clause_element__(), so the
key can't be cached unless it is coerced.  this in turn
changed how composite attributes support bulk update
to use the standard approach of ClauseElement with
annotations that are parsed in the ORM context.

memory profiling successfully caught that the Session
from Query was getting passed into _statement_20()
so that was a big win for that test suite.

Apparently Compiler had .execute() and .scalar() methods
stuck on it, these date back to version 0.4 and there
was a single test in the PostgreSQL dialect tests
that exercised it for no apparent reason.   Removed
these methods as well as the concept of a Compiler
holding onto a "bind".

Fixes: #5386

Change-Id: I990b43aab96b42665af1b2187ad6020bee778784
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
A variety of caching issues found by running
all tests with statement caching turned on.

The cache system now has a more conservative approach where
any subclass of a SQL element will by default invalidate
the cache key unless it adds the flag inherit_cache=True
at the class level, or if it implements its own caching.

Add working caching to a few elements that were
omitted previously; fix some caching implementations
to suit lesser used edge cases such as json casts
and array slices.

Refine the way BaseCursorResult and CursorMetaData
interact with caching; to suit cases like Alembic
modifying table structures, don't cache the
cursor metadata if it were created against a
cursor.description using non-positional matching,
e.g. "select *".   if a table re-ordered its columns
or added/removed, now that data is obsolete.

Additionally we have to adapt the cursor metadata
_keymap regardless of if we just processed
cursor.description, because if we ran against
a cached SQLCompiler we won't have the right
columns in _keymap.

Other refinements to how and when we do this
adaption as some weird cases
were exposed in the Postgresql dialect,
a text() construct that names just one column that
is not actually in the statement.   Fixed that
also as it looks like a cut-and-paste artifact
that doesn't actually affect anything.

Various issues with re-use of compiled result maps
and cursor metadata in conjunction with tables being
changed, such as change in order of columns.

mappers can be cleared but the class remains, meaning
a mapper has to use itself as the cache key not the class.

lots of bound parameter / literal issues, due to Alembic
creating a straight subclass of bindparam that renders
inline directly.   While we can update Alembic to not
do this, we have to assume other people might be doing
this, so bindparam() implements the inherit_cache=True
logic as well that was a bit involved.

turn on cache stats in logging.

Includes a fix to subqueryloader which moves all setup to
the create_row_processor() phase and elminates any storage
within the compiled context.   This includes some changes
to create_row_processor() signature and a revising of the
technique used to determine if the loader can participate
in polymorphic queries, which is also applied to
selectinloading.

DML update.values() and ordered_values() now coerces the
keys as we have tests that pass an arbitrary class here
which only includes __clause_element__(), so the
key can't be cached unless it is coerced.  this in turn
changed how composite attributes support bulk update
to use the standard approach of ClauseElement with
annotations that are parsed in the ORM context.

memory profiling successfully caught that the Session
from Query was getting passed into _statement_20()
so that was a big win for that test suite.

Apparently Compiler had .execute() and .scalar() methods
stuck on it, these date back to version 0.4 and there
was a single test in the PostgreSQL dialect tests
that exercised it for no apparent reason.   Removed
these methods as well as the concept of a Compiler
holding onto a "bind".

Fixes: #5386

Change-Id: I990b43aab96b42665af1b2187ad6020bee778784
</pre>
</div>
</content>
</entry>
<entry>
<title>Convert execution to move through Session</title>
<updated>2020-05-25T17:56:37+00:00</updated>
<author>
<name>Mike Bayer</name>
<email>mike_mp@zzzcomputing.com</email>
</author>
<published>2020-04-27T16:58:12+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/python-packages/sqlalchemy.git/commit/?id=6930dfc032c3f9f474e71ab4e021c0ef8384930e'/>
<id>6930dfc032c3f9f474e71ab4e021c0ef8384930e</id>
<content type='text'>
This patch replaces the ORM execution flow with a
single pathway through Session.execute() for all queries,
including Core and ORM.

Currently included is full support for ORM Query,
Query.from_statement(), select(), as well as the
baked query and horizontal shard systems.  Initial
changes have also been made to the dogpile caching
example, which like baked query makes use of a
new ORM-specific execution hook that replaces the
use of both QueryEvents.before_compile() as well
as Query._execute_and_instances() as the central
ORM interception hooks.

select() and Query() constructs alike can be passed to
Session.execute() where they will return ORM
results in a Results object.   This API is currently
used internally by Query.   Full support for
Session.execute()-&gt;results to behave in a fully
2.0 fashion will be in later changesets.

bulk update/delete with ORM support will also
be delivered via the update() and delete()
constructs, however these have not yet been adapted
to the new system and may follow in a subsequent
update.

Performance is also beginning to lag as of this
commit and some previous ones.   It is hoped that
a few central functions such as the coercions
functions can be rewritten in C to re-gain
performance.  Additionally, query caching
is now available and some subsequent patches
will attempt to cache more of the per-execution
work from the ORM layer, e.g. column getters
and adapters.

This patch also contains initial "turn on" of the
caching system enginewide via the query_cache_size
parameter to create_engine(). Still defaulting at
zero for "no caching".   The caching system still
needs adjustments in order to gain adequate performance.

Change-Id: I047a7ebb26aa85dc01f6789fac2bff561dcd555d
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
This patch replaces the ORM execution flow with a
single pathway through Session.execute() for all queries,
including Core and ORM.

Currently included is full support for ORM Query,
Query.from_statement(), select(), as well as the
baked query and horizontal shard systems.  Initial
changes have also been made to the dogpile caching
example, which like baked query makes use of a
new ORM-specific execution hook that replaces the
use of both QueryEvents.before_compile() as well
as Query._execute_and_instances() as the central
ORM interception hooks.

select() and Query() constructs alike can be passed to
Session.execute() where they will return ORM
results in a Results object.   This API is currently
used internally by Query.   Full support for
Session.execute()-&gt;results to behave in a fully
2.0 fashion will be in later changesets.

bulk update/delete with ORM support will also
be delivered via the update() and delete()
constructs, however these have not yet been adapted
to the new system and may follow in a subsequent
update.

Performance is also beginning to lag as of this
commit and some previous ones.   It is hoped that
a few central functions such as the coercions
functions can be rewritten in C to re-gain
performance.  Additionally, query caching
is now available and some subsequent patches
will attempt to cache more of the per-execution
work from the ORM layer, e.g. column getters
and adapters.

This patch also contains initial "turn on" of the
caching system enginewide via the query_cache_size
parameter to create_engine(). Still defaulting at
zero for "no caching".   The caching system still
needs adjustments in order to gain adequate performance.

Change-Id: I047a7ebb26aa85dc01f6789fac2bff561dcd555d
</pre>
</div>
</content>
</entry>
<entry>
<title>Performance fixes for new result set</title>
<updated>2020-05-21T18:16:03+00:00</updated>
<author>
<name>Mike Bayer</name>
<email>mike_mp@zzzcomputing.com</email>
</author>
<published>2020-05-20T17:41:44+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/python-packages/sqlalchemy.git/commit/?id=4550983e0ce2f35b3585e53894c941c23693e71d'/>
<id>4550983e0ce2f35b3585e53894c941c23693e71d</id>
<content type='text'>
A few small mistakes led to huge callcounts.   Additionally,
the warn-on-get behavior which is attempting to warn for
deprecated access in SQLAlchemy 2.0 is very expensive; it's not clear
if its feasible to have this warning or to somehow alter how it
works.

Fixes: #5340
Change-Id: I73bdd2d7b6f1b25cc0222accabd585cf761a5af4
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
A few small mistakes led to huge callcounts.   Additionally,
the warn-on-get behavior which is attempting to warn for
deprecated access in SQLAlchemy 2.0 is very expensive; it's not clear
if its feasible to have this warning or to somehow alter how it
works.

Fixes: #5340
Change-Id: I73bdd2d7b6f1b25cc0222accabd585cf761a5af4
</pre>
</div>
</content>
</entry>
<entry>
<title>Deprecate plain string in execute and introduce `exec_driver_sql`</title>
<updated>2020-03-21T21:03:45+00:00</updated>
<author>
<name>Federico Caselli</name>
<email>cfederico87@gmail.com</email>
</author>
<published>2020-03-14T13:02:44+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/python-packages/sqlalchemy.git/commit/?id=9ec75882203b2c01aa1d362f939e21ebcd188e8d'/>
<id>9ec75882203b2c01aa1d362f939e21ebcd188e8d</id>
<content type='text'>
Execution of literal sql string is deprecated in the
:meth:`.Connection.execute` and a warning is raised when used stating
that it will be coerced to :func:`.text` in a future release.
To execute a raw sql string the new connection method
:meth:`.Connection.exec_driver_sql` was added, that will retain the previous
behavior, passing the string to the DBAPI driver unchanged.
Usage of scalar or tuple positional parameters in :meth:`.Connection.execute`
is also deprecated.

Fixes: #4848
Fixes: #5178
Change-Id: I2830181054327996d594f7f0d59c157d477c3aa9
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Execution of literal sql string is deprecated in the
:meth:`.Connection.execute` and a warning is raised when used stating
that it will be coerced to :func:`.text` in a future release.
To execute a raw sql string the new connection method
:meth:`.Connection.exec_driver_sql` was added, that will retain the previous
behavior, passing the string to the DBAPI driver unchanged.
Usage of scalar or tuple positional parameters in :meth:`.Connection.execute`
is also deprecated.

Fixes: #4848
Fixes: #5178
Change-Id: I2830181054327996d594f7f0d59c157d477c3aa9
</pre>
</div>
</content>
</entry>
<entry>
<title>Enable F841</title>
<updated>2019-06-20T17:50:41+00:00</updated>
<author>
<name>Mike Bayer</name>
<email>mike_mp@zzzcomputing.com</email>
</author>
<published>2019-05-30T15:31:03+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/python-packages/sqlalchemy.git/commit/?id=190e0139e834e4271268652e058c280787ae69eb'/>
<id>190e0139e834e4271268652e058c280787ae69eb</id>
<content type='text'>
This is a very useful assertion which prevents unused variables
from being set up allows code to be more readable and sometimes
even more efficient.  test suites seem to be where the most
problems are and there do not seem to be documentation examples
that are using this, or at least the linter is not taking effect
within rst blocks.

Change-Id: I2b3341d8dd14da34879d8425838e66a4b9f8e27d
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
This is a very useful assertion which prevents unused variables
from being set up allows code to be more readable and sometimes
even more efficient.  test suites seem to be where the most
problems are and there do not seem to be documentation examples
that are using this, or at least the linter is not taking effect
within rst blocks.

Change-Id: I2b3341d8dd14da34879d8425838e66a4b9f8e27d
</pre>
</div>
</content>
</entry>
<entry>
<title>Post black reformatting</title>
<updated>2019-01-06T23:23:11+00:00</updated>
<author>
<name>Mike Bayer</name>
<email>mike_mp@zzzcomputing.com</email>
</author>
<published>2019-01-06T06:19:47+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/python-packages/sqlalchemy.git/commit/?id=1e278de4cc9a4181e0747640a960e80efcea1ca9'/>
<id>1e278de4cc9a4181e0747640a960e80efcea1ca9</id>
<content type='text'>
Applied on top of a pure run of black -l 79 in
I7eda77fed3d8e73df84b3651fd6cfcfe858d4dc9,  this set of changes
resolves all remaining flake8 conditions for those codes
we have enabled in setup.cfg.

Included are resolutions for all remaining flake8 issues
including shadowed builtins, long lines, import order, unused
imports, duplicate imports, and docstring issues.

Change-Id: I4f72d3ba1380dd601610ff80b8fb06a2aff8b0fe
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Applied on top of a pure run of black -l 79 in
I7eda77fed3d8e73df84b3651fd6cfcfe858d4dc9,  this set of changes
resolves all remaining flake8 conditions for those codes
we have enabled in setup.cfg.

Included are resolutions for all remaining flake8 issues
including shadowed builtins, long lines, import order, unused
imports, duplicate imports, and docstring issues.

Change-Id: I4f72d3ba1380dd601610ff80b8fb06a2aff8b0fe
</pre>
</div>
</content>
</entry>
<entry>
<title>Run black -l 79 against all source files</title>
<updated>2019-01-06T17:34:50+00:00</updated>
<author>
<name>Mike Bayer</name>
<email>mike_mp@zzzcomputing.com</email>
</author>
<published>2019-01-06T06:14:26+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/python-packages/sqlalchemy.git/commit/?id=1e1a38e7801f410f244e4bbb44ec795ae152e04e'/>
<id>1e1a38e7801f410f244e4bbb44ec795ae152e04e</id>
<content type='text'>
This is a straight reformat run using black as is, with no edits
applied at all.

The black run will format code consistently, however in
some cases that are prevalent in SQLAlchemy code it produces
too-long lines.   The too-long lines will be resolved in the
following commit that will resolve all remaining flake8 issues
including shadowed builtins, long lines, import order, unused
imports, duplicate imports, and docstring issues.

Change-Id: I7eda77fed3d8e73df84b3651fd6cfcfe858d4dc9
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
This is a straight reformat run using black as is, with no edits
applied at all.

The black run will format code consistently, however in
some cases that are prevalent in SQLAlchemy code it produces
too-long lines.   The too-long lines will be resolved in the
following commit that will resolve all remaining flake8 issues
including shadowed builtins, long lines, import order, unused
imports, duplicate imports, and docstring issues.

Change-Id: I7eda77fed3d8e73df84b3651fd6cfcfe858d4dc9
</pre>
</div>
</content>
</entry>
<entry>
<title>print() is a function in Python 3</title>
<updated>2018-11-28T09:38:57+00:00</updated>
<author>
<name>cclauss</name>
<email>cclauss@bluewin.ch</email>
</author>
<published>2018-11-28T09:38:57+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/python-packages/sqlalchemy.git/commit/?id=971f0867856d04cb7da5eeea10daad5ad984d776'/>
<id>971f0867856d04cb7da5eeea10daad5ad984d776</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>Make all tests to be PEP8 compliant</title>
<updated>2017-02-07T16:21:56+00:00</updated>
<author>
<name>Khairi Hafsham</name>
<email>jumanjisama@gmail.com</email>
</author>
<published>2017-02-02T18:02:21+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/python-packages/sqlalchemy.git/commit/?id=772374735da27df1ddb907f4a0f5085b46dbe82b'/>
<id>772374735da27df1ddb907f4a0f5085b46dbe82b</id>
<content type='text'>
tested using pycodestyle version 2.2.0

Fixes: #3885
Change-Id: I5df43adc3aefe318f9eeab72a078247a548ec566
Pull-request: https://github.com/zzzeek/sqlalchemy/pull/343
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
tested using pycodestyle version 2.2.0

Fixes: #3885
Change-Id: I5df43adc3aefe318f9eeab72a078247a548ec566
Pull-request: https://github.com/zzzeek/sqlalchemy/pull/343
</pre>
</div>
</content>
</entry>
<entry>
<title>- Added a new extension suite :mod:`sqlalchemy.ext.baked`.  This</title>
<updated>2015-03-12T00:31:11+00:00</updated>
<author>
<name>Mike Bayer</name>
<email>mike_mp@zzzcomputing.com</email>
</author>
<published>2015-03-12T00:22:42+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/python-packages/sqlalchemy.git/commit/?id=e3b46bd62405b6ff57119e164718118f3e3565e0'/>
<id>e3b46bd62405b6ff57119e164718118f3e3565e0</id>
<content type='text'>
simple but unusual system allows for a dramatic savings in Python
overhead for the construction and processing of orm :class:`.Query`
objects, from query construction up through rendering of a string
SQL statement.
fixes #3054
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
simple but unusual system allows for a dramatic savings in Python
overhead for the construction and processing of orm :class:`.Query`
objects, from query construction up through rendering of a string
SQL statement.
fixes #3054
</pre>
</div>
</content>
</entry>
</feed>
