summaryrefslogtreecommitdiff
path: root/test/perf
Commit message (Collapse)AuthorAgeFilesLines
* Turn on caching everywhere, add loggingMike Bayer2020-06-101-3/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | A variety of caching issues found by running all tests with statement caching turned on. The cache system now has a more conservative approach where any subclass of a SQL element will by default invalidate the cache key unless it adds the flag inherit_cache=True at the class level, or if it implements its own caching. Add working caching to a few elements that were omitted previously; fix some caching implementations to suit lesser used edge cases such as json casts and array slices. Refine the way BaseCursorResult and CursorMetaData interact with caching; to suit cases like Alembic modifying table structures, don't cache the cursor metadata if it were created against a cursor.description using non-positional matching, e.g. "select *". if a table re-ordered its columns or added/removed, now that data is obsolete. Additionally we have to adapt the cursor metadata _keymap regardless of if we just processed cursor.description, because if we ran against a cached SQLCompiler we won't have the right columns in _keymap. Other refinements to how and when we do this adaption as some weird cases were exposed in the Postgresql dialect, a text() construct that names just one column that is not actually in the statement. Fixed that also as it looks like a cut-and-paste artifact that doesn't actually affect anything. Various issues with re-use of compiled result maps and cursor metadata in conjunction with tables being changed, such as change in order of columns. mappers can be cleared but the class remains, meaning a mapper has to use itself as the cache key not the class. lots of bound parameter / literal issues, due to Alembic creating a straight subclass of bindparam that renders inline directly. While we can update Alembic to not do this, we have to assume other people might be doing this, so bindparam() implements the inherit_cache=True logic as well that was a bit involved. turn on cache stats in logging. Includes a fix to subqueryloader which moves all setup to the create_row_processor() phase and elminates any storage within the compiled context. This includes some changes to create_row_processor() signature and a revising of the technique used to determine if the loader can participate in polymorphic queries, which is also applied to selectinloading. DML update.values() and ordered_values() now coerces the keys as we have tests that pass an arbitrary class here which only includes __clause_element__(), so the key can't be cached unless it is coerced. this in turn changed how composite attributes support bulk update to use the standard approach of ClauseElement with annotations that are parsed in the ORM context. memory profiling successfully caught that the Session from Query was getting passed into _statement_20() so that was a big win for that test suite. Apparently Compiler had .execute() and .scalar() methods stuck on it, these date back to version 0.4 and there was a single test in the PostgreSQL dialect tests that exercised it for no apparent reason. Removed these methods as well as the concept of a Compiler holding onto a "bind". Fixes: #5386 Change-Id: I990b43aab96b42665af1b2187ad6020bee778784
* Convert execution to move through SessionMike Bayer2020-05-251-5/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch replaces the ORM execution flow with a single pathway through Session.execute() for all queries, including Core and ORM. Currently included is full support for ORM Query, Query.from_statement(), select(), as well as the baked query and horizontal shard systems. Initial changes have also been made to the dogpile caching example, which like baked query makes use of a new ORM-specific execution hook that replaces the use of both QueryEvents.before_compile() as well as Query._execute_and_instances() as the central ORM interception hooks. select() and Query() constructs alike can be passed to Session.execute() where they will return ORM results in a Results object. This API is currently used internally by Query. Full support for Session.execute()->results to behave in a fully 2.0 fashion will be in later changesets. bulk update/delete with ORM support will also be delivered via the update() and delete() constructs, however these have not yet been adapted to the new system and may follow in a subsequent update. Performance is also beginning to lag as of this commit and some previous ones. It is hoped that a few central functions such as the coercions functions can be rewritten in C to re-gain performance. Additionally, query caching is now available and some subsequent patches will attempt to cache more of the per-execution work from the ORM layer, e.g. column getters and adapters. This patch also contains initial "turn on" of the caching system enginewide via the query_cache_size parameter to create_engine(). Still defaulting at zero for "no caching". The caching system still needs adjustments in order to gain adequate performance. Change-Id: I047a7ebb26aa85dc01f6789fac2bff561dcd555d
* Performance fixes for new result setMike Bayer2020-05-211-4/+20
| | | | | | | | | | | A few small mistakes led to huge callcounts. Additionally, the warn-on-get behavior which is attempting to warn for deprecated access in SQLAlchemy 2.0 is very expensive; it's not clear if its feasible to have this warning or to somehow alter how it works. Fixes: #5340 Change-Id: I73bdd2d7b6f1b25cc0222accabd585cf761a5af4
* Deprecate plain string in execute and introduce `exec_driver_sql`Federico Caselli2020-03-211-3/+3
| | | | | | | | | | | | | | | Execution of literal sql string is deprecated in the :meth:`.Connection.execute` and a warning is raised when used stating that it will be coerced to :func:`.text` in a future release. To execute a raw sql string the new connection method :meth:`.Connection.exec_driver_sql` was added, that will retain the previous behavior, passing the string to the DBAPI driver unchanged. Usage of scalar or tuple positional parameters in :meth:`.Connection.execute` is also deprecated. Fixes: #4848 Fixes: #5178 Change-Id: I2830181054327996d594f7f0d59c157d477c3aa9
* Enable F841Mike Bayer2019-06-201-1/+1
| | | | | | | | | | | This is a very useful assertion which prevents unused variables from being set up allows code to be more readable and sometimes even more efficient. test suites seem to be where the most problems are and there do not seem to be documentation examples that are using this, or at least the linter is not taking effect within rst blocks. Change-Id: I2b3341d8dd14da34879d8425838e66a4b9f8e27d
* Post black reformattingMike Bayer2019-01-062-23/+23
| | | | | | | | | | | | | Applied on top of a pure run of black -l 79 in I7eda77fed3d8e73df84b3651fd6cfcfe858d4dc9, this set of changes resolves all remaining flake8 conditions for those codes we have enabled in setup.cfg. Included are resolutions for all remaining flake8 issues including shadowed builtins, long lines, import order, unused imports, duplicate imports, and docstring issues. Change-Id: I4f72d3ba1380dd601610ff80b8fb06a2aff8b0fe
* Run black -l 79 against all source filesMike Bayer2019-01-062-49/+78
| | | | | | | | | | | | | | This is a straight reformat run using black as is, with no edits applied at all. The black run will format code consistently, however in some cases that are prevalent in SQLAlchemy code it produces too-long lines. The too-long lines will be resolved in the following commit that will resolve all remaining flake8 issues including shadowed builtins, long lines, import order, unused imports, duplicate imports, and docstring issues. Change-Id: I7eda77fed3d8e73df84b3651fd6cfcfe858d4dc9
* print() is a function in Python 3cclauss2018-11-281-2/+4
|
* Make all tests to be PEP8 compliantKhairi Hafsham2017-02-072-20/+28
| | | | | | | | tested using pycodestyle version 2.2.0 Fixes: #3885 Change-Id: I5df43adc3aefe318f9eeab72a078247a548ec566 Pull-request: https://github.com/zzzeek/sqlalchemy/pull/343
* - Added a new extension suite :mod:`sqlalchemy.ext.baked`. ThisMike Bayer2015-03-111-2/+2
| | | | | | | | simple but unusual system allows for a dramatic savings in Python overhead for the construction and processing of orm :class:`.Query` objects, from query construction up through rendering of a string SQL statement. fixes #3054
* - delete the file first here so this is deterministicMike Bayer2014-09-051-0/+3
|
* - stress test for invalidationMike Bayer2014-05-101-0/+51
|
* - add a CLI to determine waht kind of test to runMike Bayer2014-01-111-19/+25
|
* - the wrapped memoized_property here was not working, as the attribute nameMike Bayer2013-11-261-4/+4
| | | | didn't match. use straight memoized_props here for now, add a perf test to check it
* tweaksMike Bayer2013-08-021-6/+3
|
* rework the test here to suit 0.8/0.9 mostly, include time testsMike Bayer2013-08-021-94/+86
|
* no need for all these scripts, not worth maintainingMike Bayer2013-04-2810-1245/+0
|
* - the raw 2to3 runMike Bayer2013-04-278-70/+70
| | | | - went through examples/ and cleaned out excess list() calls
* - remove all compat items that are pre-2.5 (hooray)Mike Bayer2013-03-091-1/+1
| | | | | | - other cleanup - don't need compat.decimal, that approach never panned out. hopefully outside libs aren't pulling it in, they shouldn't be
* The long-deprecated and non-functional ``assert_unicode`` flag onMike Bayer2012-10-251-2/+2
| | | | :func:`.create_engine` as well as :class:`.String` is removed.
* trying different approaches to test layout. in this one, the testing modulesMike Bayer2012-09-276-10/+10
| | | | | | | become an externally usable package but still remains within the main sqlalchemy parent package. in this system, we use kind of an ugly hack to get the noseplugin imported outside of the "sqlalchemy" package, while still making it available within sqlalchemy for usage by third party libraries.
* don't need this is_really_none() thing anymoreMike Bayer2012-08-081-1/+1
|
* -whitespace bonanza, contdMike Bayer2012-07-281-2/+2
|
* - [feature] *Very limited* support forMike Bayer2012-07-141-16/+16
| | | | | | | | | | | | | | inheriting mappers to be GC'ed when the class itself is deferenced. The mapper must not have its own table (i.e. single table inh only) without polymorphic attributes in place. This allows for the use case of creating a temporary subclass of a declarative mapped class, with no table or mapping directives of its own, to be garbage collected when dereferenced by a unit test. [ticket:2526]
* - [feature] The of_type() construct on attributesMike Bayer2012-06-201-1/+3
| | | | | | | | | | | | | | | | | | | | | | | | now accepts aliased() class constructs as well as with_polymorphic constructs, and works with query.join(), any(), has(), and also eager loaders subqueryload(), joinedload(), contains_eager() [ticket:2438] [ticket:1106] - a rewrite of the query path system to use an object based approach for more succinct usage. the system has been designed carefully to not add an excessive method overhead. - [feature] select() features a correlate_except() method, auto correlates all selectables except those passed. Is needed here for the updated any()/has() functionality. - remove some old cruft from LoaderStrategy, init(),debug_callable() - use a namedtuple for _extended_entity_info. This method should become standard within the orm internals - some tweaks to the memory profile tests, number of runs can be customized to work around pysqlite's very annoying behavior - try to simplify PropertyOption._get_paths(), rename to _process_paths(), returns a single list now. overall works more completely as was needed for of_type() functionality
* - remove test.sql._base, test.engine._base, test.orm._base, move those ↵Mike Bayer2011-03-271-4/+4
| | | | | | | classes to a new test.lib.fixtures module - move testing.TestBase to test.lib.fixtures - massive search and replace
* - move _fixtures to work via the normal methods of _base.MappedTest, convertMike Bayer2011-03-261-2/+2
| | | | | | | all referncing tests to not use globals - tests that deal with pickle specifically load the fixture classes from test.lib.pickleable, which gets some more classes added - removed weird sa05 pickling tests that don't matter
* - remove old perf/ tests from pre-2007Mike Bayer2011-01-169-498/+0
|
* - whitespace removal bonanzaMike Bayer2011-01-024-33/+33
|
* - some good inlinings to the whole cascade_iterator() thing.Mike Bayer2010-12-181-0/+1
| | | | | cascade_iterator() should probably not yield the "instance" at all and only deal in states. 30-40K methods taken off the orm2010 test.
* - add a perf script to illustrate 0.7 speed improvementsMike Bayer2010-12-121-0/+185
|
* - support for cdecimalMike Bayer2010-12-111-1/+1
| | | | | | | | | | | | | | | | | | - add --with-cdecimal flag to tests, monkeypatches cdecimal in - fix mssql/pyodbc.py to not use private '_int' accessor in decimal conversion routines - pyodbc version 2.1.8 is needed for cdecimal in any case as previous versions also called '_int', 2.1.8 adds the same string logic as our own dialect, so that logic is skipped for modern pyodbc version - make the imports for "Decimal" consistent across the whole lib. not sure yet how we should be importing "Decimal" or what the best way forward is that would allow a clean user-invoked swap of cdecimal; for now, added docs suggesting a global monkeypatch - the two decimal libs are not compatible with each other so any chance of mixing produces serious issues. adding adapters to DBAPIs tedious and adds in-python overhead. suggestions welcome on how we should be doing Decimal/cdecimal.
* - explain that these are not part of the test suiteMike Bayer2010-12-051-0/+17
|
* - merge default tipMike Bayer2010-12-051-3/+2
|\
| * - SqlSoup overhaulMike Bayer2010-11-241-3/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | - Added "map_to()" method to SqlSoup, which is a "master" method which accepts explicit arguments for each aspect of the selectable and mapping, including a base class per mapping. [ticket:1975] - Mapped selectables used with the map(), with_labels(), join() methods no longer put the given argument into the internal "cache" dictionary. Particularly since the join() and select() objects are created in the method itself this was pretty much a pure memory leaking behavior.
* | - move sqlalchemy.test to test.libMike Bayer2010-11-1512-16/+16
|/
* a little bit of refinementMike Bayer2010-04-171-1/+1
|
* - beef up the --reversetop test option to embed RandomSet throughout the ORMMike Bayer2010-04-151-1/+1
| | | | | | | - with m2m we have to go back to the previous approach of having both sides of the DP fire off, tracking each pair of objects. history may not be consistently present in one side or the other - this revealed a whole lot of issues with self-referential m2m, which are fixed
* - starting to groom the branch for its inclusionMike Bayer2010-04-101-0/+84
| | | | | | | | | | | | | | - one-to-many relationships now maintain a list of positive parent-child associations within the flush, preventing previous parents marked as deleted from cascading a delete or NULL foreign key set on those child objects, despite the end-user not removing the child from the old association. [ticket:1764] - re-established Preprocess as unique on their arguments, as they were definitely duped in inheritance scenarios - added a "memo" feature to UOWTransaction which represents the usual pattern of using the .attributes collection - added the test case from [ticket:1081] into perf/
* - converted all lazy=True|False|None to 'select'|'joined'|'noload'Mike Bayer2010-03-243-5/+5
| | | | | - converted all eager to joined in examples - fixed beaker/advanced.py to reference RelationshipCache
* - To accomodate the fact that there are now two kinds of eagerMike Bayer2010-03-242-5/+5
| | | | | | | | | | | | | | | | loading available, the new names for eagerload() and eagerload_all() are joinedload() and joinedload_all(). The old names will remain as synonyms for the foreseeable future. - The "lazy" flag on the relationship() function now accepts a string argument for all kinds of loading: "select", "joined", "subquery", "noload" and "dynamic", where the default is now "select". The old values of True/ False/None still retain their usual meanings and will remain as synonyms for the foreseeable future. - Added documentation to tutorial,mapper doc, api docs for subqueryload, subqueryload_all, and other options.
* - The official name for the relation() function is nowMike Bayer2010-03-177-16/+16
| | | | | | relationship(), to eliminate confusion over the relational algebra term. relation() however will remain available in equal capacity for the foreseeable future. [ticket:1740]
* timings should use getitem syntax and not the slow getattr oneGaëtan de Menten2010-03-031-4/+4
|
* - Added an optional C extension to speed up the sql layer byGaëtan de Menten2010-02-132-0/+400
| | | | | | | | | | | | | | | reimplementing the highest impact functions. The actual speedups will depend heavily on your DBAPI and the mix of datatypes used in your tables, and can vary from a 50% improvement to more than 200%. It also provides a modest (~20%) indirect improvement to ORM speed for large queries. Note that it is *not* built/installed by default. See README for installation instructions. - The most common result processors conversion function were moved to the new "processors" module. Dialect authors are encouraged to use those functions whenever they correspond to their needs instead of implementing custom ones.
* - reduced a bit of overhead in attribute expiration, particularlyMike Bayer2010-02-131-1/+36
| | | | | | | | the version called by column loaders on an incomplete row (i.e. joined table inheritance). there are more dramatic changes that can be made here but this one is conservative so far as far as how much we're altering how InstanceState tracks "expired" attributes.
* misc cleanups in tests (courtesy of Michael Bayer's revisiting of myGaëtan de Menten2010-02-132-3/+3
| | | | patch)
* - For those who might use debug logging onMike Bayer2010-02-131-2/+2
| | | | | | | | | | sqlalchemy.orm.strategies, most logging calls during row loading have been removed. These were never very helpful and cluttered up the code. - Some internal streamlining of object loading grants a small speedup for large results, estimates are around 10-15%.
* merge 0.6 series to trunk.Mike Bayer2009-08-0612-53/+45
|
* - removed test.testing.ORMTest, test.fixtures, and allMike Bayer2009-06-021-4/+5
| | | | dependencies on those.
* Global propigate -> propagate change to correct spelling. Additionally found ↵Michael Trier2008-11-091-1/+1
| | | | a couple of insures that should be ensure.
View Lorry Controller Status