summaryrefslogtreecommitdiff
path: root/lib/sqlalchemy/testing/profiling.py
Commit message (Collapse)AuthorAgeFilesLines
* happy new year 2023Mike Bayer2023-01-031-1/+1
| | | | Change-Id: I625af65b3fb1815b1af17dc2ef47dd697fdc3fb1
* Try running pyupgrade on the codeFederico Caselli2022-11-161-3/+3
| | | | | | | | command run is "pyupgrade --py37-plus --keep-runtime-typing --keep-percent-format <files...>" pyupgrade will change assert_ to assertTrue. That was reverted since assertTrue does not exists in sqlalchemy fixtures Change-Id: Ie1ed2675c7b11d893d78e028aad0d1576baebb55
* inline mypy config; files ignoring type errors for the momentMike Bayer2022-04-281-0/+2
| | | | | | | | | | | | | | | | | | | to simplify pyproject.toml change the remaining files that aren't going to be typed on this first pass (unless of course someone wants to type some of these) to include # mypy: ignore-errors. for the moment, only a handful of ORM modules are to have more type checking implemented. It's important that ignore-errors is used and not "# type: ignore", as in the latter case, mypy doesn't even read the existing types in the file, which makes it impossible to type any files that refer to those modules at all. to simplify ongoing typing work use inline mypy config for remaining files that are "done" for now, indicating the level of type checking they currently have. Change-Id: I98669c1a305c2f0adba85d10b5425541f3fe9533
* pep-484 for sqlalchemy.event; use future annotationsMike Bayer2022-02-151-0/+2
| | | | | | | | | | __future__.annotations mode allows us to use non-string annotations for argument and return types in most cases, but more importantly it removes a large amount of runtime overhead that would be spent in evaluating the annotations. Change-Id: I2f5b6126fe0019713fc50001be3627b664019ede References: #6810
* happy new year 2022Mike Bayer2022-01-061-1/+1
| | | | Change-Id: I49abf2607e0eb0623650efdf0091b1fb3db737ea
* Remove all remaining removed_in_20 warnings slated for removalMike Bayer2022-01-051-10/+7
| | | | | | | | | | | | | | | | | | | | | | | | | Finalize all remaining removed-in-2.0 changes so that we can begin doing pep-484 typing without old things getting in the way (we will also have to do public_factory). note there are a few "moved_in_20()" and "became_legacy_in_20()" warnings still in place. The SQLALCHEMY_WARN_20 variable is now removed. Also removed here are the legacy "in place mutators" for Select statements, and some keyword-only argument signatures in Core have been added. Also in the big change department, the ORM mapper() function is removed entirely; the Mapper class is otherwise unchanged, just the public-facing API function. Mappers are now always given a registry in which to participate, however the argument signature of Mapper is not changed. ideally "registry" would be the first positional argument. Fixes: #7257 Change-Id: Ic70c57b9f1cf7eb996338af5183b11bdeb3e1623
* Clean up most py3k compatFederico Caselli2021-11-241-1/+1
| | | | Change-Id: I8172fdcc3103ff92aa049827728484c8779af6b7
* Remove object in class definitionFederico Caselli2021-11-221-1/+1
| | | | | References: #4600 Change-Id: I2a62ddfe00bc562720f0eae700a497495d7a987a
* removals: all unicode encoding / decodingMike Bayer2021-11-101-5/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Removed here includes: * convert_unicode parameters * encoding create_engine() parameter * description encoding support * "non-unicode fallback" modes under Python 2 * String symbols regarding Python 2 non-unicode fallbacks * any concept of DBAPIs that don't accept unicode statements, unicode bound parameters, or that return bytes for strings anywhere except an explicit Binary / BLOB type * unicode processors in Python / C Risk factors: * Whether all DBAPIs do in fact return Unicode objects for all entries in cursor.description now * There was logic for mysql-connector trying to determine description encoding. A quick test shows Unicode coming back but it's not clear if there are still edge cases where they return bytes. if so, these are bugs in that driver, and at most we would only work around it in the mysql-connector DBAPI itself (but we won't do that either). * It seems like Oracle 8 was not expecting unicode bound parameters. I'm assuming this was all Python 2 stuff and does not apply for modern cx_Oracle under Python 3. * third party dialects relying upon built in unicode encoding/decoding but it's hard to imagine any non-SQLAlchemy database driver not dealing exclusively in Python unicode strings in Python 3 Change-Id: I97d762ef6d4dd836487b714d57d8136d0310f28a References: #7257
* 2.0 removals: LegacyRow, connectionless execution, close_with_resultMike Bayer2021-10-311-6/+1
| | | | | | | | | | | | | | | | | | | in order to remove LegacyRow / LegacyResult, we have to also lose close_with_result, which connectionless execution relies upon. also includes a new profiles.txt file that's all against py310, as that's what CI is on now. some result counts changed by one function call which was enough to fail the low-count result tests. Replaces Connectable as the common interface between Connection and Engine with EngineEventsTarget. Engine is no longer Connectable. Connection and MockConnection still are. References: #7257 Change-Id: Iad5eba0313836d347e65490349a22b061356896a
* Replace all http:// links to https://Federico Caselli2021-07-041-1/+1
| | | | | | Also replace http://pypi.python.org/pypi with https://pypi.org/project Change-Id: I84b5005c39969a82140706472989f2a30b0c7685
* Minor optimization to the codeFederico Caselli2021-02-181-1/+2
| | | | | | | | | * remove the c version of distill params since it's actually slower than the python one * add a function to langhelpers to check if the cextensions are active * minor cleanup to the OrderedSet implementation Change-Id: Iec3d0c3f0f42cdf51f802aaca342ba37b8783b85
* happy new yearMike Bayer2021-01-041-1/+1
| | | | Change-Id: Ic5bb19ca8be3cb47c95a0d3315d84cb484bac47c
* Create a framework to allow all SQLALCHEMY_WARN_20 to passMike Bayer2020-09-161-8/+17
| | | | | | | | | | | | | | | | | | | | | As the test suite has widespread use of many patterns that are deprecated, enable SQLALCHEMY_WARN_20 globally for the test suite but then break the warnings filter out into a whole list of all the individual warnings we are looking for. this way individual changesets can target a specific class of warning, as many of these warnings will indivdidually affect dozens of files and potentially hundreds of lines of code. Many warnings are also resolved here as this patch started out that way. From this point forward there should be changesets that target a subset of the warnings at a time. For expediency, updates some migration 2.0 docs for ORM as well. Change-Id: I98b8defdf7c37b818b3824d02f7668e3f5f31c94
* Add complete platform data to profiling dataFederico Caselli2020-08-071-12/+7
| | | | | | | | | Initially to distinsuish between arm and x86_64 architecture, expand out the profile key to include machine, system, python impl in all cases. Ref: #5436 Change-Id: I7e48f0462ba7d9c680b2dac45ce7b0cf709b9b22
* Make call-count profiling tests on osx have their own platform keyFederico Caselli2020-07-111-0/+3
| | | | | | | They previously would use the linux profiles, but recently some discrepancies in the function call count on osx would make the tests fail. Change-Id: Ifdfdca1676972de4179f59cdaae196f6805d4a21
* Fix a wide variety of typos and broken linksaplatkouski2020-06-251-1/+1
| | | | | | | | | | | | Note the PR has a few remaining doc linking issues listed in the comment that must be addressed separately. Signed-off-by: aplatkouski <5857672+aplatkouski@users.noreply.github.com> Closes: #5371 Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/5371 Pull-request-sha: 7e7d233cf3a0c66980c27db0fcdb3c7d93bc2510 Change-Id: I9c36e8d8804483950db4b42c38ee456e384c59e3
* Add full profile sort, dumping to profile resultsMike Bayer2020-05-281-2/+10
| | | | Change-Id: Ib256ae34de15d29ee9a48e3be86073610f8d1a65
* Convert execution to move through SessionMike Bayer2020-05-251-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch replaces the ORM execution flow with a single pathway through Session.execute() for all queries, including Core and ORM. Currently included is full support for ORM Query, Query.from_statement(), select(), as well as the baked query and horizontal shard systems. Initial changes have also been made to the dogpile caching example, which like baked query makes use of a new ORM-specific execution hook that replaces the use of both QueryEvents.before_compile() as well as Query._execute_and_instances() as the central ORM interception hooks. select() and Query() constructs alike can be passed to Session.execute() where they will return ORM results in a Results object. This API is currently used internally by Query. Full support for Session.execute()->results to behave in a fully 2.0 fashion will be in later changesets. bulk update/delete with ORM support will also be delivered via the update() and delete() constructs, however these have not yet been adapted to the new system and may follow in a subsequent update. Performance is also beginning to lag as of this commit and some previous ones. It is hoped that a few central functions such as the coercions functions can be rewritten in C to re-gain performance. Additionally, query caching is now available and some subsequent patches will attempt to cache more of the per-execution work from the ORM layer, e.g. column getters and adapters. This patch also contains initial "turn on" of the caching system enginewide via the query_cache_size parameter to create_engine(). Still defaulting at zero for "no caching". The caching system still needs adjustments in order to gain adequate performance. Change-Id: I047a7ebb26aa85dc01f6789fac2bff561dcd555d
* Rename py.test to pytestGord Thompson2020-04-161-1/+1
| | | | Change-Id: I431e1ef41e26d490343204a75a5c097768749768
* Fix tests failing for SQLite file databases; repair provisioningGord Thompson2020-03-131-0/+5
| | | | | | | | | | | | | | | | | | | | | | 1. ensure provision.py loads dialect implementations when running reap_dbs.py. Reapers haven't been working since 598f2f7e557073f29563d4d567f43931fc03013f . 2. add some exclusion rules to allow the sqlite_file target to work; add to tox. 3. add reap dbs target for SQLite, repair SQLite drop_db routine which also wasn't doing the right thing for memory databases etc. 4. Fix logging in provision files, as the main provision logger is the one that's enabled by reap_dbs and maybe others, have all the provision files use the provision logger. Fixes: #5180 Fixes: #5168 Change-Id: Ibc1b0106394d20f5bcf847f37b09d185f26ac9b5
* Result initial introductionMike Bayer2020-02-211-1/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This builds on cc718cccc0bf8a01abdf4068c7ea4f3 which moved RowProxy to Row, allowing Row to be more like a named tuple. - KeyedTuple in ORM is replaced with Row - ResultSetMetaData broken out into "simple" and "cursor" versions for ORM and Core, as well as LegacyCursor version. - Row now has _mapping attribute that supplies full mapping behavior. Row and SimpleRow both have named tuple behavior otherwise. LegacyRow has some mapping features on the tuple which emit deprecation warnings (e.g. keys(), values(), etc). the biggest change for mapping->tuple is the behavior of __contains__ which moves from testing of "key in row" to "value in row". - ResultProxy breaks into ResultProxy and FutureResult (interim), the latter has the newer APIs. Made available to dialects using execution options. - internal reflection methods and most tests move off of implicit Row mapping behavior and move to row._mapping, result.mappings() method using future result - a new strategy system for cursor handling replaces the various subclasses of RowProxy - some execution context adjustments. We will leave EC in but refined things like get_result_proxy() and out parameter handling. Dialects for 1.4 will need to adjust from get_result_proxy() to get_result_cursor_strategy(), if they are using this method - out parameter handling now accommodated by get_out_parameter_values() EC method. Oracle changes for this. external dialect for DB2 for example will also need to adjust for this. - deprecate case_insensitive flag for engine / result, this feature is not used mapping-methods on Row are deprecated, and replaced with Row._mapping.<meth>, including: row.keys() -> use row._mapping.keys() row.items() -> use row._mapping.items() row.values() -> use row._mapping.values() key in row -> use key in row._mapping int in row -> use int < len(row) Fixes: #4710 Fixes: #4878 Change-Id: Ieb9085e9bcff564359095b754da9ae0af55679f0
* Limit non-backend critical profiling tests to SQLiteMike Bayer2020-02-171-1/+4
| | | | | | | | | | | | | | | | | | | | issues with backend-specific profiling should be limited to tests that are explcitly against resultset, compiler, etc. MySQL in particular has an often varying callcount that isn't worth running these tests against nor is it worth profiling them for other backends like Oracle and SQL Server. Also add the REQUIRE_SQLALCHEMY_CEXT flag to the regen_callcounts.tox.ini script, which is part of some review somewhere but is needed here to generate callcounts correctly. Add a "warmup" phase for some of the ORM tests for join conditions that have varying profile counts based on whether mappings have been used already or not; profiling should always be against the "warmed up" version of a function. Change-Id: If483820235fa4cc4360cbd067a9b68d83512d587
* Remove jython code, remove all jython / pypy symbolsMike Bayer2020-01-171-7/+7
| | | | | | | | | | | | | | | | | | | | | | Removed all dialect code related to support for Jython and zxJDBC. Jython has not been supported by SQLAlchemy for many years and it is not expected that the current zxJDBC code is at all functional; for the moment it just takes up space and adds confusion by showing up in documentation. At the moment, it appears that Jython has achieved Python 2.7 support in its releases but not Python 3. If Jython were to be supported again, the form it should take is against the Python 3 version of Jython, and the various zxJDBC stubs for various backends should be implemented as a third party dialect. Additionally modernized logic that distinguishes between "cpython" and "pypy" to instead look at platform.python_distribution() which reliably tells us if we are cPython or not; all booleans which previously checked for pypy and sometimes jython are now converted to be "not cpython", this impacts the test suite for tests that are cPython centric. Fixes: #5094 Change-Id: I226cb55827f997daf6b4f4a755c18e7f4eb8d9ad
* Fix cext for Python 2; ensure C extensions build successfullyMike Bayer2020-01-041-10/+13
| | | | | | | | | | | | | | | | | | | | The C extensions have been broken since cc718cccc0bf8a01abdf4068c however CI did not find this, because the build degraded to non-C extensions without failing. Ensure that if cext is set, there is no fallback to non-cext build if the C extension build fails. Repair C related issues introduced in cc718cccc0bf8a01abdf4068c. As C extensions have been silently failing on 2.7 for some commits, the callcounts also needed to be adjusted for recent performance-related changes. That in turn required a fix to the profiling decorator to use signature rewriting in order to support py.test's fixture mechanism under Python 2, usage introduced under profiling in 89bf6d80a9. Fixes: #5076 Change-Id: Id968f10c85d6bf489298b1c318a1f869ad3e7d80
* happy new yearMike Bayer2020-01-011-1/+1
| | | | Change-Id: I08440dc25e40ea1ccea1778f6ee9e28a00808235
* Don't apply aliasing + adaption for simple relationship joinsMike Bayer2019-12-191-2/+5
| | | | | | | | | | | Identified a performance issue in the system by which a join is constructed based on a mapped relationship. The clause adaption system would be used for the majority of join expressions including in the common case where no adaptation is needed. The conditions under which this adaptation occur have been refined so that average non-aliased joins along a simple relationship without a "secondary" table use about 70% less function calls. Change-Id: Ifbe04214576e5a9fac86ca80c1dc7145c27cd50a
* Add profile sort option to test suiteMike Bayer2019-07-141-2/+3
| | | | | | | | I use the nfl sort a lot to see what calling changes are happening between two versions in order to identify the offending code, so add it as a command line option. Change-Id: Ia1ab6dd98012a78298b325bb5c7c050fa9b767c2
* profiling plugin fixesMike Bayer2019-07-111-6/+41
| | | | | | | | support that we have pytest-xdist arguments now, as well as allow force-write to reset the list of callcounts which seem to accumulate commas for some reason Change-Id: I54dd164c21ffbb9139937d5c3ffb1df7e9598594
* happy new yearMike Bayer2019-01-111-1/+1
| | | | Change-Id: I6a71f4924d046cf306961c58dffccf21e9c03911
* Post black reformattingMike Bayer2019-01-061-5/+10
| | | | | | | | | | | | | Applied on top of a pure run of black -l 79 in I7eda77fed3d8e73df84b3651fd6cfcfe858d4dc9, this set of changes resolves all remaining flake8 conditions for those codes we have enabled in setup.cfg. Included are resolutions for all remaining flake8 issues including shadowed builtins, long lines, import order, unused imports, duplicate imports, and docstring issues. Change-Id: I4f72d3ba1380dd601610ff80b8fb06a2aff8b0fe
* Run black -l 79 against all source filesMike Bayer2019-01-061-36/+35
| | | | | | | | | | | | | | This is a straight reformat run using black as is, with no edits applied at all. The black run will format code consistently, however in some cases that are prevalent in SQLAlchemy code it produces too-long lines. The too-long lines will be resolved in the following commit that will resolve all remaining flake8 issues including shadowed builtins, long lines, import order, unused imports, duplicate imports, and docstring issues. Change-Id: I7eda77fed3d8e73df84b3651fd6cfcfe858d4dc9
* Assorted pre-Black fixesMike Bayer2019-01-051-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixes to the test suite, a few errant imports, and setup.py: - mysql and postgresql have unused 'json' imports; remove - postgresql is exporting the 'json' symbol, remove - make sure setup.py can find __version__ using " or ' - retry logic in provision create database for postgresql fixed - refactor test_magazine to use cls.tables rather than globals - remove unused class in test_scoping - add a comment to test_deprecations that this test suite itself is deprecated - don't use mapper() and orm_mapper() in test_unitofwork, just use mapper() - remove dupe test_scalar_set_None test in test_attributes - Python 2.7 and above includes unittest.SkipTest, remove pre-2.7 fallback - use imported SkipTest in profiling - declarative test_reflection tests with "reflectable_autoincrement" already don't run on oracle or firebird; remove conditional logic for these, which also removes an "id" symbol - clean up test in test_functions, remove print statement - remove dupe test_literal_processor_coercion_native_int_out_of_range in test/sql/test_types.py - fix psycopg2_hstore ref Change-Id: I7b3444f8546aac82be81cd1e7b6d8b2ad6834fe6
* happy new yearMike Bayer2018-01-121-1/+1
| | | | Change-Id: I3ef36bfd0cb0ba62b3123c8cf92370a43156cf8f
* update for 2017 copyrightMike Bayer2017-01-041-1/+1
| | | | Change-Id: I4e8c2aa8fe817bb2af8707410fa0201f938781de
* - happy new yearMike Bayer2016-01-291-1/+1
|
* - add a new qualifier "native" vs. "dbapi" unicode to avoidMike Bayer2016-01-241-0/+5
| | | | | | profiling problems here - add extras_require to setup.py for the most common DBAPIs - rework tox.ini to use extras, specify a test matrix built in
* - move away from explicit raises of SkipTest, instead call aMike Bayer2015-05-011-5/+5
| | | | | | | function patched onto config. nose/pytest backends now fill in their exception class here only when loaded - use more public seeming api to get at py.test Skipped exception
* - copyright 2015Mike Bayer2015-03-101-1/+1
|
* - more callcountsMike Bayer2015-01-051-2/+4
| | | | - add the platform key to the error output
* - add a new option --force-write-profiles to rewrite profiles even if theyMike Bayer2014-11-251-2/+6
| | | | are passing
* - correct this to rewrite a multiple profile line correctlyMike Bayer2014-11-141-1/+5
|
* - rework profiling, zoomark tests into single tests so thatMike Bayer2014-08-161-138/+78
| | | | they can be used under xdist
* - apply pep8 formatting to sqlalchemy/sql, sqlalchemy/util, sqlalchemy/dialects,Brian Jarrett2014-07-201-30/+30
| | | | sqlalchemy/orm, sqlalchemy/event, sqlalchemy/testing
* - break up the <authors> copyright comment as part of a passMike Bayer2014-07-091-1/+2
| | | | to get all flake8 passing
* fixes to get profiling tests working againMike Bayer2014-03-261-1/+1
|
* - remove nose dependencyMike Bayer2014-03-261-1/+1
|
* - fix some function mismatch in profilingMike Bayer2014-01-091-6/+3
|
* - happy new yearMike Bayer2014-01-051-1/+1
|
* - add copyright to source files missing itMike Bayer2013-10-261-0/+6
|
View Lorry Controller Status