diff options
| author | Mike Bayer <mike_mp@zzzcomputing.com> | 2016-01-06 12:40:34 -0500 |
|---|---|---|
| committer | Mike Bayer <mike_mp@zzzcomputing.com> | 2016-01-06 12:47:48 -0500 |
| commit | a80bb4e5aabc4850a202f3a4d114c543357e37d5 (patch) | |
| tree | 203dbaaa9002a2b9372dcee0976d7e1c3e0baf6b /doc | |
| parent | 16746dd1a63198e3c27422517fa22ec76f441ceb (diff) | |
| download | sqlalchemy-a80bb4e5aabc4850a202f3a4d114c543357e37d5.tar.gz | |
- Added :class:`.mysql.JSON` for MySQL 5.7. The JSON type provides
persistence of JSON values in MySQL as well as basic operator support
of "getitem" and "getpath", making use of the ``JSON_EXTRACT``
function in order to refer to individual paths in a JSON structure.
fixes #3547
- Added a new type to core :class:`.types.JSON`. This is the
base of the PostgreSQL :class:`.postgresql.JSON` type as well as that
of the new :class:`.mysql.JSON` type, so that a PG/MySQL-agnostic
JSON column may be used. The type features basic index and path
searching support.
fixes #3619
- reorganization of migration docs etc. to try to refer both to
the fixes to JSON that helps Postgresql while at the same time
indicating these are new features of the new base JSON type.
- a rework of the Array/Indexable system some more, moving things
that are specific to Array out of Indexable.
- new operators for JSON indexing added to core so that these can
be compiled by the PG and MySQL dialects individually
- rename sqltypes.Array to sqltypes.ARRAY - as there is no generic
Array implementation, this is an uppercase type for now, consistent
with the new sqltypes.JSON type that is also not a generic implementation.
There may need to be some convention change to handle the case of
datatypes that aren't generic, rely upon DB-native implementations,
but aren't necessarily all named the same thing.
Diffstat (limited to 'doc')
| -rw-r--r-- | doc/build/changelog/changelog_11.rst | 35 | ||||
| -rw-r--r-- | doc/build/changelog/migration_11.rst | 231 | ||||
| -rw-r--r-- | doc/build/core/type_basics.rst | 23 | ||||
| -rw-r--r-- | doc/build/dialects/mysql.rst | 2 | ||||
| -rw-r--r-- | doc/build/dialects/postgresql.rst | 3 |
5 files changed, 185 insertions, 109 deletions
diff --git a/doc/build/changelog/changelog_11.rst b/doc/build/changelog/changelog_11.rst index 83a57ba7c..63e0ca472 100644 --- a/doc/build/changelog/changelog_11.rst +++ b/doc/build/changelog/changelog_11.rst @@ -22,6 +22,33 @@ :version: 1.1.0b1 .. change:: + :tags: feature, mysql + :tickets: 3547 + + Added :class:`.mysql.JSON` for MySQL 5.7. The JSON type provides + persistence of JSON values in MySQL as well as basic operator support + of "getitem" and "getpath", making use of the ``JSON_EXTRACT`` + function in order to refer to individual paths in a JSON structure. + + .. seealso:: + + :ref:`change_3547` + + .. change:: + :tags: feature, sql + :tickets: 3619 + + Added a new type to core :class:`.types.JSON`. This is the + base of the PostgreSQL :class:`.postgresql.JSON` type as well as that + of the new :class:`.mysql.JSON` type, so that a PG/MySQL-agnostic + JSON column may be used. The type features basic index and path + searching support. + + .. seealso:: + + :ref:`change_3619` + + .. change:: :tags: bug, sql :tickets: 3616 @@ -275,7 +302,7 @@ :tickets: 3132 Added support for the SQL-standard function :class:`.array_agg`, - which automatically returns an :class:`.Array` of the correct type + which automatically returns an :class:`.postgresql.ARRAY` of the correct type and supports index / slice operations, as well as :func:`.postgresql.array_agg`, which returns a :class:`.postgresql.ARRAY` with additional comparison features. As arrays are only @@ -292,8 +319,8 @@ :tags: feature, sql :tickets: 3516 - Added a new type to core :class:`.types.Array`. This is the - base of the PostgreSQL :class:`.ARRAY` type, and is now part of Core + Added a new type to core :class:`.types.ARRAY`. This is the + base of the PostgreSQL :class:`.postgresql.ARRAY` type, and is now part of Core to begin supporting various SQL-standard array-supporting features including some functions and eventual support for native arrays on other databases that have an "array" concept, such as DB2 or Oracle. @@ -399,7 +426,7 @@ :tickets: 3514 Additional fixes have been made regarding the value of ``None`` - in conjunction with the Postgresql :class:`.JSON` type. When + in conjunction with the Postgresql :class:`.postgresql.JSON` type. When the :paramref:`.JSON.none_as_null` flag is left at its default value of ``False``, the ORM will now correctly insert the Json "'null'" string into the column whenever the value on the ORM diff --git a/doc/build/changelog/migration_11.rst b/doc/build/changelog/migration_11.rst index b5889c763..70182091c 100644 --- a/doc/build/changelog/migration_11.rst +++ b/doc/build/changelog/migration_11.rst @@ -502,33 +502,138 @@ UNIONs with parenthesized SELECT statements is much less common than the :ticket:`2528` +.. _change_3619: + +JSON support added to Core +-------------------------- + +As MySQL now has a JSON datatype in addition to the Postgresql JSON datatype, +the core now gains a :class:`sqlalchemy.types.JSON` datatype that is the basis +for both of these. Using this type allows access to the "getitem" operator +as well as the "getpath" operator in a way that is agnostic across Postgresql +and MySQL. + +The new datatype also has a series of improvements to the handling of +NULL values as well as expression handling. + +.. seealso:: + + :ref:`change_3547` + + :class:`.types.JSON` + + :class:`.postgresql.JSON` + + :class:`.mysql.JSON` + +:ticket:`3619` + +.. _change_3514: + +JSON "null" is inserted as expected with ORM operations, regardless of column default present +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The :class:`.types.JSON` type and its descendant types :class:`.postgresql.JSON` +and :class:`.mysql.JSON` have a flag :paramref:`.types.JSON.none_as_null` which +when set to True indicates that the Python value ``None`` should translate +into a SQL NULL rather than a JSON NULL value. This flag defaults to False, +which means that the column should *never* insert SQL NULL or fall back +to a default unless the :func:`.null` constant were used. However, this would +fail in the ORM under two circumstances; one is when the column also contained +a default or server_default value, a positive value of ``None`` on the mapped +attribute would still result in the column-level default being triggered, +replacing the ``None`` value:: + + obj = MyObject(json_value=None) + session.add(obj) + session.commit() # would fire off default / server_default, not encode "'none'" + +The other is when the :meth:`.Session.bulk_insert_mappings` +method were used, ``None`` would be ignored in all cases:: + + session.bulk_insert_mappings( + MyObject, + [{"json_value": None}]) # would insert SQL NULL and/or trigger defaults + +The :class:`.types.JSON` type now implements the +:attr:`.TypeEngine.should_evaluate_none` flag, +indicating that ``None`` should not be ignored here; it is configured +automatically based on the value of :paramref:`.types.JSON.none_as_null`. +Thanks to :ticket:`3061`, we can differentiate when the value ``None`` is actively +set by the user versus when it was never set at all. + +If the attribute is not set at all, then column level defaults *will* +fire off and/or SQL NULL will be inserted as expected, as was the behavior +previously. Below, the two variants are illustrated:: + + obj = MyObject(json_value=None) + session.add(obj) + session.commit() # *will not* fire off column defaults, will insert JSON 'null' + + obj = MyObject() + session.add(obj) + session.commit() # *will* fire off column defaults, and/or insert SQL NULL + +The feature applies as well to the new base :class:`.types.JSON` type +and its descendant types. + +:ticket:`3514` + +.. _change_3514_jsonnull: + +New JSON.NULL Constant Added +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +To ensure that an application can always have full control at the value level +of whether a :class:`.types.JSON`, :class:`.postgresql.JSON`, :class:`.mysql.JSON`, +or :class:`.postgresql.JSONB` column +should receive a SQL NULL or JSON ``"null"`` value, the constant +:attr:`.types.JSON.NULL` has been added, which in conjunction with +:func:`.null` can be used to determine fully between SQL NULL and +JSON ``"null"``, regardless of what :paramref:`.types.JSON.none_as_null` is set +to:: + + from sqlalchemy import null + from sqlalchemy.dialects.postgresql import JSON + + obj1 = MyObject(json_value=null()) # will *always* insert SQL NULL + obj2 = MyObject(json_value=JSON.NULL) # will *always* insert JSON string "null" + + session.add_all([obj1, obj2]) + session.commit() + +The feature applies as well to the new base :class:`.types.JSON` type +and its descendant types. + +:ticket:`3514` + .. _change_3516: Array support added to Core; new ANY and ALL operators ------------------------------------------------------ -Along with the enhancements made to the Postgresql :class:`.ARRAY` -type described in :ref:`change_3503`, the base class of :class:`.ARRAY` -itself has been moved to Core in a new class :class:`.types.Array`. +Along with the enhancements made to the Postgresql :class:`.postgresql.ARRAY` +type described in :ref:`change_3503`, the base class of :class:`.postgresql.ARRAY` +itself has been moved to Core in a new class :class:`.types.ARRAY`. Arrays are part of the SQL standard, as are several array-oriented functions such as ``array_agg()`` and ``unnest()``. In support of these constructs for not just PostgreSQL but also potentially for other array-capable backends in the future such as DB2, the majority of array logic for SQL expressions -is now in Core. The :class:`.Array` type still **only works on +is now in Core. The :class:`.types.ARRAY` type still **only works on Postgresql**, however it can be used directly, supporting special array use cases such as indexed access, as well as support for the ANY and ALL:: mytable = Table("mytable", metadata, - Column("data", Array(Integer, dimensions=2)) + Column("data", ARRAY(Integer, dimensions=2)) ) expr = mytable.c.data[5][6] expr = mytable.c.data[5].any(12) -In support of ANY and ALL, the :class:`.Array` type retains the same -:meth:`.Array.Comparator.any` and :meth:`.Array.Comparator.all` methods +In support of ANY and ALL, the :class:`.types.ARRAY` type retains the same +:meth:`.types.ARRAY.Comparator.any` and :meth:`.types.ARRAY.Comparator.all` methods from the PostgreSQL type, but also exports these operations to new standalone operator functions :func:`.sql.expression.any_` and :func:`.sql.expression.all_`. These two functions work in more @@ -541,7 +646,7 @@ as:: For the PostgreSQL-specific operators "contains", "contained_by", and "overlaps", one should continue to use the :class:`.postgresql.ARRAY` -type directly, which provides all functionality of the :class:`.Array` +type directly, which provides all functionality of the :class:`.types.ARRAY` type as well. The :func:`.sql.expression.any_` and :func:`.sql.expression.all_` operators @@ -564,7 +669,7 @@ such as:: New Function features, "WITHIN GROUP", array_agg and set aggregate functions ---------------------------------------------------------------------------- -With the new :class:`.Array` type we can also implement a pre-typed +With the new :class:`.types.ARRAY` type we can also implement a pre-typed function for the ``array_agg()`` SQL function that returns an array, which is now available using :class:`.array_agg`:: @@ -767,8 +872,9 @@ As described in :ref:`change_3499`, the ORM relies upon being able to produce a hash function for column values when a query's selected entities mixes full ORM entities with column expressions. The ``hashable=False`` flag is now correctly set on all of PG's "data structure" types, including -:class:`.ARRAY` and :class:`.JSON`. The :class:`.JSONB` and :class:`.HSTORE` -types already included this flag. For :class:`.ARRAY`, +:class:`.postgresql.ARRAY` and :class:`.postgresql.JSON`. +The :class:`.JSONB` and :class:`.HSTORE` +types already included this flag. For :class:`.postgresql.ARRAY`, this is conditional based on the :paramref:`.postgresql.ARRAY.as_tuple` flag, however it should no longer be necessary to set this flag in order to have an array value present in a composed ORM row. @@ -840,7 +946,7 @@ The JSON cast() operation now requires ``.astext`` is called explicitly As part of the changes in :ref:`change_3503`, the workings of the :meth:`.ColumnElement.cast` operator on :class:`.postgresql.JSON` and :class:`.postgresql.JSONB` no longer implictly invoke the -:attr:`.JSON.Comparator.astext` modifier; Postgresql's JSON/JSONB types +:attr:`.postgresql.JSON.Comparator.astext` modifier; Postgresql's JSON/JSONB types support CAST operations to each other without the "astext" aspect. This means that in most cases, an application that was doing this:: @@ -852,88 +958,6 @@ Will now need to change to this:: expr = json_col['somekey'].astext.cast(Integer) - -.. _change_3514: - -Postgresql JSON "null" is inserted as expected with ORM operations, regardless of column default present ------------------------------------------------------------------------------------------------------------ - -The :class:`.JSON` type has a flag :paramref:`.JSON.none_as_null` which -when set to True indicates that the Python value ``None`` should translate -into a SQL NULL rather than a JSON NULL value. This flag defaults to False, -which means that the column should *never* insert SQL NULL or fall back -to a default unless the :func:`.null` constant were used. However, this would -fail in the ORM under two circumstances; one is when the column also contained -a default or server_default value, a positive value of ``None`` on the mapped -attribute would still result in the column-level default being triggered, -replacing the ``None`` value:: - - obj = MyObject(json_value=None) - session.add(obj) - session.commit() # would fire off default / server_default, not encode "'none'" - -The other is when the :meth:`.Session.bulk_insert_mappings` -method were used, ``None`` would be ignored in all cases:: - - session.bulk_insert_mappings( - MyObject, - [{"json_value": None}]) # would insert SQL NULL and/or trigger defaults - -The :class:`.JSON` type now implements the -:attr:`.TypeEngine.should_evaluate_none` flag, -indicating that ``None`` should not be ignored here; it is configured -automatically based on the value of :paramref:`.JSON.none_as_null`. -Thanks to :ticket:`3061`, we can differentiate when the value ``None`` is actively -set by the user versus when it was never set at all. - -If the attribute is not set at all, then column level defaults *will* -fire off and/or SQL NULL will be inserted as expected, as was the behavior -previously. Below, the two variants are illustrated:: - - obj = MyObject(json_value=None) - session.add(obj) - session.commit() # *will not* fire off column defaults, will insert JSON 'null' - - obj = MyObject() - session.add(obj) - session.commit() # *will* fire off column defaults, and/or insert SQL NULL - -:ticket:`3514` - -.. seealso:: - - :ref:`change_3250` - - :ref:`change_3514_jsonnull` - -.. _change_3514_jsonnull: - -New JSON.NULL Constant Added ----------------------------- - -To ensure that an application can always have full control at the value level -of whether a :class:`.postgresql.JSON` or :class:`.postgresql.JSONB` column -should receive a SQL NULL or JSON ``"null"`` value, the constant -:attr:`.postgresql.JSON.NULL` has been added, which in conjunction with -:func:`.null` can be used to determine fully between SQL NULL and -JSON ``"null"``, regardless of what :paramref:`.JSON.none_as_null` is set -to:: - - from sqlalchemy import null - from sqlalchemy.dialects.postgresql import JSON - - obj1 = MyObject(json_value=null()) # will *always* insert SQL NULL - obj2 = MyObject(json_value=JSON.NULL) # will *always* insert JSON string "null" - - session.add_all([obj1, obj2]) - session.commit() - -.. seealso:: - - :ref:`change_3514` - -:ticket:`3514` - .. _change_2729: ARRAY with ENUM will now emit CREATE TYPE for the ENUM @@ -975,6 +999,25 @@ emits:: Dialect Improvements and Changes - MySQL ============================================= +.. _change_3547: + +MySQL JSON Support +------------------ + +A new type :class:`.mysql.JSON` is added to the MySQL dialect supporting +the JSON type newly added to MySQL 5.7. This type provides both persistence +of JSON as well as rudimentary indexed-access using the ``JSON_EXTRACT`` +function internally. An indexable JSON column that works across MySQL +and Postgresql can be achieved by using the :class:`.types.JSON` datatype +common to both MySQL and Postgresql. + +.. seealso:: + + :ref:`change_3619` + +:ticket:`3547` + + .. _change_mysql_3216: No more generation of an implicit KEY for composite primary key w/ AUTO_INCREMENT diff --git a/doc/build/core/type_basics.rst b/doc/build/core/type_basics.rst index ec3c14dd6..9edba0061 100644 --- a/doc/build/core/type_basics.rst +++ b/doc/build/core/type_basics.rst @@ -38,9 +38,6 @@ database column type available on the target database when issuing a type is emitted in ``CREATE TABLE``, such as ``VARCHAR`` see `SQL Standard Types`_ and the other sections of this chapter. -.. autoclass:: Array - :members: - .. autoclass:: BigInteger :members: @@ -101,12 +98,19 @@ Standard Types`_ and the other sections of this chapter. .. _types_sqlstandard: -SQL Standard Types ------------------- +SQL Standard and Multiple Vendor Types +-------------------------------------- + +This category of types refers to types that are either part of the +SQL standard, or are potentially found within a subset of database backends. +Unlike the "generic" types, the SQL standard/multi-vendor types have **no** +guarantee of working on all backends, and will only work on those backends +that explicitly support them by name. That is, the type will always emit +its exact name in DDL with ``CREATE TABLE`` is issued. -The SQL standard types always create database column types of the same -name when ``CREATE TABLE`` is issued. Some types may not be supported -on all databases. + +.. autoclass:: ARRAY + :members: .. autoclass:: BIGINT @@ -140,6 +144,9 @@ on all databases. .. autoclass:: INT +.. autoclass:: JSON + :members: + .. autoclass:: sqlalchemy.types.INTEGER diff --git a/doc/build/dialects/mysql.rst b/doc/build/dialects/mysql.rst index 33a0d783b..100f2d2e3 100644 --- a/doc/build/dialects/mysql.rst +++ b/doc/build/dialects/mysql.rst @@ -74,6 +74,8 @@ construction arguments, are as follows: .. autoclass:: INTEGER :members: __init__ +.. autoclass:: JSON + :members: .. autoclass:: LONGBLOB :members: __init__ diff --git a/doc/build/dialects/postgresql.rst b/doc/build/dialects/postgresql.rst index 7e2a20ef7..e158cacbf 100644 --- a/doc/build/dialects/postgresql.rst +++ b/doc/build/dialects/postgresql.rst @@ -78,9 +78,6 @@ construction arguments, are as follows: .. autoclass:: JSONB :members: -.. autoclass:: JSONElement - :members: - .. autoclass:: MACADDR :members: __init__ |
