- add support for pickling with mutable scalars, mutable composites

- add pickle/unpickle events to ORM events.  these are needed
for the mutable extension.
- finish mutable extension documentation, consolidate examples,
add full descriptions

4 files changed, 398 insertions, 114 deletions

diff --git a/lib/sqlalchemy/ext/mutable.py b/lib/sqlalchemy/ext/mutable.py
index 11a7977f6..594e664fb 100644
--- a/lib/sqlalchemy/ext/mutable.py
+++ b/lib/sqlalchemy/ext/mutable.py
@@ -5,12 +5,310 @@
 # the MIT License: http://www.opensource.org/licenses/mit-license.php
 
 """Provide support for tracking of in-place changes to scalar values,
-which are propagated to owning parent objects.
+which are propagated into ORM change events on owning parent objects.
 
-The ``mutable`` extension is a replacement for the :class:`.types.MutableType`
-class as well as the ``mutable=True`` flag available on types which subclass
-it.
+The :mod:`sqlalchemy.ext.mutable` extension replaces SQLAlchemy's legacy approach to in-place
+mutations of scalar values, established by the :class:`.types.MutableType`
+class as well as the ``mutable=True`` type flag, with a system that allows
+change events to be propagated from the value to the owning parent, thereby
+removing the need for the ORM to maintain copies of values as well as the very
+expensive requirement of scanning through all "mutable" values on each flush
+call, looking for changes.
 
+.. _mutable_scalars:
+
+Establishing Mutability on Scalar Column Values
+===============================================
+
+A typical example of a "mutable" structure is a Python dictionary.
+Following the example introduced in :ref:`types_toplevel`, we 
+begin with a custom type that marshals Python dictionaries into 
+JSON strings before being persisted::
+
+    from sqlalchemy.types import TypeDecorator, VARCHAR
+    import json
+
+    class JSONEncodedDict(TypeDecorator):
+        "Represents an immutable structure as a json-encoded string."
+
+        impl = VARCHAR
+
+        def process_bind_param(self, value, dialect):
+            if value is not None:
+                value = json.dumps(value)
+            return value
+
+        def process_result_value(self, value, dialect):
+            if value is not None:
+                value = json.loads(value)
+            return value
+
+The usage of ``json`` is only for the purposes of example. The :mod:`sqlalchemy.ext.mutable` 
+extension can be used
+with any type whose target Python type may be mutable, including
+:class:`.PickleType`, :class:`.postgresql.ARRAY`, etc.
+
+When using the :mod:`sqlalchemy.ext.mutable` extension, the value itself
+tracks all parents which reference it.  Here we will replace the usage
+of plain Python dictionaries with a dict subclass that implements
+the :class:`.Mutable` mixin::
+
+    import collections
+    from sqlalchemy.ext.mutable import Mutable
+
+    class MutationDict(Mutable, collections.MutableMapping, dict):
+        @classmethod
+        def coerce(cls, key, value):
+            "Convert plain dictionaries to MutationDict."
+
+            if not isinstance(value, MutationDict):
+                if isinstance(value, dict):
+                    return MutationDict(value)
+
+                # this call will raise ValueError
+                return Mutable.coerce(key, value)
+            else:
+                return value
+
+        def __setitem__(self, key, value):
+            "Detect dictionary set events and emit change events."
+
+            dict.__setitem__(self, key, value)
+            self.changed()
+
+        def __delitem__(self, key):
+            "Detect dictionary del events and emit change events."
+
+            dict.__delitem__(self, key)
+            self.changed()
+
+The above dictionary class takes the approach of subclassing the Python
+built-ins ``collections.MutableMapping`` and ``dict`` to produce a dict
+subclass which routes all mutation events through ``__setitem__``. There are
+many variants on this approach, such as subclassing ``UserDict.UserDict``,
+etc. The part that's important to this example is that the
+:meth:`.Mutable.changed` method is called whenever an in-place change to the
+datastructure takes place.
+
+We also redefine the :meth:`.Mutable.coerce` method which will be used to
+convert any values that are not instances of ``MutationDict``, such
+as the plain dictionaries returned by the ``json`` module, into the
+appropriate type.  Defining this method is optional; we could just as well created our
+``JSONEncodedDict`` such that it always returns an instance of ``MutationDict``,
+and additionally ensured that all calling code uses ``MutationDict`` 
+explicitly.  When :meth:`.Mutable.coerce` is not overridden, any values
+applied to a parent object which are not instances of the mutable type
+will raise a ``ValueError``.
+
+Our new ``MutationDict`` type offers a class method
+:meth:`~.Mutable.associate_with` which we can use within column metadata
+to associate with types. This method grabs the given type object and
+associates a listener that will detect all future mappings of this type,
+applying event listening instrumentation to the mapped attribute. Such as,
+with classical table metadata::
+
+    from sqlalchemy import Table, Column, Integer
+
+    my_data = Table('my_data', metadata,
+        Column('id', Integer, primary_key=True),
+        Column('data', MutationDict.associate_with(JSONEncodedDict))
+    )
+
+Above, :meth:`~.Mutable.associate_with` returns an instance of ``JSONEncodedDict``
+(if the type object was not an instance already), which will intercept any 
+attributes which are mapped against this type.  Below we establish a simple
+mapping against the ``my_data`` table::
+
+    from sqlalchemy import mapper
+
+    class MyDataClass(object):
+        pass
+
+    # associates mutation listeners with MyDataClass.data
+    mapper(MyDataClass, my_data)
+
+The ``MyDataClass.data`` member will now be notified of in place changes
+to its value.
+
+There's no difference in usage when using declarative::
+
+    from sqlalchemy.ext.declarative import declarative_base
+
+    Base = declarative_base()
+
+    class MyDataClass(Base):
+        __tablename__ = 'my_data'
+        id = Column(Integer, primary_key=True)
+        data = Column(MutationDict.associate_with(JSONEncodedDict))
+
+Any in-place changes to the ``MyDataClass.data`` member
+will flag the attribute as "dirty" on the parent object::
+
+    >>> from sqlalchemy.orm import Session
+
+    >>> sess = Session()
+    >>> m1 = MyDataClass(data={'value1':'foo'})
+    >>> sess.add(m1)
+    >>> sess.commit()
+
+    >>> m1.data['value1'] = 'bar'
+    >>> assert m1 in sess.dirty
+    True
+
+Supporting Pickling
+--------------------
+
+The key to the :mod:`sqlalchemy.ext.mutable` extension relies upon the
+placement of a ``weakref.WeakKeyDictionary`` upon the value object, which
+stores a mapping of parent mapped objects keyed to the attribute name under
+which they are associated with this value. ``WeakKeyDictionary`` objects are
+not picklable, due to the fact that they contain weakrefs and function
+callbacks. In our case, this is a good thing, since if this dictionary were
+picklable, it could lead to an excessively large pickle size for our value
+objects that are pickled by themselves outside of the context of the parent.
+The developer responsiblity here is only to provide a ``__getstate__`` method
+that excludes the :meth:`~.MutableBase._parents` collection from the pickle
+stream::
+
+    class MyMutableType(Mutable):
+        def __getstate__(self):
+            d = self.__dict__.copy()
+            d.pop('_parents', None)
+            return d
+
+With our dictionary example, we need to return the contents of the dict itself
+(and also restore them on __setstate__)::
+
+    class MutationDict(Mutable, collections.MutableMapping, dict):
+        # ....
+
+        def __getstate__(self):
+            return dict(self)
+
+        def __setstate__(self, state):
+            self.update(state)
+
+In the case that our mutable value object is pickled as it is attached to one
+or more parent objects that are also part of the pickle, the :class:`.Mutable`
+mixin will re-establish the :attr:`.Mutable._parents` collection on each value
+object as the owning parents themselves are unpickled.
+
+.. _mutable_composites:
+
+Establishing Mutability on Composites
+=====================================
+
+Composites are a special ORM feature which allow a single scalar attribute to
+be assigned an object value which represents information "composed" from one
+or more columns from the underlying mapped table. The usual example is that of
+a geometric "point", and is introduced in :ref:`mapper_composite`.
+
+As of SQLAlchemy 0.7, the internals of :func:`.orm.composite` have been
+greatly simplified and in-place mutation detection is no longer enabled by
+default; instead, the user-defined value must detect changes on its own and
+propagate them to all owning parents. The :mod:`sqlalchemy.ext.mutable`
+extension provides the helper class :class:`.MutableComposite`, which is a
+slight variant on the :class:`.Mutable` class.
+
+As is the case with :class:`.Mutable`, the user-defined composite class
+subclasses :class:`.MutableComposite` as a mixin, and detects and delivers
+change events to its parents via the :meth:`.MutableComposite.changed` method.
+In the case of a composite class, the detection is usually via the usage of
+Python descriptors (i.e. ``@property``), or alternatively via the special
+Python method ``__setattr__()``. Below we expand upon the ``Point`` class
+introduced in :ref:`mapper_composite` to subclass :class:`.MutableComposite`
+and to also route attribute set events via ``__setattr__`` to the
+:meth:`.MutableComposite.changed` method::
+
+    from sqlalchemy.ext.mutable import MutableComposite
+
+    class Point(MutableComposite):
+        def __init__(self, x, y):
+            self.x = x
+            self.y = y
+
+        def __setattr__(self, key, value):
+            "Intercept set events"
+
+            # set the attribute
+            object.__setattr__(self, key, value)
+
+            # alert all parents to the change
+            self.changed()
+
+        def __composite_values__(self):
+            return self.x, self.y
+
+        def __eq__(self, other):
+            return isinstance(other, Point) and \\
+                other.x == self.x and \\
+                other.y == self.y
+
+        def __ne__(self, other):
+            return not self.__eq__(other)
+
+The :class:`.MutableComposite` class uses a Python metaclass to automatically
+establish listeners for any usage of :func:`.orm.composite` that specifies our
+``Point`` type. Below, when ``Point`` is mapped to the ``Vertex`` class,
+listeners are established which will route change events from ``Point``
+objects to each of the ``Vertex.start`` and ``Vertex.end`` attributes::
+
+    from sqlalchemy.orm import composite, mapper
+    from sqlalchemy import Table, Column
+
+    vertices = Table('vertices', metadata,
+        Column('id', Integer, primary_key=True),
+        Column('x1', Integer),
+        Column('y1', Integer),
+        Column('x2', Integer),
+        Column('y2', Integer),
+        )
+
+    class Vertex(object):
+        pass
+
+    mapper(Vertex, vertices, properties={
+        'start': composite(Point, vertices.c.x1, vertices.c.y1),
+        'end': composite(Point, vertices.c.x2, vertices.c.y2)
+    })
+
+Any in-place changes to the ``Vertex.start`` or ``Vertex.end`` members
+will flag the attribute as "dirty" on the parent object::
+
+    >>> from sqlalchemy.orm import Session
+
+    >>> sess = Session()
+    >>> v1 = Vertex(start=Point(3, 4), end=Point(12, 15))
+    >>> sess.add(v1)
+    >>> sess.commit()
+
+    >>> v1.end.x = 8
+    >>> assert v1 in sess.dirty
+    True
+
+Supporting Pickling
+--------------------
+
+As is the case with :class:`.Mutable`, the :class:`.MutableComposite` helper
+class uses a ``weakref.WeakKeyDictionary`` available via the
+:meth:`.MutableBase._parents` attribute which isn't picklable. If we need to
+pickle instances of ``Point`` or its owning class ``Vertex``, we at least need
+to define a ``__getstate__`` that doesn't include the ``_parents`` dictionary.
+Below we define both a ``__getstate__`` and a ``__setstate__`` that package up
+the minimal form of our ``Point`` class::
+
+    class Point(MutableComposite):
+        # ...
+        
+        def __getstate__(self):
+            return self.x, self.y
+        
+        def __setstate__(self, state):
+            self.x, self.y = state
+
+As with :class:`.Mutable`, the :class:`.MutableComposite` augments the
+pickling process of the parent's object-relational state so that the
+:meth:`.MutableBase._parents` collection is restored to all ``Point`` objects.
 
 """
 from sqlalchemy.orm.attributes import flag_modified
@@ -19,37 +317,23 @@ from sqlalchemy.orm import mapper, object_mapper
 from sqlalchemy.util import memoized_property
 import weakref
 
-class Mutable(object):
-    """Mixin that defines transparent propagation of change
-    events to a parent object.
-
-    """
+class MutableBase(object):
+    """Common base class to :class:`.Mutable` and :class:`.MutableComposite`."""
 
     @memoized_property
     def _parents(self):
-        """Dictionary of parent object->attribute name on the parent."""
-
-        return weakref.WeakKeyDictionary()
-
-    def change(self):
-        """Subclasses should call this method whenever change events occur."""
-
-        for parent, key in self._parents.items():
-            flag_modified(parent, key)
-
-    @classmethod
-    def coerce(cls, key, value):
-        """Given a value, coerce it into this type.
-
-        By default raises ValueError.
+        """Dictionary of parent object->attribute name on the parent.
+        
+        This attribute is a so-called "memoized" property.  It initializes
+        itself with a new ``weakref.WeakKeyDictionary`` the first time
+        it is accessed, returning the same object upon subsequent access.
+        
         """
-        if value is None:
-            return None
-        raise ValueError("Attribute '%s' accepts objects of type %s" % (key, cls))
 
+        return weakref.WeakKeyDictionary()
 
     @classmethod
-    def associate_with_attribute(cls, attribute):
+    def _listen_on_attribute(cls, attribute, coerce):
         """Establish this type as a mutation listener for the given 
         mapped descriptor.
 
@@ -66,8 +350,9 @@ class Mutable(object):
             """
             val = state.dict.get(key, None)
             if val is not None:
-                val = cls.coerce(key, val)
-                state.dict[key] = val
+                if coerce:
+                    val = cls.coerce(key, val)
+                    state.dict[key] = val
                 val._parents[state.obj()] = key
 
         def set(target, value, oldvalue, initiator):
@@ -87,11 +372,55 @@ class Mutable(object):
                 oldvalue._parents.pop(state.obj(), None)
             return value
 
+        def pickle(state, state_dict):
+            val = state.dict.get(key, None)
+            if val is not None:
+                if 'ext.mutable.values' not in state_dict:
+                    state_dict['ext.mutable.values'] = []
+                state_dict['ext.mutable.values'].append(val)
+
+        def unpickle(state, state_dict):
+            if 'ext.mutable.values' in state_dict:
+                for val in state_dict['ext.mutable.values']:
+                    val._parents[state.obj()] = key
+
         event.listen(parent_cls, 'load', load, raw=True)
         event.listen(parent_cls, 'refresh', load, raw=True)
         event.listen(attribute, 'set', set, raw=True, retval=True)
+        event.listen(parent_cls, 'pickle', pickle, raw=True)
+        event.listen(parent_cls, 'unpickle', unpickle, raw=True)
+
+class Mutable(MutableBase):
+    """Mixin that defines transparent propagation of change
+    events to a parent object.
+
+    See the example in :ref:`mutable_scalars` for usage information.
+
+    """
+
+    def changed(self):
+        """Subclasses should call this method whenever change events occur."""
+
+        for parent, key in self._parents.items():
+            flag_modified(parent, key)
+
+    @classmethod
+    def coerce(cls, key, value):
+        """Given a value, coerce it into this type.
+
+        By default raises ValueError.
+        """
+        if value is None:
+            return None
+        raise ValueError("Attribute '%s' accepts objects of type %s" % (key, cls))
+
+    @classmethod
+    def associate_with_attribute(cls, attribute):
+        """Establish this type as a mutation listener for the given 
+        mapped descriptor.
 
-        # TODO: need a deserialize hook here
+        """
+        cls._listen_on_attribute(attribute, True)
 
     @classmethod
     def associate_with(cls, sqltype):
@@ -161,46 +490,18 @@ class Mutable(object):
 
         return sqltype
 
-
 class _MutableCompositeMeta(type):
     def __init__(cls, classname, bases, dict_):
         cls._setup_listeners()
         return type.__init__(cls, classname, bases, dict_)
 
-class MutableComposite(object):
+class MutableComposite(MutableBase):
     """Mixin that defines transparent propagation of change
     events on a SQLAlchemy "composite" object to its
     owning parent or parents.
-
-    Composite classes, in addition to meeting the usage contract
-    defined in :ref:`mapper_composite`, also define some system
-    of relaying change events to the given :meth:`.change` 
-    method, which will notify all parents of the change.  Below
-    the special Python method ``__setattr__`` is used to intercept
-    all changes::
-
-        class Point(MutableComposite):
-            def __init__(self, x, y):
-                self.x = x
-                self.y = y
-
-            def __setattr__(self, key, value):
-                object.__setattr__(self, key, value)
-                self.change()
-
-            def __composite_values__(self):
-                return self.x, self.y
-
-            def __eq__(self, other):
-                return isinstance(other, Point) and \
-                    other.x == self.x and \
-                    other.y == self.y
-
-    :class:`.MutableComposite` defines a metaclass which augments
-    the creation of :class:`.MutableComposite` subclasses with an event
-    that will listen for any :func:`~.orm.composite` mappings against the 
-    new type, establishing listeners that will track parent associations.
-
+    
+    See the example in :ref:`mutable_composites` for usage information.
+    
     .. warning:: The listeners established by the :class:`.MutableComposite`
        class are *global* to all mappers, and are *not* garbage collected.   Only use 
        :class:`.MutableComposite` for types that are permanent to an application,
@@ -210,13 +511,7 @@ class MutableComposite(object):
     """
     __metaclass__ = _MutableCompositeMeta
 
-    @memoized_property
-    def _parents(self):
-        """Dictionary of parent object->attribute name on the parent."""
-
-        return weakref.WeakKeyDictionary()
-
-    def change(self):
+    def changed(self):
         """Subclasses should call this method whenever change events occur."""
 
         for parent, key in self._parents.items():
@@ -228,48 +523,6 @@ class MutableComposite(object):
                 setattr(parent, attr_name, value)
 
     @classmethod
-    def _listen_on_attribute(cls, attribute):
-        """Establish this type as a mutation listener for the given 
-        mapped descriptor.
-
-        """
-        key = attribute.key
-        parent_cls = attribute.class_
-
-        def load(state, *args):
-            """Listen for objects loaded or refreshed.
-
-            Wrap the target data member's value with 
-            ``Mutable``.
-
-            """
-
-            val = state.dict.get(key, None)
-            if val is not None:
-                val._parents[state.obj()] = key
-
-        def set(target, value, oldvalue, initiator):
-            """Listen for set/replace events on the target
-            data member.
-
-            Establish a weak reference to the parent object
-            on the incoming value, remove it for the one 
-            outgoing.
-
-            """
-
-            value._parents[target.obj()] = key
-            if isinstance(oldvalue, cls):
-                oldvalue._parents.pop(state.obj(), None)
-            return value
-
-        event.listen(parent_cls, 'load', load, raw=True)
-        event.listen(parent_cls, 'refresh', load, raw=True)
-        event.listen(attribute, 'set', set, raw=True, retval=True)
-
-        # TODO: need a deserialize hook here
-
-    @classmethod
     def _setup_listeners(cls):
         """Associate this wrapper with all future mapped compoistes
         of the given type.
@@ -281,7 +534,7 @@ class MutableComposite(object):
         def listen_for_type(mapper, class_):
             for prop in mapper.iterate_properties:
                 if hasattr(prop, 'composite_class') and issubclass(prop.composite_class, cls):
-                    cls._listen_on_attribute(getattr(class_, prop.key))
+                    cls._listen_on_attribute(getattr(class_, prop.key), False)
 
         event.listen(mapper, 'mapper_configured', listen_for_type)
 
diff --git a/lib/sqlalchemy/orm/__init__.py b/lib/sqlalchemy/orm/__init__.py
index 0b77b0239..73b079e03 100644
--- a/lib/sqlalchemy/orm/__init__.py
+++ b/lib/sqlalchemy/orm/__init__.py
@@ -671,6 +671,7 @@ def composite(class_, *cols, **kwargs):
       an :class:`.AttributeExtension` instance,
       or list of extensions, which will be prepended to the list of
       attribute listeners for the resulting descriptor placed on the class.
+      **Deprecated.**  Please see :class:`.AttributeEvents`.
 
     """
     return CompositeProperty(class_, *cols, **kwargs)
diff --git a/lib/sqlalchemy/orm/events.py b/lib/sqlalchemy/orm/events.py
index 761ba315d..8e4d4a147 100644
--- a/lib/sqlalchemy/orm/events.py
+++ b/lib/sqlalchemy/orm/events.py
@@ -195,6 +195,33 @@ class InstanceEvents(event.Events):
 
         """
 
+    def pickle(self, target, state_dict):
+        """Receive an object instance when its associated state is
+        being pickled.
+
+        :param target: the mapped instance.  If 
+         the event is configured with ``raw=True``, this will 
+         instead be the :class:`.InstanceState` state-management
+         object associated with the instance.
+        :param state_dict: the dictionary returned by 
+         :class:`.InstanceState.__getstate__`, containing the state
+         to be pickled.
+         
+        """
+
+    def unpickle(self, target, state_dict):
+        """Receive an object instance after it's associated state has
+        been unpickled.
+
+        :param target: the mapped instance.  If 
+         the event is configured with ``raw=True``, this will 
+         instead be the :class:`.InstanceState` state-management
+         object associated with the instance.
+        :param state_dict: the dictionary sent to
+         :class:`.InstanceState.__setstate__`, containing the state
+         dictionary which was pickled.
+        
+        """
 
 class MapperEvents(event.Events):
     """Define events specific to mappings.
diff --git a/lib/sqlalchemy/orm/state.py b/lib/sqlalchemy/orm/state.py
index da91a353e..7a93b5cb0 100644
--- a/lib/sqlalchemy/orm/state.py
+++ b/lib/sqlalchemy/orm/state.py
@@ -147,6 +147,9 @@ class InstanceState(object):
         )
         if self.load_path:
             d['load_path'] = interfaces.serialize_path(self.load_path)
+
+        self.manager.dispatch.pickle(self, d)
+
         return d
 
     def __setstate__(self, state):
@@ -182,7 +185,7 @@ class InstanceState(object):
         if 'load_path' in state:
             self.load_path = interfaces.deserialize_path(state['load_path'])
 
-        # TODO: need an event here, link to composite, mutable
+        manager.dispatch.unpickle(self, state)
 
     def initialize(self, key):
         """Set this attribute to an empty value or collection,