Implementation of internal notification callbacks within Keystone

Keystone subsystems (extensions or other modules) can now
subscribe to Create, Update, and Delete events.  This allows any
subscribed system to act upon the event and perform actions (such
as cleanup on delete).

Co-Authored-By: Guang Yee <guang.yee@hp.com>
Co-Authored-By: Morgan Fainberg <m@metacloud.com>

bp: internal-callbacks
Change-Id: I03713b7f1f94480f76e0121eb6184226062af1a6

6 files changed, 308 insertions, 2 deletions

diff --git a/doc/source/EXTENSIONS_HOWTO.rst b/doc/source/EXTENSIONS_HOWTO.rst
index 70cb2f666..a02cbc9dc 100644
--- a/doc/source/EXTENSIONS_HOWTO.rst
+++ b/doc/source/EXTENSIONS_HOWTO.rst
@@ -44,7 +44,8 @@ Extension code base in the `keystone/contrib/example` folder.
   file, these should be kept disabled as default and have their own element.
 - If configuration changes are required/introduced in the `keystone-paste.ini`,
   the new filter must be declared.
-
+- The module may register to listen to events by declaring the corresponding
+  callbacks in the ``core.py`` file.
 
 `keystone.conf.sample` File
 ---------------------------
@@ -241,3 +242,77 @@ must be executed.
 Example::
 
      ./bin/keystone-manage db_sync --extension example
+
+
+Event Callbacks
+-----------
+
+Extensions may provide callbacks to Keystone (Identity) events.
+Extensions must provide the list of events of interest and the corresponding
+callbacks. Events are issued upon successful creation, modification, and
+deletion of the following Keystone resources:
+
+* ``group``
+* ``project``
+* ``role``
+* ``user``
+
+The extension's ``Manager`` class must contain the
+``event_callbacks`` attribute. It is a dictionary listing as keys
+those events that are of interest and the values should be the respective
+callbacks. Event callback registration is done via the
+dependency injection mechanism. During dependency provider registration, the
+``dependency.provider`` decorator looks for the ``event_callbacks``
+class attribute. If it exists the event callbacks are registered
+accordingly. In order to enable event callbacks, the extension's ``Manager``
+class must also be a dependency provider.
+
+Example:
+
+.. code:: python
+
+    # Since this is a dependency provider. Any code module using this or any
+    # other dependency provider (uses the dependency.provider decorator)
+    # will be enabled for the attribute based notification
+
+    @dependency.provider('example_api')
+    class ExampleManager(manager.Manager):
+        """Example Manager.
+
+        See :mod:`keystone.common.manager.Manager` for more details on
+        how this dynamically calls the backend.
+
+        """
+
+        def __init__(self):
+            self.event_callbacks = {
+                # Here we add the the event_callbacks class attribute that
+                # calls project_deleted_callback when a project is deleted.
+                'deleted': {
+                    'project': [
+                        self.project_deleted_callback]}}
+            super(ExampleManager, self).__init__(
+                'keystone.contrib.example.core.ExampleDriver')
+
+        def project_deleted_callback(self, context, message):
+            # cleanup data related to the deleted project here
+
+A callback must accept the following parameters:
+
+* ``service`` - the service information (e.g. identity)
+* ``resource_type`` - the resource type (e.g. project)
+* ``operation`` - the operation (updated, created, deleted)
+* ``payload`` - the actual payload info of the resource that was acted on
+
+Current callback operations:
+
+* ``created``
+* ``deleted``
+* ``updated``
+
+Example:
+
+.. code:: python
+      def project_deleted_callback(self, service, resource_type, operation,
+                                   payload):
+
diff --git a/keystone/common/dependency.py b/keystone/common/dependency.py
index 5eaa4d886..9190f6e7a 100644
--- a/keystone/common/dependency.py
+++ b/keystone/common/dependency.py
@@ -26,6 +26,9 @@ See also:
     https://en.wikipedia.org/wiki/Dependency_injection
 """
 
+from keystone import notifications
+
+
 REGISTRY = {}
 
 _future_dependencies = {}
@@ -64,10 +67,43 @@ def provider(name):
     """
     def wrapper(cls):
         def wrapped(init):
+            def register_event_callbacks(self):
+                # NOTE(morganfainberg): A provider who has an implicit
+                # dependency on other providers may utilize the event callback
+                # mechanism to react to any changes in those providers. This is
+                # performed at the .provider() mechanism so that we can ensure
+                # that the callback is only ever called once and guaranteed
+                # to be on the properly configured and instantiated backend.
+                if not hasattr(self, 'event_callbacks'):
+                    return
+
+                if not isinstance(self.event_callbacks, dict):
+                    msg = _('event_callbacks must be a dict')
+                    raise ValueError(msg)
+
+                for event in self.event_callbacks:
+                    if not isinstance(self.event_callbacks[event], dict):
+                        msg = _('event_callbacks[%s] must be a dict') % event
+                        raise ValueError(msg)
+                    for resource_type in self.event_callbacks[event]:
+                        # Make sure we register the provider for each event it
+                        # cares to call back.
+                        callbacks = self.event_callbacks[event][resource_type]
+                        if not callbacks:
+                            continue
+                        if not hasattr(callbacks, '__iter__'):
+                            # ensure the callback information is a list
+                            # allowing multiple callbacks to exist
+                            callbacks = [callbacks]
+                        notifications.register_event_callback(event,
+                                                              resource_type,
+                                                              callbacks)
+
             def __wrapped_init__(self, *args, **kwargs):
                 """Initialize the wrapped object and add it to the registry."""
                 init(self, *args, **kwargs)
                 REGISTRY[name] = self
+                register_event_callbacks(self)
 
                 resolve_future_dependencies(name)
 
diff --git a/keystone/contrib/example/core.py b/keystone/contrib/example/core.py
index 12a7bc689..f66af2ba2 100644
--- a/keystone/contrib/example/core.py
+++ b/keystone/contrib/example/core.py
@@ -36,7 +36,46 @@ class ExampleManager(manager.Manager):
     """
 
     def __init__(self):
-        super(ExampleManager, self).__init__(CONF.ExampleDriver.driver)
+        # The following is an example of event callbacks. In this setup,
+        # ExampleManager's data model is depended on project's data model.
+        # It must create additional aggregates when a new project is created,
+        # and it must cleanup data related to the project whenever a project
+        # has been deleted.
+        #
+        # In this example, the project_deleted_callback will be invoked
+        # whenever a project has been deleted. Similarly, the
+        # project_created_callback will be invoked whenever a new project is
+        # created.
+
+        # This information is used when the @dependency.provider decorator acts
+        # on the class.
+        self.event_callbacks = {
+            'deleted': {
+                'project': [
+                    self.project_deleted_callback]},
+            'created': {
+                'project': [
+                    self.project_created_callback]}}
+        super(ExampleManager, self).__init__(
+            'keystone.contrib.example.core.ExampleDriver')
+
+    def project_deleted_callback(self, service, resource_type, operation,
+                                 payload):
+        # The code below is merely an example.
+        msg = _('Received the following notification: service %(service)s, '
+                'resource_type: %(resource_type)s, operation %(operation)s '
+                'payload %(payload)s')
+        LOG.info(msg, {'service': service, 'resource_type': resource_type,
+                       'operation': operation, 'payload': payload})
+
+    def project_created_callback(self, service, resource_type, operation,
+                                 payload):
+        # The code below is merely an example.
+        msg = _('Received the following notification: service %(service)s, '
+                'resource_type: %(resource_type)s, operation %(operation)s '
+                'payload %(payload)s')
+        LOG.info(msg, {'service': service, 'resource_type': resource_type,
+                       'operation': operation, 'payload': payload})
 
 
 class ExampleDriver(object):
diff --git a/keystone/notifications.py b/keystone/notifications.py
index a6f95222e..3158e19ef 100644
--- a/keystone/notifications.py
+++ b/keystone/notifications.py
@@ -16,11 +16,19 @@
 
 """Notifications module for OpenStack Identity Service resources"""
 
+import logging
+
 from keystone.openstack.common import log
 from keystone.openstack.common.notifier import api as notifier_api
 
 
 LOG = log.getLogger(__name__)
+# NOTE(gyee): actions that can be notified. One must update this list whenever
+# a new action is supported.
+ACTIONS = frozenset(['created', 'deleted', 'updated'])
+# resource types that can be notified
+RESOURCE_TYPES = set()
+SUBSCRIBERS = {}
 
 
 class ManagerNotificationWrapper(object):
@@ -35,6 +43,7 @@ class ManagerNotificationWrapper(object):
     def __init__(self, operation, resource_type, host=None):
         self.operation = operation
         self.resource_type = resource_type
+        RESOURCE_TYPES.add(resource_type)
         self.host = host
 
     def __call__(self, f):
@@ -70,6 +79,65 @@ def deleted(*args, **kwargs):
     return ManagerNotificationWrapper('deleted', *args, **kwargs)
 
 
+def _get_callback_info(callback):
+    if getattr(callback, 'im_class', None):
+        return [getattr(callback, '__module__', None),
+                callback.im_class.__name__,
+                callback.__name__]
+    else:
+        return [getattr(callback, '__module__', None), callback.__name__]
+
+
+def register_event_callback(event, resource_type, callbacks):
+    if event not in ACTIONS:
+        raise ValueError(_('%(event)s is not a valid notification event, must '
+                           'be one of: %(actions)s') %
+                         {'event': event, 'actions': ', '.join(ACTIONS)})
+    if resource_type not in RESOURCE_TYPES:
+        raise ValueError(_('%(resource_type)s is not a valid notification '
+                           'resource, must be one of: %(types)s') %
+                         {'resource_type': resource_type,
+                          'types': ', '.join(RESOURCE_TYPES)})
+
+    if not hasattr(callbacks, '__iter__'):
+        callbacks = [callbacks]
+
+    for callback in callbacks:
+        if not callable(callback):
+            msg = _('Method not callable: %s') % callback
+            LOG.error(msg)
+            raise TypeError(msg)
+        SUBSCRIBERS.setdefault(event, {}).setdefault(resource_type, set())
+        SUBSCRIBERS[event][resource_type].add(callback)
+
+        if LOG.logger.getEffectiveLevel() <= logging.INFO:
+            # Do this only if its going to appear in the logs.
+            msg = _('Callback: `%(callback)s` subscribed to event '
+                    '`%(event)s`.')
+            callback_info = _get_callback_info(callback)
+            callback_str = '.'.join(
+                filter(lambda i: i is not None, callback_info))
+            event_str = '.'.join(['identity', resource_type, event])
+            LOG.info(msg, {'callback': callback_str, 'event': event_str})
+
+
+def notify_event_callbacks(service, resource_type, operation, payload):
+    """Sends a notification to registered extensions."""
+    if operation in SUBSCRIBERS:
+        if resource_type in SUBSCRIBERS[operation]:
+            for cb in SUBSCRIBERS[operation][resource_type]:
+                subst_dict = {'cb_name': cb.__name__,
+                              'service': service,
+                              'resource_type': resource_type,
+                              'operation': operation,
+                              'payload': payload}
+                LOG.debug(_('Invoking callback %(cb_name)s for event '
+                          '%(service)s %(resource_type)s %(operation)s for'
+                          '%(payload)s'),
+                          subst_dict)
+                cb(service, resource_type, operation, payload)
+
+
 def _send_notification(operation, resource_type, resource_id, host=None):
     """Send notification to inform observers about the affected resource.
 
@@ -89,6 +157,8 @@ def _send_notification(operation, resource_type, resource_id, host=None):
         'resource_type': resource_type,
         'operation': operation}
 
+    notify_event_callbacks(service, resource_type, operation, payload)
+
     try:
         notifier_api.notify(
             context, publisher_id, event_type, notifier_api.INFO, payload)
diff --git a/keystone/tests/core.py b/keystone/tests/core.py
index 69180106c..6db3b81cb 100644
--- a/keystone/tests/core.py
+++ b/keystone/tests/core.py
@@ -58,6 +58,7 @@ from keystone.common import utils
 from keystone.common import wsgi
 from keystone import config
 from keystone import exception
+from keystone import notifications
 from keystone.openstack.common.db.sqlalchemy import session
 from keystone.openstack.common import log
 from keystone.openstack.common import timeutils
@@ -334,6 +335,9 @@ class TestCase(testtools.TestCase):
 
         self.addCleanup(timeutils.clear_time_override)
 
+        # Ensure Notification subscriotions and resource types are empty
+        self.addCleanup(notifications.SUBSCRIBERS.clear)
+
     def config(self, config_files):
         CONF(args=[], project='keystone', default_config_files=config_files)
 
diff --git a/keystone/tests/test_notifications.py b/keystone/tests/test_notifications.py
index 64c038730..d336e317b 100644
--- a/keystone/tests/test_notifications.py
+++ b/keystone/tests/test_notifications.py
@@ -16,6 +16,7 @@
 
 import uuid
 
+from keystone.common import dependency
 from keystone import notifications
 from keystone.openstack.common.fixture import moxstubout
 from keystone.openstack.common.notifier import api as notifier_api
@@ -250,3 +251,84 @@ class NotificationsForEntities(test_v3.RestfulTestCase):
         self.identity_api.create_user(user_ref['id'], user_ref)
         self.identity_api.update_user(user_ref['id'], user_ref)
         self._assertLastNotify(user_ref['id'], 'updated', 'user')
+
+
+class TestEventCallbacks(test_v3.RestfulTestCase):
+
+    def setUp(self):
+        super(TestEventCallbacks, self).setUp()
+        notifications.SUBSCRIBERS = {}
+        self.has_been_called = False
+
+    def _project_deleted_callback(self, service, resource_type, operation,
+                                  payload):
+        self.has_been_called = True
+
+    def _project_created_callback(self, service, resource_type, operation,
+                                  payload):
+        self.has_been_called = True
+
+    def test_notification_received(self):
+        notifications.register_event_callback('created',
+                                              'project',
+                                              self._project_created_callback)
+        project_ref = self.new_project_ref(domain_id=self.domain_id)
+        self.assignment_api.create_project(project_ref['id'], project_ref)
+        self.assertTrue(self.has_been_called)
+
+    def test_notification_method_not_callable(self):
+        fake_method = None
+        notifications.SUBSCRIBERS = {}
+        self.assertRaises(TypeError,
+                          notifications.register_event_callback,
+                          'updated',
+                          'project',
+                          [fake_method])
+
+    def test_notification_event_not_valid(self):
+        self.assertRaises(ValueError,
+                          notifications.register_event_callback,
+                          uuid.uuid4().hex,
+                          'project',
+                          self._project_deleted_callback)
+
+    def test_resource_type_not_valid(self):
+        self.assertRaises(ValueError,
+                          notifications.register_event_callback,
+                          'deleted',
+                          uuid.uuid4().hex,
+                          self._project_deleted_callback)
+
+    def test_provider_event_callbacks_subscription(self):
+        @dependency.provider('foo_api')
+        class Foo:
+            def __init__(self):
+                self.event_callbacks = {
+                    'created': {
+                        'project': [self.foo_callback]}}
+
+            def foo_callback(self, service, resource_type, operation,
+                             payload):
+                pass
+
+        notifications.SUBSCRIBERS = {}
+        Foo()
+        self.assertIn('created', notifications.SUBSCRIBERS)
+
+    def test_invalid_event_callbacks(self):
+        @dependency.provider('foo_api')
+        class Foo:
+            def __init__(self):
+                self.event_callbacks = 'bogus'
+
+        notifications.SUBSCRIBERS = {}
+        self.assertRaises(ValueError, Foo)
+
+    def test_invalid_event_callbacks_event(self):
+        @dependency.provider('foo_api')
+        class Foo:
+            def __init__(self):
+                self.event_callbacks = {'created': 'bogus'}
+
+        notifications.SUBSCRIBERS = {}
+        self.assertRaises(ValueError, Foo)