baserock_openid_provider: Update on top of Fedora 23, and without Packer

This means we no longer use Packer for anything, which is good.

The switch from Django 1.7 to Django 1.9 caused some problems in the
openid_provider module. Upstream for that module is here:

    https://bitbucket.org/romke/django_openid_provider/

At the time of writing there was no fix upstream for these issues, it
would be good to submit them. We have other unsubmitted changes against
that upstream in our openid_provider code.

One issue was use of import_module (which is now available from
importlib, doesn't need to be imported from django.utils any more).

Another is use of WSGIRequest.REQUEST, which is deprecated since
Django 1.7 and removed in Django 1.9. We now need to use .POST or
.GET to get that info.

Change-Id: I60793aaf0d84d81b89ff59efbe08240d99b7973f

12 files changed, 60 insertions, 172 deletions

diff --git a/README.mdwn b/README.mdwn
index 280f071a..a9db18d5 100644
--- a/README.mdwn
+++ b/README.mdwn
@@ -131,46 +131,6 @@ NOTE: to run these playbooks you need to have the public ssh key of the backups
 instance in `keys/backup.key.pub`.
 
 
-Deployment with Packer
-----------------------
-
-> **NOTE**: I no longer think that Packer is the right tool for our needs. This
-> is partly because of critical bugs that have not been fixed since I started
-> using it (e.g. <https://github.com/mitchellh/packer/issues/1462>), and partly
-> because I realised that I was just using it to wrap `nova` and
-> `ansible-playbook`, and it is simple enough to use those commands directly.
->
-> I had hoped that we could make use of Packer's multiple backends in order to
-> test systems locally in Docker before deploying them to OpenStack. It turns
-> out Docker is sufficiently different to OpenStack that this doesn't make life
-> any easier during development. Networking setup is different, systemd doesn't
-> work inside Docker by default, base images are different in other ways, etc.
->
-> So I recommend not using Packer for future systems, and I will try to
-> migrate the definitions for the existing ones to just use Ansible.
->
-> Sam Thursfield 10/04/15
-
-Some of the systems are built with [Packer]. I chose Packer because it provides
-similar functionality to the `morph deploy` command, although its
-implementation makes different tradeoffs. The documentation below shows the
-commands you need to run to build systems with Packer. Some of the systems can
-be deployed as Docker images as well as OpenStack images, to enable local
-development and testing.
-
-The following error from Packer means that you didn't set your credentials
-correctly in the `OS_...` environment variables, or they were not accepted.
-
-> Build 'production' errored: Missing or incorrect provider
-
-The the Packer tool requires a floating IP to be available at the time a system
-is being deployed to OpenStack. Currently 185.43.218.169 should be used for
-this.  If you specify a floating IP that is in use by an existing instance, you
-will steal it for your own instance and probably break one of our web services.
-
-[Packer]: http://www.packer.io/
-
-
 Systems
 -------
 
@@ -279,35 +239,27 @@ receive mail, or relay mail sent by systems outside the baserock.org cloud.
 
 ### OpenID provider
 
-To deploy a development instance:
-
-    packer build -only=development baserock_openid_provider/packer_template.json
-    baserock_openid_provider/develop.sh
-    # Now you have a root shell inside your container
-    cd /srv/baserock_openid_provider
-    python ./manage.py runserver 0.0.0.0:80
-    # Now you can browse to http://localhost:80/ and see the server.
-
 To deploy this system to production:
 
     vim baserock_openid_provider/baserock_openid_provider/settings.py
 
-Edit the DATABASES['default']['HOST'] to point to the fixed IP of the
-'database' machine, and check the settings. See:
-https://docs.djangoproject.com/en/1.7/howto/deployment/checklist/
-
-    packer build -only=production baserock_openid_provider/packer_template.json
+Check the DATABASE_HOST IP, and check the other settings against the [Django
+deployment
+checklist](https://docs.djangoproject.com/en/1.7/howto/deployment/checklist/).
 
     nova boot openid.baserock.org \
         --key-name $keyname \
         --flavor dc1.1x1 \
-        --image 'baserock_openid_provider' \
-        --nic "net-id=$network_id',v4-fixed-ip=192.168.222.67" \
-        --security-groups default,web-server
+        --image $fedora_image_id \
+        --nic "net-id=$network_id,v4-fixed-ip=192.168.222.144" \
+        --security-groups default,web-server \
         --user-data ./baserock-ops-team.cloud-config
 
+    ansible-playbook -i hosts baserock_openid_provider/image-config.yml
     ansible-playbook -i hosts baserock_openid_provider/instance-config.yml
 
+The baserock_openid_provider system is stateless.
+
 To change Cherokee configuration, it's usually easiest to use the
 cherokee-admin tool in a running instance. SSH in as normal but forward port
 9090 to localhost (pass `-L9090:localhost:9090` to SSH). Backup the old
diff --git a/baserock_frontend/haproxy.cfg b/baserock_frontend/haproxy.cfg
index 0ab58574..9124ba7d 100644
--- a/baserock_frontend/haproxy.cfg
+++ b/baserock_frontend/haproxy.cfg
@@ -99,7 +99,7 @@ backend baserock_mason_x86_64_http
     server baserock_mason_x86_64 192.168.222.80:80
 
 backend baserock_openid_provider_http
-    server baserock_openid_provider 192.168.222.67:80
+    server baserock_openid_provider 192.168.222.144:80
 
 backend baserock_webserver_http
     server baserock_webserver 192.168.222.127:80
diff --git a/baserock_hosts b/baserock_hosts
index da16f609..e84fd91a 100644
--- a/baserock_hosts
+++ b/baserock_hosts
@@ -20,7 +20,7 @@ mason-x86-64 ansible_ssh_host=192.168.222.80
 frontend-haproxy ansible_ssh_host=185.43.218.170
 database-mariadb ansible_ssh_host=192.168.222.30
 mail ansible_ssh_host=192.168.222.111
-openid ansible_ssh_host=192.168.222.67
+openid ansible_ssh_host=192.168.222.144
 webserver ansible_ssh_host=192.168.222.127
 
 [ubuntu]
diff --git a/baserock_openid_provider/baserock_openid_provider/settings.py b/baserock_openid_provider/baserock_openid_provider/settings.py
index a7e892ba..d9d3ffec 100644
--- a/baserock_openid_provider/baserock_openid_provider/settings.py
+++ b/baserock_openid_provider/baserock_openid_provider/settings.py
@@ -12,6 +12,9 @@ import yaml
 
 import os
 
+# You must ensure this is the correct IP address!
+DATABASE_HOST = '192.168.222.30'
+
 BASE_DIR = os.path.dirname(os.path.dirname(__file__))
 
 # Quick-start development settings - unsuitable for production
@@ -114,12 +117,7 @@ DATABASES = {
         'USER': 'openid',
         'PORT': '3306',
 
-        # You must change this to the correct IP address when
-        # deploying to production! For development deployments this
-        # gets the IP of the 'baserock-database' container from the
-        # environment, which Docker will have set if you passed it
-        # `--link=baseock-database:db`.
-        'HOST': os.environ.get('DB_PORT_3306_TCP_ADDR', '192.168.222.30')
+        'HOST': DATABASE_HOST
     }
 }
 
diff --git a/baserock_openid_provider/develop.sh b/baserock_openid_provider/develop.sh
deleted file mode 100755
index 534a1333..00000000
--- a/baserock_openid_provider/develop.sh
+++ /dev/null
@@ -1,11 +0,0 @@
-#!/bin/sh
-
-# Set up a development environment in a container.
-
-exec docker run -i -t --rm \
-    --name=baserock-openid-provider \
-    --link=baserock-database:db \
-    --publish=127.0.0.1:80:80 \
-    --volume=`pwd`:/srv/test-baserock-infrastructure \
-    baserock/openid-provider
-
diff --git a/baserock_openid_provider/image-config.yml b/baserock_openid_provider/image-config.yml
index 3c0d8bb5..92ba0951 100644
--- a/baserock_openid_provider/image-config.yml
+++ b/baserock_openid_provider/image-config.yml
@@ -2,46 +2,61 @@
 #
 # This playbook is run at image-creation time by Packer.
 ---
-- hosts: localhost
+- hosts: openid
   gather_facts: False
+  sudo: yes
   tasks:
+  # See: https://fedoramagazine.org/getting-ansible-working-fedora-23/
+  - name: install Python2 and required deps for Ansible modules
+    raw: dnf install -y python2 python2-dnf libselinux-python
+
   - name: enable persistant journal
     shell: mkdir /var/log/journal
     args:
       creates: /var/log/journal
 
-  - name: install Cherokee web server
-    yum: name=cherokee state=latest
+  - name: ensure system up to date
+    dnf: name=* state=latest
 
-  - name: install PIP package manager
-    yum: name=python-pip state=latest
+  - name: install Cherokee web server
+    dnf: name=cherokee state=latest
 
   - name: install Sendmail mail transfer agent
-    yum: name=sendmail state=latest
+    dnf: name=sendmail state=latest
 
   - name: install uWSGI application container server and Python plugin
-    yum: name=uwsgi-plugin-python state=latest
+    dnf: name=uwsgi-plugin-python state=latest
+
+  - name: install PyYAML
+    dnf: name=PyYAML state=latest
 
+  # All this stuff is installed with Pip, which isn't really necessary except
+  # for django-registration-redux. Fedora packages django-registration but not
+  # the better django-registration-redux (I think).
+  #
   - name: install Django
-    pip: name=django
+    pip: name=django executable=pip2.7
+
+  - name: install South (Django migrations tool)
+    pip: name=South executable=pip2.7
 
   # This is a fork of django-registration which supports Django 1.7.
   # Source: https://github.com/macropin/django-registration
   # The original django-registration (which seems to be abandoned) lives at:
   # https://bitbucket.org/ubernostrum/django-registration/
   - name: install django-registration-redux
-    pip: name=django-registration-redux
+    pip: name=django-registration-redux executable=pip2.7
 
   - name: install python-openid
-    pip: name=python-openid
+    pip: name=python-openid executable=pip2.7
 
-  # Install the MySQL-python package from Yum, because if it's installed from
+  # Install the MySQL-python package from DNF, because if it's installed from
   # PyPI you need to have the mariadb-devel package installed to build the C
   # code and that's an extra 21MB of dependencies or so. Note that this driver
   # doesn't support Python 3, but there is a fork available which does, see:
   # https://docs.djangoproject.com/en/dev/ref/databases/#mysql-db-api-drivers
   - name: install MySQL-python
-    yum: name=MySQL-python state=latest
+    dnf: name=MySQL-python state=latest
 
   - name: install Cherokee configuration
     file: src=/srv/baserock_openid_provider/cherokee.conf dest=/etc/cherokee/cherokee.conf state=link force=yes
@@ -49,5 +64,12 @@
   - name: create log directory for baserock_openid_provider
     file: path=/var/log/baserock_openid_provider owner=cherokee group=cherokee state=directory
 
-  - name: create directory for static content
-    file: path=/var/www/static owner=cherokee group=cherokee state=directory
+  - name: upload application
+    copy: src=. dest=/srv owner=fedora group=fedora
+
+  # Yes, SELinux prevents Cherokee from working.
+  - name: disable SELinux on subsequent boots
+    selinux: state=disabled
+
+  - name: disable SELinux on current boot
+    command: setenforce 0
diff --git a/baserock_openid_provider/instance-config.yml b/baserock_openid_provider/instance-config.yml
index 7eac185d..3311d51e 100644
--- a/baserock_openid_provider/instance-config.yml
+++ b/baserock_openid_provider/instance-config.yml
@@ -7,9 +7,6 @@
   gather_facts: False
   sudo: yes
   tasks:
-  - name: ensure system up to date
-    yum: name=* state=latest
-
   - name: install database password
     copy: src=../database/baserock_openid_provider.database_password.yml dest=/etc owner=cherokee group=cherokee mode=400
 
diff --git a/baserock_openid_provider/openid_provider/migrations/0001_initial.py b/baserock_openid_provider/openid_provider/south_migrations/0001_initial.py
index 1857f59a..1857f59a 100644
--- a/baserock_openid_provider/openid_provider/migrations/0001_initial.py
+++ b/baserock_openid_provider/openid_provider/south_migrations/0001_initial.py
diff --git a/baserock_openid_provider/openid_provider/migrations/__init__.py b/baserock_openid_provider/openid_provider/south_migrations/__init__.py
index e69de29b..e69de29b 100644
--- a/baserock_openid_provider/openid_provider/migrations/__init__.py
+++ b/baserock_openid_provider/openid_provider/south_migrations/__init__.py
diff --git a/baserock_openid_provider/openid_provider/utils.py b/baserock_openid_provider/openid_provider/utils.py
index ae704001..dc0c714f 100644
--- a/baserock_openid_provider/openid_provider/utils.py
+++ b/baserock_openid_provider/openid_provider/utils.py
@@ -13,7 +13,8 @@ from django.core.exceptions import ImproperlyConfigured
 from django.core.urlresolvers import reverse
 from django.http import HttpResponse
 from django.shortcuts import render_to_response
-from django.utils.importlib import import_module
+
+from importlib import import_module
 
 import logging
 
diff --git a/baserock_openid_provider/openid_provider/views.py b/baserock_openid_provider/openid_provider/views.py
index 2633abf0..1b8ef6d5 100644
--- a/baserock_openid_provider/openid_provider/views.py
+++ b/baserock_openid_provider/openid_provider/views.py
@@ -56,7 +56,13 @@ def openid_server(request):
     if request.session.get('AuthorizationInfo', None):
         del request.session['AuthorizationInfo']
 
-    querydict = dict(request.REQUEST.items())
+    if request.method == 'GET':
+        querydict = dict(request.GET.items())
+    elif request.method == 'POST':
+        querydict = dict(request.POST.items())
+    else:
+        return HTTPResponseNotAllowed(['GET', 'POST'])
+
     orequest = server.decodeRequest(querydict)
     if not orequest:
         orequest = server.decodeRequest(request.session.get('OPENID_REQUEST', None))
diff --git a/baserock_openid_provider/packer_template.json b/baserock_openid_provider/packer_template.json
deleted file mode 100644
index 0de9bc84..00000000
--- a/baserock_openid_provider/packer_template.json
+++ /dev/null
@@ -1,77 +0,0 @@
-{
-    "builders": [
-        {
-            "name": "development",
-            "type": "docker",
-            "image": "fedora:20",
-            "commit": true,
-            "run_command": ["-d", "-i", "-t", "{{.Image}}", "/bin/sh"]
-        },
-        {
-            "name": "production",
-            "type": "openstack",
-            "image_name": "baserock_openid_provider",
-            "flavor": "f0577618-9125-4948-b450-474e225bbc4c",
-            "source_image": "742e0414-c985-4994-b307-4aafade942b3",
-            "networks": ["d079fa3e-2558-4bcb-ad5a-279040c202b5"],
-            "floating_ip": "185.43.218.169",
-            "use_floating_ip": true,
-            "ssh_username": "fedora"
-        }
-    ],
-    "provisioners": [
-        {
-            "type": "shell",
-            "inline": [
-               "sudo chown fedora:fedora /srv"
-            ],
-            "only": ["production"]
-        },
-        {
-            "type": "file",
-            "source": "baserock_openid_provider",
-            "destination": "/srv",
-            "only": ["production"]
-        },
-        {
-            "type": "shell",
-            "inline": [ "sudo yum install -y ansible"]
-        },
-        {
-            "type": "ansible-local",
-            "playbook_file": "baserock_openid_provider/image-config.yml",
-            "command": "sudo ansible-playbook"
-        },
-        {
-            "type": "shell",
-            "inline": [
-                "ln -s /srv/test-baserock-infrastructure/baserock_openid_provider /srv"
-            ],
-            "only": ["development"]
-        },
-        {
-            "type": "shell",
-            "inline": [
-                "sudo yum install -y libselinux-python",
-                "sudo ansible localhost -m selinux -a state=disabled",
-                "sudo setenforce 0"
-            ],
-            "only": ["production"]
-        },
-        {
-            "type": "shell",
-            "inline": [ "sync; sync; sleep 10; sync" ],
-            "only": ["production"]
-        }
-    ],
-    "post-processors": [
-        [
-            {
-                "type": "docker-tag",
-                "repository": "baserock/openid-provider",
-                "tag": "latest",
-                "only": ["development"]
-            }
-        ]
-    ]
-}