path: root/README.mdwn

Baserock project public infrastructure
======================================

This repository contains the definitions for all of the Baserock Project's
infrastructure. This includes every service used by the project, except for
the mailing lists (hosted by [Pepperfish]) and the wiki (hosted by
[Branchable]).

Some of these systems are Baserock systems. Other are Ubuntu or Fedora based.
Eventually we want to move all of these to being Baserock systems.

The infrastructure is set up in a way that parallels the preferred Baserock
approach to deployment. All files necessary for (re)deploying the systems
should be contained in this Git repository, with the exception of certain
private tokens (which should be simple to inject at deploy time).

[Pepperfish]: http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo
[Branchable]: http://www.branchable.com/


General notes
-------------

When instantiating a machine that will be public, remember to give shell
access everyone on the ops team. This can be done using a post-creation
customisation script that injects all of their SSH keys. The SSH public
keys of the Baserock Operations team are collected in
`baserock-ops-team.cloud-config.`.

Ensure SSH password login is disabled in all systems you deploy! See:
<https://testbit.eu/is-ssh-insecure/> for why. The Ansible playbook
`admin/sshd_config.yaml` can ensure that all systems have password login
disabled.


Administration
--------------

You can use [Ansible] to automate tasks on the baserock.org systems.

To run a playbook:

    ansible-playbook -i hosts $PLAYBOOK.yaml

To run an ad-hoc command (upgrading, for example):

    ansible-playbook -i hosts fedora -m command -a 'sudo yum update -y'
    ansible-playbook -i hosts ubuntu -m command -a 'sudo apt-get update -y'

[Ansible]: http://www.ansible.com


Backups
-------

The database server doesn't yet have automated backups running. You can
manually take a backup like this:

    sudo systemctl stop mariadb.service
    sudo lvcreate \
        --name database-backup-20150126 \
        --snapshot /dev/vg0/database \
        --extents 100%ORIGIN \
        --permission=r
    sudo systemctl start mariadb.service
    sudo mount /dev/vg0/database-backup-20150126 /mnt
    # use your preferred backup tool (`rsync` is recommended) to extract the
    # contents of /mnt somewhere safe.
    sudo umount /dev/vg0/database-backup-20150126
    sudo lvremove /dev/vg0/database-backup-20150126

The Gerrit instance stores the Gerrit site path on an LVM volume and can be
manually backed up in exactly the same way.

git.baserock.org has automated backups of /home and /etc, which are run by
Codethink to an internal Codethink server.


Deployment with Packer
----------------------

Some of the systems are built with [Packer]. I chose Packer because it provides
similar functionality to the `morph deploy` command, although its
implementation makes different tradeoffs. The documentation below shows the
commands you need to run to build systems with Packer. Some of the systems can
be deployed as Docker images as well as OpenStack images, to enable local
development and testing.

The following error from Packer means that you didn't set your credentials
correctly in the `OS_...` environment variables, or they were not accepted.

> Build 'production' errored: Missing or incorrect provider

The the Packer tool requires a floating IP to be available at the time a system
is being deployed to OpenStack. Currently 185.43.218.169 should be used for
this.  If you specify a floating IP that is in use by an existing instance, you
will steal it for your own instance and probably break one of our web services.

[Packer]: http://www.packer.io/


Systems
-------

### Front-end

The front-end provides a reverse proxy, to allow more flexible routing than
simply pointing each subdomain to a different instance using separate public
IPs. It also provides a starting point for future load-balancing and failover
configuration.

If you want to add a new service to the Baserock Project infrastructure via
the frontend, alter the haproxy.cfg file in the frontend/ directory. Our
OpenStack instance doesn't provide any kind of internal DNS service, so you
must put the fixed IP of each instance.

To deploy this system:

    packer build -only=production frontend/packer_template.json

Full HAProxy 1.5 documentation: <https://cbonte.github.io/haproxy-dconv/configuration-1.5.html>.

When setting up a new instance with the frontend already deployed, do the
following:

- request a subdomain that points at 85.199.252.162
- log in to the frontend-haproxy machine
- edit /etc/haproxy/haproxy.conf, and make the same changes to the copy in this
  repo.
- run: `sudo haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -sf
  $(cat /var/run/haproxy.pid)` to reload the configuration without interrupting
  the service (this confuses systemd, but I'm not sure how to avoid that)

### Database

Baserock infrastructure uses a shared [MariaDB] database. MariaDB was chosen
because Storyboard only supports MariaDB.

To deploy this system to production:

    packer build -only=production database/packer_template.json
    nova boot database-mariadb \
        --key-name=$keyname \
        --flavor dc1.1x1 \
        --image 'database-mariadb' \
        --nic='net-id=d079fa3e-2558-4bcb-ad5a-279040c202b5,v4-fixed-ip=192.168.222.30' \
        --security-groups default,database-mariadb
        --user-data ./baserock-ops-team.cloud-config
    nova volume-create \
        --display-name database-volume \
        --display-description 'Database volume' \
        --volume-type Ceph \
        100
    nova volume-attach database-mariadb <volume ID> /dev/vdb

    ansible-playbook -i hosts database/instance-config.yml

To add the required users and databases, run the following playbook. This can
be altered and rerun whenever you need to add more users or databases.

    ansible-playbook -i hosts database/instance-mariadb-config.yml

[MariaDB]: https://www.mariadb.org

### OpenID provider

To deploy a development instance:

    packer build -only=development baserock_openid_provider/packer_template.json
    baserock_openid_provider/develop.sh
    # Now you have a root shell inside your container
    cd /srv/baserock_openid_provider
    python ./manage.py runserver 0.0.0.0:80
    # Now you can browse to http://localhost:80/ and see the server.

To deploy this system to production:

    vim baserock_openid_provider/baserock_openid_provider/settings.py

Edit the DATABASES['default']['HOST'] to point to the fixed IP of the
'database' machine, and check the settings. See:
https://docs.djangoproject.com/en/1.7/howto/deployment/checklist/

    packer build -only=production baserock_openid_provider/packer_template.json

    nova boot openid.baserock.org \
        --key-name $keyname \
        --flavor dc1.1x1 \
        --image 'baserock_openid_provider' \
        --nic 'net-id=d079fa3e-2558-4bcb-ad5a-279040c202b5',v4-fixed-ip=192.168.222.67 \
        --security-groups default,web-server
        --user-data ./baserock-ops-team.cloud-config

    ansible-playbook -i hosts baserock_openid_provider/instance-config.yml

### Gerrit

To deploy to production:

    nova volume-create \
        --display-name gerrit-volume \
        --display-description 'Gerrit volume' \
        --volume-type Ceph \
        100

    morph build baserock_gerrit/gerrit-system-x86_64.morph
    morph deploy baserock_gerrit/baserock_gerrit.morph

    nova boot gerrit.baserock.org \
        --key-name $keyname \
        --flavor 'dc1.2x4.40' \
        --image baserock_gerrit \
        --nic net-id=d079fa3e-2558-4bcb-ad5a-279040c202b5,v4-fixed-ip=192.168.222.69 \
        --security-groups default,gerrit,git-server,web-server
        --user-data baserock-ops-team.cloud-config

    nova volume-attach gerrit.baserock.org <volume-id> /dev/vdb

Accept the license and download the latest Java Runtime Environment from
http://www.oracle.com/technetwork/java/javase/downloads/server-jre8-downloads-2133154.html

Accept the license and download the latest Java Cryptography Extensions from
http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html

Save these two files in the baserock_gerrit/ folder. The instance-config.yml
Ansible playbook will upload them to the new system.

    # Don't copy-paste this! Use the Oracle website instead!
    wget --no-cookies --no-check-certificate \
         --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" \
         "http://download.oracle.com/otn-pub/java/jdk/8u40-b25/server-jre-8u40-linux-x64.tar.gz" 
    wget --no-cookies --no-check-certificate \
          --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" \
          "http://download.oracle.com/otn-pub/java/jce/8/jce_policy-8.zip"

    ansible-playbook -i hosts baserock_gerrit/instance-config.yml

#### Access control

Now, log into the new Gerrit instance with your OpenID. Make sure you're the
first one to have registered, and you will automatically have been added to the
Administrators group.

You can add more users into the Administrators group later on using the [gerrit
set-members] command, or the web interface.

Go to the settings page, 'HTTP Password' and generate a HTTP password for
yourself. You'll need it in the next step. The password can take a long time to
appear for some reason, or it might not work at all. Click off the page and
come back to it and it might suddenly have appeared. I've not investigated why
this happens.

Generate the SSH keys you need, if you don't have them.

    mkdir -p keys
    ssh-keygen -t rsa -b 4096 -C 'lorry@gerrit.baserock.org' -N '' -f keys/lorry-gerrit.key

Now set up the Gerrit access configuration. You'll need to have cloned 
<https://github.com/ssssam/ansible-gerrit> alongside infrastructure.git, and ran
`make` in that directory. You'll also need 'pygerrit' installed somewhere
Python can find it.

    export GERRIT_URL=http://gerrit.baserock.org
    export GERRIT_ADMIN_USERNAME=your username
    export GERRIT_ADMIN_PASSWORD=your generated HTTP password

    ANSIBLE_LIBRARY=../ansible-gerrit \
    ansible-playbook baserock_gerrit/gerrit-access-config.yml

As well as creating all the groups and initial users in the new Gerrit
instance, Ansible should update the file baserock_gerrit/All-Projects/groups,
which will be needed in the next step.

    git clone ssh://$GERRIT_ADMIN_USERNAME@gerrit.baserock.org:29418/All-Projects.git /tmp/All-Projects
    cp -a baserock_gerrit/All-Projects/*  /tmp/All-Projects
    cd /tmp/All-Projects
    git checkout -b config
    git commit -a -m "Set up initial access controls."
    git push origin config:refs/meta/config
    cd -

[gerrit set-members]: https://gerrit-documentation.storage.googleapis.com/Documentation/2.9.4/cmd-set-members.html

#### Mirroring

Run:

    ansible-playbook -i hosts baserock_gerrit/instance-mirroring-config.yml

Now clone the lorry-controller configuration repository, commit the
configuration file to it, and push.

    git clone ssh://$GERRIT_ADMIN_USERNAME@gerrit.baserock.org:29418/local-config/lorries.git /tmp/lorries
    cp baserock_gerrit/lorry-controller.conf /tmp/lorries
    cd /tmp/lorries
    git checkout -b master
    git add .
    git commit -m "Add initial Lorry Controller mirroring configuration"
    git push origin master
    cd -

Now SSH in as 'root' to gerrit.baserock.org, tunnelling the lorry-controller
webapp's port to your local machine:

    ssh -L 12765:localhost:12765 root@gerrit.baserock.org

Visit <http://localhost/1.0/status-html>. You should see the lorry-controller
status page. Click 'Re-read configuration', if there are any errors in the
configuration it'll tell you. If not, it should start mirroring stuff from
your Trove.

Create a Gitano account on the Trove you want to push changes to for the Gerrit
user. The `instance-config.yml` Ansible playbook will have generated an SSH
key. Do this on the Gerrit instance.

    ssh git@git.baserock.org user add gerrit "gerrit.baserock.org" gerrit@baserock.org
    ssh git@git.baserock.org as gerrit sshkey add main < ~gerrit/.ssh/id_rsa.pub

Add the user to the necessary -writers groups, so they can push updates to the
repos. The 'replication.config' file controls what the gerrit-replication plugin
tries to push. By default force-push is disabled and only 'master' and tags are
pushed.

    ssh git@git.baserock.org group adduser baserock-writers gerrit

Add the host key of the remote trove.

    sudo -u gerrit sh -c 'ssh-keyscan git.baserock.org >> ~gerrit/.ssh/known_hosts'

Check the account is working.

    sudo -u gerrit ssh git@git.baserock.org whoami

Now enable the gerrit-replication plugin, check that it's now in the list of
plugins, and manually start a replication cycle. You should see log output from
the final SSH command showing any errors.

    ssh $GERRIT_ADMIN_USERNAME@gerrit.baserock.org -p 29418 gerrit plugin enable replication
    ssh $GERRIT_ADMIN_USERNAME@gerrit.baserock.org -p 29418 gerrit plugin ls
    ssh $GERRIT_ADMIN_USERNAME@gerrit.baserock.org -p 29418 replication start --all --wait

### Storyboard

We use a slightly adapted version of
<https://github.com/openstack-infra/puppet-storyboard> to deploy Storyboard.

There's no development deployment for Storyboard at this time: the Puppet
script expects to start services using systemd, and that doesn't work by
default in a Docker container.

To deploy the production version:

    packer build -only=production baserock_storyboard/packer_template.json
    nova boot openid_provider
        --flavor dc1.1x1 --image 'baserock_storyboard' \
        --key-name=$keyname storyboard.baserock.org \
        --nic='net-id=d079fa3e-2558-4bcb-ad5a-279040c202b5'
        --security-groups default,web-server
        --user-data baserock-ops-team.cloud-config

Storyboard deployment does not yet work fully (you can manually kludge it into
working after deploying it, though).