Add a simple data backup mechanism

The technique used is: create a new SSH key for backup automation, and
authorize it to log in as 'root' to instances.

To reduce potential harm if the key somehow gets compromised, it is
limited to logging in from a single IP, and it is limited to running
the 'backup-snapshot' program on the instances.

Inside each instance, the `backup-snapshot` script is used as a wrapper
for the `rsync --server` process. This script pauses running services,
takes a snapshot of the data volume, and then runs the RSync server.

Change-Id: I3c98ffe3dc2fa1373bd0df2388145636e491bf57

3 files changed, 58 insertions, 0 deletions

diff --git a/baserock_backup/backup.sh b/baserock_backup/backup.sh
new file mode 100755
index 00000000..f16ba447
--- /dev/null
+++ b/baserock_backup/backup.sh
@@ -0,0 +1,25 @@
+#!/bin/sh
+
+# These aren't normal invocations of rsync: the targets use the
+# 'command' option in /root/.ssh/authorized_keys to force execution of
+# the 'backup-snapshot' script at the remote end, which then starts the
+# rsync server process. So the backup SSH key can only be used to make
+# backups, nothing more.
+
+# Don't make the mistake of trying to run this from a systemd unit. There is
+# some brokenness in systemd that causes the SSH connection forwarding to not
+# work, so you will not be able to connect to the remote machines.
+
+# Database
+/usr/bin/rsync --archive --delete-before --delete-excluded \
+    --hard-links --human-readable --progress --sparse \
+    root@192.168.222.30: /srv/backup/database
+date > /srv/backup/database.timestamp
+
+# Gerrit
+/usr/bin/rsync --archive --delete-before --delete-excluded \
+    --hard-links --human-readable --progress --sparse \
+    --exclude='cache/' --exclude='tmp/' \
+    root@192.168.222.69: /srv/backup/gerrit
+date > /srv/backup/gerrit.timestamp
+
diff --git a/baserock_backup/instance-config.yml b/baserock_backup/instance-config.yml
new file mode 100644
index 00000000..327b84e9
--- /dev/null
+++ b/baserock_backup/instance-config.yml
@@ -0,0 +1,29 @@
+# Configuration for a machine that runs data backups of baserock.org.
+#
+# The current backup machine is not a reproducible deployment, but this
+# playbook should be easily adaptable to produce a properly reproducible
+# one.
+---
+- hosts: baserock-backup1
+  gather_facts: false
+  tasks:
+    - name: user for running backups
+      user: name=backup
+
+    # You'll need to copy in the SSH key manually for this user.
+
+    - name: SSH config for backup user
+      copy: src=ssh_config dest=/home/backup/.ssh/config
+
+    - name: backup script
+      copy: src=backup.sh dest=/home/backup/backup.sh mode=755
+
+    # You will need https://github.com/ansible/ansible-modules-core/pull/986
+    # for this to work.
+    - name: backup cron job, runs every day at midnight
+      cron:
+        hour: 00
+        minute: 00
+        job: /home/backup/backup.sh
+        name: baserock.org data backup
+        user: backup
diff --git a/baserock_backup/ssh_config b/baserock_backup/ssh_config
new file mode 100644
index 00000000..e14b38a0
--- /dev/null
+++ b/baserock_backup/ssh_config
@@ -0,0 +1,4 @@
+# SSH configuration to route all requests to baserock.org systems
+# via the frontend system, 185.43.218.170.
+Host 192.168.222.*
+  ProxyCommand ssh backup@185.43.218.170 -W %h:%p