Transfer sparse files faster for kvm, vbox deploymentbaserock/liw/xfer-hole

The KVM and VirtualBox deployments use sparse files for raw disk
images. This means they can store a large disk (say, tens or hundreds
of gigabytes) without using more disk space than is required for the
actual content (e.g., a gigabyte or so for the files in the root
filesystem). The kernel and filesystem make the unwritten parts of the
disk image look as if they are filled with zero bytes. This is good.

However, during deployment those sparse files get transferred as if
there really are a lot of zeroes. Those zeroes take a lot of time to
transfer. rsync, for example, does not handle large holes efficiently.

This change introduces a couple of helper tools (morphlib/xfer-hole
and morphlib/recv-hole), which transfer the holes more efficiently.
The xfer-hole program reads a file and outputs records like these:

    DATA
    123
    binary data (exaclyt 123 bytes and no newline at the end)

    HOLE
    3245

xfer-hole can do this efficiently, without having to read through all
the zeroes in the holes, using the SEEK_DATA and SEEK_HOLE arguments
to lseek.

Using this, the holes take only take a few bytes each, making it
possible to transfer a disk image faster. In my benchmarks,
transferring a 100G byte disk image took about 100 seconds for KVM,
and 220 seconds for VirtualBox (which needs to more work at the
receiver to convert the raw disk to a VDI). Both benchmarks were from
a VM on my laptop to the laptop itself.

The interesting bit here is that the receiver (recv-hole) is simple
enough that it can be implemented in a bit of shell script, and the
text of the shell script can be run on the remote end by giving it to
ssh as a command line argument. This means there is no need to install
any special tools on the receiver, which makes using this improvement
much simpler.

1 files changed, 107 insertions, 0 deletions

diff --git a/morphlib/xfer-hole b/morphlib/xfer-hole
new file mode 100755
index 00000000..08222322
--- /dev/null
+++ b/morphlib/xfer-hole
@@ -0,0 +1,107 @@
+#!/usr/bin/env python
+
+
+import errno
+import os
+import sys
+
+
+SEEK_DATA = 3
+SEEK_HOLE = 4
+
+
+filename = sys.argv[1]
+fd = os.open(filename, os.O_RDONLY)
+pos = 0
+
+
+DATA = 'data'
+HOLE = 'hole'
+EOF = 'eof'
+
+
+def safe_lseek(fd, pos, whence):
+    try:
+        return os.lseek(fd, pos, whence)
+    except OSError as e:
+        if e.errno == errno.ENXIO:
+            return -1
+        raise
+
+
+def current_data_or_pos(fd, pos):
+    length = safe_lseek(fd, 0, os.SEEK_END)
+    next_data = safe_lseek(fd, pos, SEEK_DATA)
+    next_hole = safe_lseek(fd, pos, SEEK_HOLE)
+
+    if pos == length:
+        return EOF, pos
+    elif pos == next_data:
+        return DATA, pos
+    elif pos == next_hole:
+        return HOLE, pos
+    else:
+        assert False, \
+            ("Do not understand: pos=%d next_data=%d next_hole=%d" %
+             (pos, next_data, next_hole))
+
+
+def next_data_or_hole(fd, pos):
+    length = safe_lseek(fd, 0, os.SEEK_END)
+    next_data = safe_lseek(fd, pos, SEEK_DATA)
+    next_hole = safe_lseek(fd, pos, SEEK_HOLE)
+
+    if pos == length:
+        return EOF, pos
+    elif pos == next_data:
+        # We are at data.
+        if next_hole == -1 or next_hole == length:
+            return EOF, length
+        else:
+            return HOLE, next_hole
+    elif pos == next_hole:
+        # We are at a hole.
+        if next_data == -1 or next_data == length:
+            return EOF, length
+        else:
+            return DATA, next_data
+    else:
+        assert False, \
+            ("Do not understand: pos=%d next_data=%d next_hole=%d" %
+             (pos, next_data, next_hole))
+
+
+def find_data_and_holes(fd):
+    pos = safe_lseek(fd, 0, os.SEEK_CUR)
+
+    kind, pos = current_data_or_pos(fd, pos)
+    while kind != EOF:
+        yield kind, pos
+        kind, pos = next_data_or_hole(fd, pos)
+    yield kind, pos
+
+
+def make_xfer_instructions(fd):
+    prev_kind = None
+    prev_pos = None
+    for kind, pos in find_data_and_holes(fd):
+        if prev_kind == DATA:
+            yield (DATA, prev_pos, pos)
+        elif prev_kind == HOLE:
+            yield (HOLE, prev_pos, pos)
+        prev_kind = kind
+        prev_pos = pos
+
+
+def copy_slice_from_file(to, fd, start, end):
+    safe_lseek(fd, start, os.SEEK_SET)
+    data = os.read(fd, end - start)
+    to.write(data)
+    
+
+for kind, start, end in make_xfer_instructions(fd):
+    if kind == HOLE:
+        sys.stdout.write('HOLE\n%d\n' % (end - start))
+    elif kind == DATA:
+        sys.stdout.write('DATA\n%d\n' % (end - start))
+        copy_slice_from_file(sys.stdout, fd, start, end)