distbuild: Kill the whole process tree when cancelling a build

We discovered a case where a user of distbuild began a build of
'qtbase', then cancelled it 2 minutes in. The `morph worker-build`
process didn't exit for over an hour -- it ran right through until the
chunk artifacts had been created. Then it exited with code -9 (SIGKILL).

This seems to be due to the fact that SIGKILL doesn't kill subprocesses,
and so any file descriptors the subprocesses have open will remain open.

If we set up the `morph worker-build` process as a process group
leader, using os.setpgid(), then we can use os.killpg() to kill the
entire process group. This should ensure that the `morph worker-build`
command exits straight away, as all of its subprocesses will be killed
at the same time it is.

Change-Id: I38707d18004d8c5bc994fd0cb99e90fd5def58e4

2 files changed, 5 insertions, 2 deletions

diff --git a/distbuild-helper b/distbuild-helper
index 76a39d5f..1f648dd4 100755
--- a/distbuild-helper
+++ b/distbuild-helper
@@ -131,6 +131,7 @@ class HelperMachine(distbuild.StateMachine):
             'JsonMachine: exec request: stdin=%s', repr(stdin_contents))
 
         p = subprocess.Popen(argv,
+                             preexec_fn=os.setpgrp,
                              stdin=subprocess.PIPE,
                              stdout=subprocess.PIPE,
                              stderr=subprocess.PIPE)
diff --git a/distbuild/subprocess_eventsrc.py b/distbuild/subprocess_eventsrc.py
index 52121502..e025161e 100644
--- a/distbuild/subprocess_eventsrc.py
+++ b/distbuild/subprocess_eventsrc.py
@@ -16,6 +16,8 @@
 
 
 import logging
+import os
+import signal
 
 import distbuild
 
@@ -92,8 +94,8 @@ class SubprocessEventSource(distbuild.EventSource):
         logging.debug('SES: Killing all processes for %s', request_id)
         for id, process in self.procs:
             if id == request_id:
-                logging.debug('SES: killing %s', repr(process))
-                process.kill()
+                logging.debug('SES: killing process group of %r', process)
+                os.killpg(process.pid, signal.SIGKILL)
 
     def close(self):
         self.procs = []