summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* Add '--' when calling run-parts to specify the end of the options.baserock/pedroalvarez/system-integration-fixup2Pedro Alvarez2014-06-161-1/+1
| | | | Using '--' we prevent passing options in SYSTEM_INTEGRATION_PATH.
* Merge remote-tracking branch 'origin/sam/distbuild-gc-fix'Sam Thursfield2014-06-161-0/+3
|\ | | | | | | Reviewed-By: Lars Wirzenius <lars.wirzenius@codethink.co.uk>
| * Don't delete subdirectories of tempdir in gc pluginSam Thursfield2014-06-131-0/+3
| | | | | | | | | | | | There is code that assumes these exist in at least one place: StagingArea.abort(). That code should be fixed, but we should also stop deleting them every time we run 'morph gc'.
* | Update NEWS for releasebaserock-14.24Richard Maw2014-06-121-0/+14
| |
* | Check for presence of btrfs before trying to use itSam Thursfield2014-06-115-1/+49
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | If btrfs is not present in the kernel we end up with strange output like this: Error creating disk image2014-06-10 16:00:40 [devel-system-x86_64-generic][my-raw-disk-image][rawdisk.write]Failure to create disk image at /src/tmp/testdev.img ERROR: Command failed: mount -o loop /src/tmp/testdev.img /src/tmp/deployments/tmpQ7wXO1/tmp4lVDcu/tmpvHSzDE mount: mounting /dev/loop0 on /src/tmp/deployments/tmpQ7wXO1/tmp4lVDcu/tmpvHSzDE failed: Device or resource busy To avoid this confusing error, Morph should explicitly check first.
* | Merge branch 'baserock/richardipsum/distbuild_err_when_no_controller2'Richard Ipsum2014-06-113-12/+62
|\ \ | | | | | | | | | | | | | | | Reviewed by: Lars Wirzenius Richard Maw
| * | Make buildcommand use InitiatorConnectionMachineRichard Ipsum2014-06-111-2/+11
| | | | | | | | | | | | | | | | | | After a failed attempt to connect to the controller node the initiator will wait 30 seconds before attempting a reconnect, if this reconnect fails the initiator gives up.
| * | Import InitiatorConnectionMachineRichard Ipsum2014-06-111-1/+2
| | |
| * | Use superRichard Ipsum2014-06-111-1/+1
| | | | | | | | | | | | This change is made just for consistency.
| * | Add InitiatorConnectionMachineRichard Ipsum2014-06-111-5/+36
| | | | | | | | | | | | | | | | | | The InitiatorConnectionMachine wraps the ConnectionMachine, so we can continue to use ConnectionMachine without providing it with an app.
| * | Add max_retries optionRichard Ipsum2014-06-111-5/+14
|/ / | | | | | | | | | | | | By default there is no limit on the number of reconnection attempts. We make the reconnect_interval a parameter, but the default interval remains 1 second.
* | Fix: force select to wait on timeoutRichard Ipsum2014-06-101-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | Currently the mainloop unintentionally busy-waits if we can't connect to the controller. We want the mainloop's select call to wait for the timeout not for this socket's descriptors (which are always ready). We could just call stop_reading() and stop_writing() but since we won't be needing this socket again we may as well close the entire event source, which calls stop_reading(), stop_writing() and then closes the socket.
* | Note future improvement for fetching artifacts from remote cacheSam Thursfield2014-06-102-0/+7
| |
* | Merge branch 'sam/remote-artifact-cache-failure'Sam Thursfield2014-06-103-50/+60
|\ \ | | | | | | | | | | | | Reviewed-By: Richard Ipsum <richard.ipsum@codethink.co.uk> Reviewed-By: Richard Maw <richard.maw@codethink.co.uk>
| * | Fix up sam/remote-artifact-cache-failure branch after reviewSam Thursfield2014-06-101-3/+10
| | |
| * | Ensure that transferring an artifact from the remote cache is atomicSam Thursfield2014-06-031-15/+29
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Artifacts can have multiple parts; while this may not be an ideal design, changing the format of artifacts has implications for backwards compatibility. We should transfer all parts at once and delete them all if we encounter any errors, to reduce the change of getting the local artifact cache into an inconsistent state.
| * | Improve robustness when fetching artifacts from remote artifact cacheSam Thursfield2014-06-032-30/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously Morph would check if an artifact is present in the remote artifact cache, then fetch the necessary files. If an error occured during fetching, it would raise an error and abort. Instead, we should just try and fetch the files and if anything fails, move on to building locally. This avoids the situation where an error in the remote cache makes local building impossible, which we experienced recently.
| * | Tweak exception message of remote artifact cache GetErrorSam Thursfield2014-06-031-2/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Before: ERROR: Failed to get metadata meta for the artifact file:///src/ws-baserock-hawk/baserock/ps/build-system/baserock/baserock/definitions|refs/heads/baserock/builds/778b1a370a1f43c497c1354a2a949de1/56c9ec89d09240fd80faa7d2226b7eda|core|core-devel from the artifact cache http://git.baserock.org:8080/ After: ERROR: Failed to get metadata meta for the artifact f896a081beacd4a99ded38d28b44fbf02970038fb53349265f85f8f3298ead9d.stratum.core-devel from the artifact cache http://git.baserock.org:8080/ When debugging artifact cache issues, the information that's most useful is the filename of the artfact.
| * | Make all remote artifact cache get errors inherit from GetErrorSam Thursfield2014-06-031-3/+3
| | | | | | | | | | | | This saves from having to catch three separate exceptions.
* | | Merge branch 'baserock/pedroalvarez/deploy-dont-update-gits'Pedro Alvarez2014-06-101-0/+1
|\ \ \ | | | | | | | | | | | | | | | | Reviewed-by: Adam Coldrick Reviewed-by: Pedro Alvarez
| * | | Morph deploy should never update git reposPaul Sherwood2014-06-101-0/+1
|/ / / | | | | | | | | | | | | | | | | | | | | | The deploy plugin is relying on existing code which respects the 'no-git-update' setting and updates gits if it is not set. Since deploy can only work for built systems anyway, we can force this True for deployments, to avoid wasting the user/caller's time.
* | | Merge branch 'baserock/richardmaw/S11226/yarns-v2'Richard Maw2014-06-093-2/+128
|\ \ \ | |_|/ |/| | | | | Reviewed-by: Sam Thursfield and Richard Ipsum
| * | Add yarn to test initramfs deploymentsRichard Maw2014-06-062-0/+126
| | |
| * | initramfs.write: create parent directories of locationRichard Maw2014-06-061-2/+2
| | |
* | | Merge branch 'baserock/ps/simple-edit-syntax'Sam Thursfield2014-06-0925-179/+82
|\ \ \ | | | | | | | | | | | | | | | | | | | | | | | | Reviewed-By: Adam Coldrick <adam.coldrick@codethink.co.uk> Reviewed-By: Richard Ipsum <richard.ipsum@codethink.co.uk> Reviewed-By: Richard Maw <richard.maw@codethink.co.uk> Reviewed-By: Sam Thursfield <sam.thursfield@codethink.co.uk>
| * | | Fix up before mergeSam Thursfield2014-06-092-37/+44
| | | | | | | | | | | | | | | | | | | | | | | | | | | | - All tests now pass - The odd case of chunks with the same name but different repo URLs now correctly informs the user of the multiple checkouts that were done. - Tidyups
| * | | Fix tests for new `morph edit` syntaxPaul Sherwood2014-06-0923-70/+33
| | | |
| * | | Add messages to help user know what is happeningPaul Sherwood2014-06-081-0/+16
| | | |
| * | | Simplify morph edit syntax and logicPaul Sherwood2014-06-081-124/+41
|/ / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Since we removed ref: fields from the system morph files, some of the logic in morph edit is no longer needed. In particular, running morph edit <system> <stratum> is a no-op. This is because the version of <system> and <stratum> are now implicit from the context of what we are doing. In other words we're always working with the current version of <system> and <stratum> from the system branch we are in. Because of the complexity of morph's logic, we didn't notice this when dropping the ref: fields, and we missed the opportunity to simplify our logic for 'morph edit' This patch aims to provide the simplest possible implementation of morph edit: morph edit <chunk> It checks all strata for instances of <chunk>, and does what morph edit should do for the instances it finds. A later patch can add warnings to help users deal with situations where <chunk> is not found, or is found more than once. Also since this changes the syntax of morph, it breaks many of our tests. Later patches will address that.
* | | Make morph gc clear out deployments in tmpdirPaul Sherwood2014-06-071-1/+6
|/ / | | | | | | | | | | | | | | Generally deployment temp dirs are removed by morph during deployment, but in some cases deployment dirs may not be cleared up, for example if morph gets a SIGKILL or something unexpected happens that causes morph to terminate without having a chance to cleanup.
* | Merge branch 'baserock/richardmaw/S11226/writeext-fstab'Richard Maw2014-06-061-5/+7
|\ \ | | | | | | | | | | | | Reviewed-by: Pedro Alvarez Reviewed-by: Sam Thursfield
| * | Use UUID in fstab entries if providedRichard Maw2014-06-041-4/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This makes systems use the UUID of the disk in the fstab when there is no pre-existing fstab entry for /. This happens whether the system has an initramfs or not, since it should be harmless, and by this point we're in userland, so can know what the UUIDs are. Passing the UUID to `complete_fstab_for_btrfs_layout` is optional, and defaults to the old behaviour of using /dev/sda, since it is called directly by some write extensions for doing upgrades, and upgrading systems that use an initramfs will be part of a later patch.
| * | Make uuid mandatory when calling create_btrfs_system_layoutRichard Maw2014-06-041-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | It's only ever called from the core of the write extensions library, so we can change it to be mandatory in all cases. install_extlinux is left with the uuid being optional and defaulting to /dev/sda, since it is called directly from the rawdisk write extension currently, and upgrades of systems that have an initramfs will be part of a later patch series.
* | | Update NEWS for Baserock 14.23 releasebaserock-14.23Adam Coldrick2014-06-051-0/+10
| | |
* | | Merge branch 'sam/list-artifacts'Sam Thursfield2014-06-053-143/+125
|\ \ \ | | | | | | | | | | | | | | | | Reviewed-By: Adam Coldrick <adam.coldrick@codethink.co.uk> Reviewed-By: Richard Maw <richard.maw@codethink.co.uk>
| * | | Rewrite copy_artifacts plugin as list_artifactsSam Thursfield2014-06-053-143/+125
| | |/ | |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The copy-artifacts and list-artifacts commands are mainly useful for making releases. As part of the release process we copy artifacts for the entire build graph of the release to the artifact cache on trove.baserock.org, to provide Baserock users with ready-built artifacts. This part of the release process is now automated, and the automation require the list-artifacts command to function as a 'plumbing' command. The copy-artifacts command is no longer required. It can be replaced with: morph list-artifacts --quiet REPO REF MORPH | rsync --files-from=- $TARGET The previous version of this plugin looked in the system artifact's metadata for the list of artifacts. This is flawed as the final system does not necessarily contain every build dependency. The new version of the plugin calculates the build graph from source, using the same process as the 'buildcommand' module. It also required looking in Morph's artifact cache for the system artifact file to analyse.
* | | Move presence check for job into remove functionRichard Ipsum2014-06-041-6/+6
| | | | | | | | | | | | We always want to warn if we attempt to remove a job that's not present
* | | Add comment to explain the use of _JobFailed eventRichard Ipsum2014-06-041-0/+6
| | |
* | | Merge branch 'baserock/richardipsum/distbuild_state_fix'Richard Ipsum2014-06-041-23/+28
|\ \ \ | | | | | | | | | | | | | | | | | | | | | | | | Reviewed by: Sam Thursfield Adam Coldrick Richard Maw
| * | | Make Jobs finish when caching is completeRichard Ipsum2014-06-031-24/+24
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | If a new build request makes a request for an artifact that is currently being cached then the artifact will be needlessly rebuilt. To avoid this the new build request should wait for caching to finish. We rename _ExecStarted, _ExecEnded, _ExecFailed to _JobStarted, _JobFinished, _JobFailed and Job's is_building attribute is renamed to running.
| * | | Make job fail if caching failsRichard Ipsum2014-06-031-0/+5
| |/ / | | | | | | | | | | | | This fixes the bug that causes the distbuild controller to crash when population of the artifact cache fails.
* | | Improve error message in _notify_annotation_failedRichard Ipsum2014-06-041-4/+6
| |/ |/|
* | Merge branch 'baserock/richardmaw/S11159/initramfs-support'Richard Maw2014-06-036-8/+128
|\ \ | |/ |/| | | Reviewed-by: Sam Thursfield and Richard Ipsum
| * Add initramfs support to write extensions that produce disksRichard Maw2014-06-024-8/+66
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | If INITRAMFS_PATH is specified and the file exists, then the produced kernel command line will use root=UUID=$uuid_of_created_disk rather than root=/dev/sda, which may be incorrect. Help files have been updated to mention the new option. This leads to an unfortunate duplication of the path to the initramfs, in both the location field of the nested deployment and the INITRAMFS_PATH of the disk image creation. However, an initramfs could be produced by a chunk and put in the same place, so it doesn't make sense to couple the rawdisk and initramfs write extensions to remove this duplication. Similarly, there may be multiple valid initramfs in the rootfs e.g. extlinux loads a hypervisor, which is Linux + initramfs, and the initramfs then boots a guest Linux system, which uses a different initramfs. This makes it important to explicitly let the rootfs write extensions know which to use, or not as the case may be. util-linux's blkid is required, since the busybox version ignores the options to filter its output, and parsing the output is undesirable. Because syslinux's btrfs subvolume support is limited to being able to use a non-0 default subvolume, the initramfs has to be copied out of the run-time rootfs subvolume and into the boot subvolume. This pushed the required disk space of a minimal system over the 512M threshold because we do not have the userland tooling support to be able to do a btrfs file contents clone.
| * Add initramfs write extensionRichard Maw2014-06-022-0/+62
|/ | | | | | | | | | | | | | | This creates a gzipped cpio archive that may be used as an initramfs. It is hard-coded to use gzip to compress the initramfs, since it's the most common way to do it. This is unfortunate, since the busybox gzip utility only allows maximum compression, which is rather slow and doesn't give progress reporting, so you can easily think it's gotten stuck. It's possible to use other compression formats, but they need the kernel to be built with them supported, and in the case of lz4, unusual userland tools to create it, since the version of lz4 supported in the kernel is not what the standard lz4 tools produce.
* Update NEWS for Baserock 14.22 releasebaserock-14.22Sam Thursfield2014-05-291-0/+12
|
* Merge remote-tracking branch 'origin/sam/ssh-rsync-check'Sam Thursfield2014-05-232-26/+24
|\ | | | | | | | | Reviewed-By: Richard Maw <richard.maw@codethink.co.uk> Reviewed-By: Lars Wirzenius <lars.wirzenius@codethink.co.uk>
| * deploy: Do sanity checks earlier in ssh-rsync (upgrade) extensionSam Thursfield2014-05-202-26/+24
| |
* | Fix long lineDaniel Silverstone2014-05-161-1/+2
| |
* | Merge branch 'baserock/danielsilverstone/vagrant-support'Daniel Silverstone2014-05-162-8/+16
|\ \ | |/ |/|