summaryrefslogtreecommitdiff
path: root/morphlib
Commit message (Collapse)AuthorAgeFilesLines
...
| * | Make uuid mandatory when calling create_btrfs_system_layoutRichard Maw2014-06-041-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | It's only ever called from the core of the write extensions library, so we can change it to be mandatory in all cases. install_extlinux is left with the uuid being optional and defaulting to /dev/sda, since it is called directly from the rawdisk write extension currently, and upgrades of systems that have an initramfs will be part of a later patch series.
* | | Merge branch 'sam/list-artifacts'Sam Thursfield2014-06-052-142/+124
|\ \ \ | |/ / |/| | | | | | | | Reviewed-By: Adam Coldrick <adam.coldrick@codethink.co.uk> Reviewed-By: Richard Maw <richard.maw@codethink.co.uk>
| * | Rewrite copy_artifacts plugin as list_artifactsSam Thursfield2014-06-052-142/+124
| |/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The copy-artifacts and list-artifacts commands are mainly useful for making releases. As part of the release process we copy artifacts for the entire build graph of the release to the artifact cache on trove.baserock.org, to provide Baserock users with ready-built artifacts. This part of the release process is now automated, and the automation require the list-artifacts command to function as a 'plumbing' command. The copy-artifacts command is no longer required. It can be replaced with: morph list-artifacts --quiet REPO REF MORPH | rsync --files-from=- $TARGET The previous version of this plugin looked in the system artifact's metadata for the list of artifacts. This is flawed as the final system does not necessarily contain every build dependency. The new version of the plugin calculates the build graph from source, using the same process as the 'buildcommand' module. It also required looking in Morph's artifact cache for the system artifact file to analyse.
* | Add initramfs support to write extensions that produce disksRichard Maw2014-06-024-8/+66
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | If INITRAMFS_PATH is specified and the file exists, then the produced kernel command line will use root=UUID=$uuid_of_created_disk rather than root=/dev/sda, which may be incorrect. Help files have been updated to mention the new option. This leads to an unfortunate duplication of the path to the initramfs, in both the location field of the nested deployment and the INITRAMFS_PATH of the disk image creation. However, an initramfs could be produced by a chunk and put in the same place, so it doesn't make sense to couple the rawdisk and initramfs write extensions to remove this duplication. Similarly, there may be multiple valid initramfs in the rootfs e.g. extlinux loads a hypervisor, which is Linux + initramfs, and the initramfs then boots a guest Linux system, which uses a different initramfs. This makes it important to explicitly let the rootfs write extensions know which to use, or not as the case may be. util-linux's blkid is required, since the busybox version ignores the options to filter its output, and parsing the output is undesirable. Because syslinux's btrfs subvolume support is limited to being able to use a non-0 default subvolume, the initramfs has to be copied out of the run-time rootfs subvolume and into the boot subvolume. This pushed the required disk space of a minimal system over the 512M threshold because we do not have the userland tooling support to be able to do a btrfs file contents clone.
* | Add initramfs write extensionRichard Maw2014-06-022-0/+62
|/ | | | | | | | | | | | | | | This creates a gzipped cpio archive that may be used as an initramfs. It is hard-coded to use gzip to compress the initramfs, since it's the most common way to do it. This is unfortunate, since the busybox gzip utility only allows maximum compression, which is rather slow and doesn't give progress reporting, so you can easily think it's gotten stuck. It's possible to use other compression formats, but they need the kernel to be built with them supported, and in the case of lz4, unusual userland tools to create it, since the version of lz4 supported in the kernel is not what the standard lz4 tools produce.
* Merge remote-tracking branch 'origin/sam/ssh-rsync-check'Sam Thursfield2014-05-232-26/+24
|\ | | | | | | | | Reviewed-By: Richard Maw <richard.maw@codethink.co.uk> Reviewed-By: Lars Wirzenius <lars.wirzenius@codethink.co.uk>
| * deploy: Do sanity checks earlier in ssh-rsync (upgrade) extensionSam Thursfield2014-05-202-26/+24
| |
* | Fix long lineDaniel Silverstone2014-05-161-1/+2
| |
* | Fix state subvolume generator to preserve permissionsbaserock/danielsilverstone/vagrant-supportDaniel Silverstone2014-05-161-1/+1
| | | | | | | | | | | | shutil.move() does not preserve permissions, file modes, ownerships etc, resulting in much confusion when prepopulating a non-root user during deployment. This change to `mv` fixes that.
* | VirtualBox Write Extension: Vagrant supportDaniel Silverstone2014-05-161-7/+15
|/ | | | | | | Add support to the VirtualBox write extension to notice if we are doing a Vagrant Basebox installation and not do the clever network setup we normally do to allow machines to talk to one another since this confuses Vagrant quite a bit if it is left in.
* Add distbuild-morphology command.Sam Thursfield2014-05-141-0/+23
| | | | | | | | | This is useful to build releases using distbuild. It avoids having the SHA1 fields in the artifact metadata files pointing to commits that exist only on temporary build branches. It also avoids file:// URLs in the repo fields. Note that the repo URL still points to the Trove used by the distbuild network, rather than being an upstream URL pointing to git.baserock.org.
* Merge branch 'baserock/richardipsum/build-log-on-stdout-tee2'Richard Ipsum2014-05-145-26/+73
|\ | | | | | | | | | | | | Reviewed by: Lars Wirzenius Sam Thursfield Daniel Silverstone
| * Use logfile kwarg to generate build logRichard Ipsum2014-05-141-24/+47
| | | | | | | | | | | | | | | | | | We could just set stdout to subprocess.PIPE then read from the pipe, but then we won't get the output till the command's finished and some commands take a long time. Using the logfile kwarg a file will be created by tee and the output will be written to it in 'real time'.
| * Add logfile kwarg to staging area runcmdRichard Ipsum2014-05-141-1/+8
| | | | | | | | | | | | | | | | | | | | | | | | We use tee to write the output to a file as well as to stdout. Using Popen it should be straight forward to send the output to a pipe and then read that pipe and write to wherever. At the moment morph uses cliapp's runcmd rather than Popen. cliapp's runcmd is a blocking call, so we're not able to read from the pipe until the command has completed, which prevents real time logging to a number of files. One solution to this problem might be to spawn a thread which opens a pipe to the command being executed, the thread then reads from the pipe and writes to our collection of logfiles.
| * Add --build-log-on-stdout flagRichard Ipsum2014-05-141-0/+3
| |
| * Add test for get_source_metadata_filenameRichard Ipsum2014-05-141-0/+11
| |
| * Add get_source_metadata_filenameRichard Ipsum2014-05-141-1/+4
| |
* | Add help for install-files.configureRichard Ipsum2014-05-092-28/+60
|/ | | | | Move the help out of the comment and into a help file, and add a clearer example.
* Fix bad indentationRichard Ipsum2014-05-071-2/+2
|
* A more useful morphology syntax error messageRichard Ipsum2014-05-071-4/+3
| | | | | There's no longer any need to log the exception separately, it will be logged with the MorphologySyntaxError
* fix the Virtualbox version checkingPaul Sherwood2014-04-201-2/+2
|
* deploy: Extra checks for KVM deploymentsSam Thursfield2014-04-151-0/+21
| | | | | Test that a VM with the given name does not already exist, and check that the files specified in ATTACH_DISKS do already exist.
* deploy: Share SSH connectivity check in the common writeexts.py codeSam Thursfield2014-04-153-10/+11
| | | | | Also, change it to log the real error message in morph.log before raising a more general exception to the user.
* deploy: Check SSH connection for KVM deployment before startingSam Thursfield2014-04-142-6/+32
| | | | | | Slight duplication is necessary, but it's only a few lines. We could move the duplicated code into the base class in 'morphlib.writeexts' if there was more duplication.
* distbuild: Label state machine transition tablesSam Thursfield2014-04-141-0/+1
| | | | Makes it easier to see what they mean at a glance.
* distbuild: Stop workers blowing their entire cache away every timeDaniel Silverstone2014-04-141-6/+14
|
* Tweak some user-facing error messages in 'gitdir'Sam Thursfield2014-04-081-6/+8
| | | | | | | | | | | | | | | The default Python __repr__() is useless for both developers and users, so avoid using it in error messages. The following message: ERROR: Push to remote <morphlib.gitdir.Remote object at 0x905096c>, push url None with refspecs (<morphlib.gitdir.RefSpec object at 0xf72fec2c>,) failed with exit code 128 Becomes: ERROR: Push to remote "origin", push url None failed with exit code 128
* Adding support to add extra kernel args in extlinux.confPedro Alvarez2014-04-041-1/+5
|
* Merge branch 'baserock/adamcoldrick/git-fat-handle-whitespace-rebase'Adam Coldrick2014-04-031-1/+21
|\ | | | | | | | | | | | | | | Signed-off by: Adam Coldrick Reviewed by: Daniel Silverstone <daniel.silverstone@codethink.co.uk> Richard Maw <richard.maw@codethink.co.uk>
| * Escape special characters and handle whitespace in gitattributesbaserock/adamcoldrick/git-fat-handle-whitespace-rebaseAdam Coldrick2014-04-021-1/+21
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Since git's parsing of .gitattributes can't deal with spaces, or even with quotes, replace any whitespace with [[:space:]]. Also, make sure to escape any glob metacharacters to avoid accidental pattern matches. Note: This approach could cause a file "foo<tab>bar" to be treated as a binary if a file "foo bar" has already been added with morph add-binary. However this is as good of a workaround as I can find. Signed-off by: Adam Coldrick
* | Add help files for nfsboot, rawdisk and tar.baserock/markdoffman/s10617/add-yaml-help-optionMark Doffman2014-04-013-0/+24
| | | | | | | | | | | | Begin adding help documentation for configuration and write extensions starting with nfsboot.write, rawdisk.write and tar.write.
* | Add write and configuration extensions to help.Mark Doffman2014-03-311-32/+45
| | | | | | | | | | | | Add a command 'help-extensions' to list all extensions. Add the ability to find help on an extension by calling 'morph help [extension name]'.
* | Add utilities for listing and finding extensions.Mark Doffman2014-03-313-43/+176
| | | | | | | | | | | | | | | | | | Add a module to morphlib that can list all write and configuration extensions either in morph itself or the morphology repository. The module also contains methods to find an extension filename from the name and type.
* | CacheKeyComputer Tests: Update to explicitly test for memoisationbaserock/danielsilverstone/memoise-cache-keys-betterDaniel Silverstone2014-03-281-0/+13
| | | | | | | | | | | | | | | | | | | | Explicitly test that we get the same result twice when computing ids and keys for the same artifact. This has the effect of verifying that the memoisation code is correctly operating. Signed-Off-By: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
* | CacheKeyComputer: Memoise cache keys more effectively.Daniel Silverstone2014-03-281-10/+23
|/ | | | | | | | | We were previously memoising the computation of the dictionaries but this patch adds support for memoising the computation of the cache key itself (the SHA string). This massively improves cache key computation performance. Signed-Off-By: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
* Merge branch 'baserock/richardipsum/merge_distbuild'Richard Ipsum2014-03-275-10/+368
|\ | | | | | | | | | | | | | | | | | | Conflicts: morphlib/plugins/deploy_plugin.py without-test-modules Reviewed by: Richard Maw Lars Wirzenius
| * UI change: distbuild has its own subcommandRichard Ipsum2014-03-261-7/+39
| |
| * Remove Initiator and InitiatorBuildCommandRichard Ipsum2014-03-261-49/+0
| |
| * Bring InitiatorBuildCommand into buildcommandRichard Ipsum2014-03-261-0/+30
| |
| * Fix deployRichard Ipsum2014-03-261-2/+3
| | | | | | | | | | | | | | | | | | tarfile's open needs the file-like object to have a tell() method, objects returned from sockets don't have this method. So instead we fetch the artifact from the remote and cache it locally
| * Make InitiatorBuildCommand construct BuildCommandRichard Ipsum2014-03-261-0/+1
| | | | | | | | | | | | | | The constructor for BuildCommand sets up the caches, we want the caches to be set up for distbuild too so that we can deploy a system (system artifact will be fetched from the artifact cache)
| * Fix push_build_branchesRichard Ipsum2014-03-261-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | A semantic error in the BuildBranch class meant that it was not possible to push temporary branches. This escaped testing since BuildBranch interacts too tightly with other components to be easily unit-tested, so testing was deferred to a yarn test. However, coverage isn't measured in yarn tests, so this code path was forgotten.
| * Add the distbuild pluginRichard Ipsum2014-03-211-0/+343
| |
* | Stop using GitDirectory().list_files() to check for .gitfatAdam Coldrick2014-03-251-1/+1
| | | | | | | | | | This function causes a UnicodeDecodeError for some repositories when building. Use os.path.isfile() when looking for .gitfat instead.
* | Use os.environ[] instead of os.environ.get() to read a required variableSam Thursfield2014-03-241-1/+1
| |
* | Be consistent with nfsboot_root pathSam Thursfield2014-03-241-3/+4
| | | | | | | | | | We were building it from a variable in some places and hardcoding it in others; now we build it from a variable everywhere.
* | Merge branch 'sam/nfs-deploy-fixes'Richard Ipsum2014-03-203-60/+74
|\ \ | | | | | | | | | | | | | | | Reviewed by: Richard Maw Lars Wirzenius
| * | Make sanity checks for nfsboot deployments run earlierSam Thursfield2014-03-182-55/+70
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Move some code to the '.check' extension to verify that the deployment can happen *before* spending 5 minutes unpacking and configuring the rootfs. This is not a perfect solution yet because when multiple systems are being deployed in a cluster, we do not check all systems and then deploy them all. Instead, we check one, then deploy it, then check the second, etc.
| * | Improve comment in nfsboot.configureSam Thursfield2014-03-181-1/+3
| | |
| * | Remove fstab hack from nfsboot.configureSam Thursfield2014-03-181-4/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | There is no longer a default /etc/fstab in the Baserock fhs-dirs chunk, and the nfsboot.write extension does not use the default Btrfs system layout so no entry is added for / to /etc/fstab at deploy-time. We cannot have / in /etc/fstab for nfsboot deployments because it causes systemd to remount / during bootup, which breaks everything.
View Lorry Controller Status