summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* Improve cache messagesam/adamcoldrick-ostree-rebaseSam Thursfield2015-04-011-3/+2
| | | | Change-Id: I1bb7ea175642ea43833503353d1f0cb866831d95
* Revert "WIP: Rough-and-ready method of reporting progress in artifact downloads"Sam Thursfield2015-03-304-49/+7
| | | | This reverts commit 923e4668b1f1d394e7d9e90d84e57e1138e544ff.
* WIP: Rough-and-ready method of reporting progress in artifact downloadsSam Thursfield2015-03-304-7/+49
| | | | Change-Id: I37d158ee8786dcd8774cde07e9385939f043e905
* Use BARE mode OStree repoSam Thursfield2015-03-301-1/+1
| | | | | | | This is *much* faster for build systems. ARCHIVE_Z2 is meant for servers, not for build systems. Change-Id: Ife24a137cf0e438cdf91eea2c647ecf8571e58e9
* Improve downloading artifacts for OSTree artifact cacheSam Thursfield2015-03-301-23/+25
| | | | | | | | | | | | | | | This fixes two issues: first, downloads were being put into the default TMPDIR, which is an in-memory file system. When large artifacts are put into /tmp such as gcc-devel, which is over 400MB, I found that my system became unusable due to out of memory / swap death. Ideally we'd fix Linux's memory management, but a more practical solution for now is to download the files to the artifact cache directory. I've not noticed any loss in speed from this change. Second, temporary files should always cleaned up now, even when there are errors during transfer or extraction. Change-Id: I227e8fbe0a9375dcc2f7534f95afb10f055d8d3f
* Output more information in --verbose modeSam Thursfield2015-03-253-11/+24
| | | | | | | | | | | | | | | | | | | Previously this info was just in the log file, but I think it should be on the console too. Also, the 'Downloading xx as tarball' message would appear even when there was no tarball to download, which was a bit weird. Output with `morph build --verbose` now looks like this: 2015-03-25 15:00:53 [Build 8/226] [stage2-gcc] Fetching to local cache: artifact stage2-gcc-locale 2015-03-25 15:00:53 [Build 8/226] [stage2-gcc] Downloading stage2-gcc-locale as a tarball. 2015-03-25 15:00:53 [Build 8/226] [stage2-gcc] Committing stage2-gcc-locale to artifact cache at 14ae4efeeaeea43f8a4f9198172ca9961f343cde663ec98e7061259d6542c35a-locale. 2015-03-25 15:00:53 [Build 8/226] [stage2-gcc] Fetching to local cache: artifact stage2-gcc-libs 2015-03-25 15:00:53 [Build 8/226] [stage2-gcc] Downloading stage2-gcc-libs as a tarball. 2015-03-25 15:00:53 [Build 8/226] [stage2-gcc] Committing stage2-gcc-libs to artifact cache at 14ae4efeeaeea43f8a4f9198172ca9961f343cde663ec98e7061259d6542c35a-libs. Change-Id: I59260c68dcc8ff1fba9d1edd4773f952662757c8
* Revert "SUGGESTED: increment metadata-version for OStree branch"Sam Thursfield2015-03-251-1/+1
| | | | This reverts commit a21def96f77772549777b0bca76ba561baebbe78.
* fixup! SUGGESTED: remove the chunk hardlink cacheSam Thursfield2015-03-251-2/+0
| | | | Change-Id: I46739aca1e71ed5878353d974c46fc07f115b915
* SUGGESTED: remove the chunk hardlink cacheSam Thursfield2015-03-253-75/+7
| | | | | | | The OStree checkout uses hardlinks anyway, so this is thankfully now redundant. Change-Id: Id763b76e49b5900487ee2539e5cde3c3e8d4a958
* SUGGESTED: increment metadata-version for OStree branchSam Thursfield2015-03-241-1/+1
| | | | | | | | We can't easily share built artifacts built with older versions of Morph because they may contain device nodes, and these can't be stored in an OSTree repo. Change-Id: Id1e7be4a5e12e8a4881c2084594d9553df18d026
* cmdtests: We can no longer meaningfully look at chunks in the cacheAdam Coldrick2015-03-2422-176/+4
| | | | | | | | | | | | | Its much harder to look at the contents of the artifact cache now that they aren't stored by chunk name (just cache key and artifact suffix). All of these were failing because they were trying to extract chunk tarballs or locate and inspect artifacts, so just make sure the builds themselves don't fail. The build-system tests are a lot of work to make into yarns, as we'll need to build systems with actual tools in as far as I can tell. Also disable the cross-bootstrap test for now.
* yarns: Disable the cross-bootstrap yarnAdam Coldrick2015-03-241-7/+9
| | | | | | | The cross-bootstrap plugin is currently full of hacks, so it makes sense to rewrite it properly rather than extending these hacks to work with OSTree. I don't have time to do this right now, so disable the cross-bootstrap yarn.
* yarns: Make the distbuild yarn expose the worker's artifact cache over HTTPAdam Coldrick2015-03-243-3/+29
| | | | | | | | | OSTree can easily pull over HTTP. All that is necessary is to expose the repo directory. This commit adds a simple HTTP server to do this in the test suite. Actual implementations should use something better, like lighttpd. Also add some logging of the cache servers in this yarn to help debug.
* morph-cache-server: Add support for an OSTree artifact cacheAdam Coldrick2015-03-241-52/+63
|
* gc: Make `morph gc` use the OSTree artifact cacheAdam Coldrick2015-03-241-2/+6
|
* deploy: Use OSTree to checkout the system for deploymentAdam Coldrick2015-03-241-7/+30
| | | | | | | | | Now that we have an OSTree artifact cache, the deploy plugin needs to use that to get the system to be deployed. Due to the changes in how we store systems, we need to get the contents of each stratum then put the system delta on top of that. This is still much quicker than unpacking stuff from tarballs.
* builder: Use the OSTree artifact cache when buildingAdam Coldrick2015-03-243-45/+49
| | | | | | | The API of the OSTree artifact cache is slightly different to that of the old tarball cache, so adjust things accordingly. Also, only store the files changed at system-construction-time rather than everything in system artifacts.
* Make morph use OSTreeArtifactCache instead of LocalArtifactCacheAdam Coldrick2015-03-241-2/+4
|
* RemoteArtifactCache: Support multiple cache methodsAdam Coldrick2015-03-241-0/+25
| | | | | This commit updates RemoteArtifactCache to enable it to interact with a remote OSTree artifact cache.
* Add an artifact cache which uses OSTreeAdam Coldrick2015-03-244-5/+232
|
* Add a class to wrap the OSTree APIAdam Coldrick2015-03-243-0/+141
|
* bins: We no longer want chunks to be tarballsAdam Coldrick2015-03-242-120/+37
| | | | | | Change create_chunk to put the contents of an artifact into a directory rather than storing them in a tarball, as we want to store chunks as directory trees in OSTree rather than tarballs now.
* Move the chunk cache logic into buildcommandAdam Coldrick2015-03-242-23/+23
| | | | | | This avoids needing to pass the cache to the staging area since lac.get returns a path to a directory tree rather than a file handle now, so we also don't need to do any unpacking.
* Create device nodes in staging areaAdam Coldrick2015-03-243-35/+39
| | | | | | | | | We can't store devices nodes in OSTree, so we can't create them at artifact build time and store them in the chunk artifacts anymore. Instead, we should create them in the staging area when installing an artifact with a source which has a morphology which defines them into the staging area.
* Add support for unionfs-fuseAdam Coldrick2015-03-244-16/+40
| | | | | | Overlayfs is new in version 3.18 of the kernel, so add support for a different implementation of a union/overlay filesystem in order to allow morph to work on older kernels.
* Use overlayfs when deployingAdam Coldrick2015-03-241-4/+34
| | | | | | | | | | | | When deploying, configuration extensions are run against the unpacked tarball of a system created by a build. If we are to use OSTree to store systems (rather than tarballs) this will not be possible as the result of `ostree checkout` would be read-only. To solve this, we use overlayfs to mount the unpacked tarball underneath a temporary directory somewhere, and run the configuration extensions on that mount point. This means that the changes are made in the temporary directory rather than directly on the tarball.
* Use overlayfs when building systemsAdam Coldrick2015-03-243-5/+45
| | | | | This will allow us to cache systems as a list of chunks and a small filesystem delta, rather than a massive tarball.
* Allow the passing of options to fsutils.mountAdam Coldrick2015-03-241-2/+5
| | | | | In order to mount using overlayfs, fsutils.mount needs to take a string of options to pass to the mount command.
* Ignore files in the copyright check scriptAdam Coldrick2015-03-241-0/+4
| | | | | | | Add the ability to ignore files when checking copyright years, and start ignoring the COPYING file. Change-Id: I0cc11456f896e27458357ab0c5729223c71b406e
* Add COPYING with the GPLv2 licenseJavier Jardón2015-03-211-0/+339
| | | | Change-Id: I00bcedae271c20273020fa3a60bb21decde85eed
* Merge "deploy: Tighten SSH connectivity check"Javier Jardón2015-03-191-1/+5
|\
| * deploy: Tighten SSH connectivity checkSam Thursfield2015-03-191-1/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | I accidentally tried to deploy a Baserock upgrade to a Fedora cloud machine. Every SSH command that Morph ran got the following output: Please login as the user "fedora" rather than the user "root". The existing implementation of check_ssh_connectivity() didn't raise an exception, leading to confusing errors further down. The new implementation produces this error: ERROR: Unexpected output from remote machine: Please login as the user "fedora" rather than the user "root". Change-Id: Ida5a82b25d759167aa842194b0d833d0565b4acf
* | Merge "morphlib/sourceresolver.py: parse VERSION file in a function"Javier Jardón2015-03-181-6/+22
|\ \
| * | morphlib/sourceresolver.py: parse VERSION file in a functionJavier Jardón2015-03-171-6/+22
| | | | | | | | | | | | Change-Id: I6e714d1994632875a5a15f840fe8ab3a66dddc77
* | | Merge "morphlib/sourceresolver.py: Fail if morph doesnt support the version ↵Javier Jardón2015-03-181-8/+12
|\ \ \ | |/ / | | | | | | of definitions format"
| * | morphlib/sourceresolver.py: Fail if morph doesnt support the version of ↵baserock-definitions-v1Javier Jardón2015-03-171-8/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | definitions format This patch will add the following restriction: if VERSION exist and its a YAML file and its a dict and has the key 'version' and the type stored in the 'version' key is an int and that int is not in the supported format, then fail. So, if someone is using 'version: 4' in VERSION, morph will fail (as opposed to current morph, that will not fail in the check but will likely fail when it tries to compile) Change-Id: I555f7e6018b9bdf18c80039df92a253acbd51c8c
* | | Merge branch 'baserock/richardipsum/add-hosts-conf-ext-v3'Richard Ipsum2015-03-183-18/+87
|\ \ \ | | | | | | | | | | | | | | | | | | | | Reviewed by: Richard Maw <richard.maw@gmail.com> (+2) Pedro Alvarez <pedro.alvarez@codethink.co.uk> (v2) (+1)
| * | | Add hosts.configureRichard Ipsum2015-03-181-0/+48
| | | | | | | | | | | | | | | | | | | | | | | | This adds a new config extension to allow deployments to write to /etc/hosts by adding HOSTS_x: </etc/hosts line> to a cluster morph in a similar manner to the fstab.configure extension.
| * | | Make fstab.configure use write_from_dictRichard Ipsum2015-03-181-18/+7
| | | |
| * | | Add write_from_dict to utilRichard Ipsum2015-03-181-0/+32
|/ / / | | | | | | | | | A function to read lines from a dictionary and append them to a file
* | | scripts/test-shell.c: Add missing license headerJavier Jardón2015-03-171-0/+15
| | | | | | | | | | | | Change-Id: I46ad9c15064de752e839611d2ce3ee016ec7cc46
* | | Fix line lengths in morphlib/exts/simple-network.configureAdam Coldrick2015-03-171-5/+9
|/ / | | | | | | | | | | | | | | Some lines were more than 79 characters long. This was causing ./check --full to fail on master of morph. This commit fixes the lines in question. Change-Id: I80969d2d89d3922e021b716c250796188d7a7f4c
* | Merge "Use python3 compatible notation for catching exceptions"Javier Jardón2015-03-1721-40/+40
|\ \
| * | Use python3 compatible notation for catching exceptionsJavier Jardón2015-03-1621-40/+40
| | | | | | | | | | | | Change-Id: Ibda7a938cd16e35517a531140f39ef4664d85c72
* | | Merge branch 'jjardon/simple-network-networkd2'Javier Jardón2015-03-161-11/+131
|\ \ \ | |/ / |/| | | | | | | | Reviewed-By: Richard Maw <richard.maw@codethink.co.uk> Reviewed-By: Sam Thursfield <sam.thursfield@codethink.co.uk>
| * | simple-network.configure: Rename networkd file generated by systemd chunkjjardon/simple-network-networkd2Javier Jardón2015-03-161-0/+29
| | | | | | | | | | | | | | | Rename instead removal in case the user is already using a 10-dhcp.network file
| * | simple-network.configure: Update documentationJavier Jardón2015-03-161-6/+9
| | |
| * | simple-network.configure: Generate default network config files in a functionJavier Jardón2015-03-161-6/+20
| | | | | | | | | | | | | | | | | | Use DHCP by defaul in the default interfaces: - for /etc/networ/interfaces: "lo:loopback;eth0:dhcp,hostname=$(hostname)" - for networkd: "e*:dhcp"
| * | simple-network.configure: process pairs of parameters in a functionJavier Jardón2015-03-161-0/+29
| | |
| * | simple-network.configure: Add function to convert mask to cidr suffixJavier Jardón2015-03-161-0/+7
| | | | | | | | | | | | 255.255.255.0 -> 24
View Lorry Controller Status