summaryrefslogtreecommitdiff
path: root/morphlib
Commit message (Collapse)AuthorAgeFilesLines
* MorphologyLoader: Set filename attribute at parse timeRichard Maw2015-05-111-2/+3
| | | | Change-Id: I0e0b8d352eb4ef1ab6c50e0ba0162263d9bac09d
* morph anchor: Handle updating refsRichard Maw2015-05-111-5/+6
| | | | | | | | | | Previously it would not attempt to make commits it needed locally available if the commit was available on the remote repo cache. Now it will do the update if the commit is not available locally, and will obey --no-git-update. Change-Id: I80f1e351ce334641e2ef733fa4c9a6ab967f9b67
* morphlib.util: add word_join_listRichard Maw2015-05-111-0/+10
| | | | | | | This is useful for representing lists of items in status or exception messages. Change-Id: I530eecdc311ac77fca9922dab063f550ea810c06
* Fix mistake in sysroot.writeSam Thursfield2015-05-081-1/+1
| | | | | | The * should not be in quotes. Change-Id: Ieebdc7532ba1bff5ba9742f72440ed00b0c0de2a
* Raise an error if a stratum build-depends on itselfAdam Coldrick2015-05-082-0/+25
| | | | | | | | | | | | If a stratum build-depends on itself, the build graph calculation gets stuck in an infinite loop as it adds the same stratum to the queue of morphologies to inspect over and over again. This commit causes MorphologyLoader.validate_stratum to raise an error if a stratum contains itself in it's build-depends, as depending on itself makes no sense and will cause the above problem. Change-Id: I76df5b7d63d010ae3b17f72bfa39b273e74279dd
* Fix distbuild-morphologyAdam Coldrick2015-05-071-1/+1
| | | | | | This fixes an error caused by not enough parameters being given to the InitiatorBuildCommand constructor in distbuild-morphology. Change-Id: I133bd2f267fd06cfe88a1cbf4711cc79ad00d209
* Fix sysroot.write trying to overwrite existing filesSam Thursfield2015-05-072-11/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Commit 807e6a90876c5469d242 changed the behaviour of sysroot.write to avoid deleting the contents of the sysroot. This was done so if you accidentally set 'sysroot=/' it wouldn't delete your whole system. It turns out that SDK deployments like clusters/sdk-example-cluster.morph depended on the contents of the directory being deleted. The system armv7lhf-cross-toolchain-system-x86_64.morph has a bunch of files installed by the cross-toolchain in /usr/armv7lhf-baserock-linux-gnueabi/sys-root. Previously sysroot.write would delete these, but since commit 807e6a90876c5469d242 it would fail with several errors like: mv: can't rename '/src/tmp/deployments/usr/armv7l.../sys-root/sbin' If we use 'cp -a' instead of 'mv' then it is slower to deploy, but there are no errors. I am still unsure why files from the cross-toolchain system are installed and then deleted. Although this patch fixes the immediate issue, I don't know if it's the right thing to do. It seems better to not install those files in the first place, if we do not need them. This commit also removes the check for the sysroot target location being empty. This doesn't work, because it runs /before/ the system being deployed is unpacked. Change-Id: I10671c2f3b2060cfb36f880675b83351c6cdd807
* Make str() of a GitDirectory return its location.Sam Thursfield2015-05-071-0/+3
| | | | | | Handy for log messages. Change-Id: I4336866c456a6225a6f3ecbfef10dfc7b864ac59
* Make listing contents of local tarball cache more robustSam Thursfield2015-05-071-1/+12
| | | | | | | | | | | Previously it would crash with a backtrace if there were unexpected filenames in the directory. It's still not amazingly robust, but I don't have time to rewrite the whole thing now. This code seems to have ignored that cachefs.walkfiles() returns filenames with a preceeding '/', which I have fixed now. Change-Id: I98b3094bd6c82b26984513ee81a1eab9bf253a34
* Show progress of downloads when --verbose is passed, not --debugSam Thursfield2015-05-064-4/+5
| | | | | | | This commit undoes behaviour changes from commit aa6dfcbb70c03dfeb3f9af02. Change-Id: Ie677fb9c4e6bcd6edeba2cdd87f4f6125dcae7a4
* GitDir: Fix setting fetch url when push url is already on-diskRichard Maw2015-05-061-2/+2
| | | | | | | | | | | | 290483010cfc7945cd4483fadd1d98c3b83efb3 broke morph checkout, which uses set_fetch_url on a repository that has been cloned, hence has its origin remote url config already on-disk. The fix prevents it changing the push_url when the fetch_url is set, unless it is an unnamed remote, as if the config is on-disk, this already does the right thing. Change-Id: I6204f664407bab3d7f8ecf0fcca72f5015dee55e
* Add `morph anchor` commandRichard Maw2015-05-011-0/+226
| | | | Change-Id: If9d92d7c75b9c4276b69c482c076c6fc1d4ccbbf
* RemoteRefManager: Fail all ref updates when one failsRichard Maw2015-04-291-1/+27
| | | | | | | There's no API to do it in one push yet, but we can send a delete for all the branches that *did* commit. Change-Id: I671e9384b84657a3e9034d62818caa0ac0d8de1e
* GitDir: Set the fetch or push url when the other is setRichard Maw2015-04-291-0/+4
| | | | Change-Id: I500cb81fd0f133bd9f4e76d46bc0ff8a4f57fe50
* CachedRepo: Fix reference to _gitdirRichard Maw2015-04-291-1/+1
| | | | | | | 87f8dbefda89bf6cb9e4b88f23a5317b054da0d4 added a method that used _gitdir, but the patch to change it to gitdir was merged afterwards. Change-Id: Ibd9bff73a0fe69b3c1c2ff6acd02df6cea4a13de
* install-files.configure: make possible to overwrite symlinksJavier Jardón2015-04-301-1/+10
| | | | | | os.symlink will fail if the origin file/link already exist Change-Id: I8175c8dce699e55c3e39e35dfd45c0c19b8bd96d
* Allow the deployment of individual chunks/strata from systemsAdam Coldrick2015-04-302-40/+219
| | | | | | | | | | | | | This commit allows the specification of one or more strata/chunks in a deployment entry in a cluster morphology to deploy instead of the full system if --partial is set. These are listed in a 'partial-deploy-components' field in each deployment definition. The components must be in the system, and this only works for tarball or sysroot deployments. It SHOULD NOT be used when deploying production systems, as it has a number of limitations. Change-Id: I04ac58af57216335d9257f6620d09f18f61ea714
* morphlib: Add command to get build-log for a given chunk and system fileLauren Perry2015-04-301-0/+90
| | | | Change-Id: I09e9b17ef2e0fb94dbf5a96dca91062d64433add
* Add ssh keys conf extRichard Ipsum2015-04-301-0/+25
| | | | Change-Id: I4e7888cbff2e4708154538f8f0a48aeaa1a8a811
* Add a CSV manifest generator.Adam Coldrick2015-04-302-2/+338
| | | | | | | | | | | | | | | | This adds a `morph generate-manifest-csv` command which creates a CSV manifest for a system containing license information, upstream URL and a guess at the version of each component in a system. The existing `morph generate-manifest` is also renamed to `morph generate-manifest-genivi` to improve the distinction between the two commands which generate a manifest. The basis of this commit was done by Michael Drake <michael.drake@codethink.co.uk>. Change-Id: I712de11eaa6e1564c5bae5f5f396e5e77484972f
* Check number of args given to build commandsRichard Ipsum2015-04-301-5/+38
| | | | Change-Id: I501632c4d7acde06e391df3752275a3afc334ba6
* Improve 'Deploy failed as system is not yet build' error.Sam Thursfield2015-04-301-4/+5
| | | | | | | | | | If a user had the wrong setting for artifact-cache-server then they could build a system with 'morph distbuild', then be told it wasn't built by 'morph deploy'. The logs do not show any info about which cache server was used, so it looks like a Morph bug. This change should make such a configuration problem much easier to spot. Change-Id: I932ca3c5caa46990f9bcd77d75b3862f49e0dfb8
* GitDirectory: Add a store_commit methodRichard Maw2015-04-301-4/+17
| | | | | | | commit-tree is usually sufficient, but I have a need to be able to create a commit directly. Change-Id: I80ba63eb9601aa1190554bb07522465ffb2cb5d9
* gitdir.PushFailureError: Show why pushes failedRichard Maw2015-04-301-3/+4
| | | | Change-Id: Ife3a809cbf241d8d5a283d7f010ccb5e4d7ea292
* CachedRepo: Expose the internal gitdir objectRichard Maw2015-04-302-27/+27
| | | | | | It's useful to be able to use this. Change-Id: Ib32d27ddb637f1b8ed683bdd8ec1db108529a163
* morph help: Expand upon the importance of the first trove-idRichard Maw2015-04-301-1/+3
| | | | Change-Id: I048687305054f08b0d8e01daa1366170b3eec323
* Install licensecheck.pl with morphlibAdam Coldrick2015-04-291-0/+604
| | | | Change-Id: If1b156a309236956d7a35136db35a8fd0ab3ad71
* distbuild: Add distbuild status commandLauren Perry2015-04-291-0/+47
| | | | | | | | | Adds a command to get the status of all recently ran distbuilds for a given server (e.g. Running, Finished, Failed, Cancelled), so as to tell if a build running via distbuild-start has finished or otherwise exited without going through the server's log files Change-Id: I5ce9fe54ae7b1bd8fe3e0d629f615042be8827ed
* distbuild: Add distbuild start and cancel functionalityLauren Perry2015-04-293-3/+80
| | | | | | | | | | | Add command for distbuild-start to build_plugin in morphlib, and create a boolean parameter to inform the initiator whether to disconnect the controller and leave the build running remotely. Add distbuild-cancel command to parse currently-running distbuild build-request IDs and cancel the one matching the given argument Change-Id: I458a5767bb768ceb2b4d8876adf1c86075d452bd
* Bump supported definitions format version to 3baserock-definitions-v3Javier Jardón2015-04-291-1/+1
| | | | | | | | | | | | Since commit 154a760fb884cee14c2604b8bfbe52b0e7c0d4b1 morph suports a new architecture (armv5) This is effectivly a change in the definitions format, as old morph doesnt recognizse this architecture and will fail if a system with this architecture is added to definitions Change-Id: I2adc3a4b103b35c8ca5ec6d4034b797fd10f2485
* Fix lines bigger than 79 characters in install-essential-files.configure.helpFrancisco Redondo Marchena2015-04-281-2/+2
| | | | | | This fixes morph check. Change-Id: I5f9c8f5e73c1976b623f9a75f97b5e9aa1887c44
* morphlib/builder.py: Add comment, os-release is now generated as an extensionJavier Jardón2015-04-281-0/+2
| | | | | | | | Do not remove this code for now as its possible that new versions of morph build old sytems without the install-essential-files extension Change-Id: I5e49d0bc6baee73bedf16469d922822e76286bc1
* Add install-essential-files configuration extensionJavier Jardón2015-04-282-0/+62
| | | | | | | This is meant to add essential system files like /etc/profile, /etc/os-release ... Change-Id: I3d67b3a452b32205c5d3c7303d128bda80ce75de
* Add version guessing function to gitdir and cached repo.Michael Drake2015-04-282-0/+15
| | | | Change-Id: I1c79ce68a7a7534d36a9e83210e18a58e7b648e8
* get-repo: Delete partially cloned repo on errorsSam Thursfield2015-04-281-2/+9
| | | | | | | | | If a `morph get-repo` command hits an error or is interrupted while the 'git clone' operation is in progress, the partial clone is left around. When the user tries to run the command again, they hit an error saying the directory already exists. Change-Id: Id4bb57f2136049ede0ea900e7e359255617427ec
* Add armv5l to list of valid architecturesJavier Jardón2015-04-284-1/+11
| | | | | | | | Also add support to allow building compatible architectures on armv7 machines, as per the rationale in the comment in _validate_architecture(). Change-Id: Ie0fe4002523f1f92f576bac8b654d4ea6fad5cf2
* Add a `morph get-repo` command to clone repositories from the cacheAdam Coldrick2015-04-271-0/+149
| | | | Change-Id: I0c175649bd98067ea69d471ebe6880aac5a25598
* Fix regression that caused original_ref to be lost in some buildsSam Thursfield2015-04-271-2/+3
| | | | | | | | | | Commit bd788c7219d8b ("Implement partial builds") caused the original_ref metadata to be lost in some cases. This meant that the metadata for built systems would have original_ref set to a SHA1 hex string, instead of the original named ref that the system was built from. Change-Id: Id31654ad60b411206ccc7ee70b46f005baa209ef
* morphlib: Add example use to morph certify and fix spelling errorLauren Perry2015-04-271-1/+6
| | | | Change-Id: I3c4b4f687703ab665841a3527044d0f669135513
* Rename help file to work with 'help-extensions' subcommandPedro Alvarez2015-04-241-0/+0
| | | | Change-Id: Ibf7cf1f81998678f9354d77f52e54344294e89f7
* Calculate the build graph in worker-buildAdam Coldrick2015-04-241-5/+25
| | | | | | | Calculate the build graph and find the artifact which is referred to by the ArtifactReference which was deserialised. Change-Id: Icbef90932979fc1a18daf0dedac16f4429f56349
* Use --verbose/-v to show build output, --debug/-d for morph debugJavier Jardón2015-04-225-11/+15
| | | | | | | | | | | Its going to be more likely that a baserock user is more interesed in the build log than actually the debug output from morph. And also the intuitive option would be to use -v for this. --verbose/-v: show build output --debug/-d: show morph debug output Change-Id: I1fb99034dc8680a5f168f6306724663aea33ebc5
* distbuild: Don't check if there is free disk space on the initiatorSam Thursfield2015-04-221-6/+7
| | | | | | | This check is intended for local builds, not distbuilds. Much less disk space is needed to distbuild. Change-Id: Iaff77805119129cbf6584de418ef97c7f1b82ad8
* Add distbuild-trove-nfsboot.writeSam Thursfield2015-04-213-0/+482
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The nfsboot.write deployment extension has been deprecated for a while because it's not generally useful. It's only used for deploying distbuild nodes to a Trove, as far as I know. We still need to support setting up a bunch of machines that boot over NFS from a Trove. But we can do this in a special-purpose .write extension. The new distbuild-trove-nfsboot.write is much more efficient than the more generic nfsboot.write: instead of treating each system individually (thus copying an almost identical ~2GB rootfs to the Trove once per node) it copies the system image to the Trove once, and /then/ sets up a rootfs per node. Upgrades are now supported, although the code assumes distbuild nodes are stateless (as they should be) so nothing special is done for upgrades, other than checking that there is already a version of the given system in existance. The new extension does not create an orig/ and run/ version of each system, because there is no need when the deployed system is stateless. There could be further gains in efficiency, but I don't have time to do them right now. This write extension is full of compromises, its goal is to better support the existing users who have a Trove and a distbuild network deployed via NFS. It is specifically not intended to be useful for other purposes. Change-Id: I9a50c58b714ed272212d1d6c55b289aaa96051b1
* deploy: Clean up `morph deploy` documentation a bitSam Thursfield2015-04-211-133/+30
| | | | | | | | | | | | Remove documentation specific to certain .configure and .write extensions, and point the user to `morph help-extensions` and `morph help xxx.write` instead. Don't mention the deprecated nfsboot.write extension. Other small cleanups. Change-Id: Ie7ed030c71ade4876aafb3d8a43e864a28ddc279
* deploy: Deploy and upgrade systems from the same 'cluster' definitionSam Thursfield2015-04-211-17/+83
| | | | | | | | | | | | | | | | | | | This patch adds two fields to deployment (cluster) .morph files: 'upgrade-type' and 'upgrade-location'. The `morph deploy` command ignores these. The `morph upgrade` command will honour them if present, instead of the existing 'type' and 'location' fields. If they are not present, `morph upgrade` will give a warning, and will use the 'type' and 'location' fields as before. This avoids the need to edit the deployment .morph file after deploying a system. Small detail: the 'type' and 'location' variables are no longer removed from the environment that is passed to the .configure and .write extensions. This shouldn't affect anything. Change-Id: Id2a4e4f229b8adebdb57eded2049ac113a82a4be
* deploy: Factor out a bit of code into its own functionSam Thursfield2015-04-211-10/+37
| | | | | | | This behaviour is complex, it shouldn't be hidden away in a larger function. Change-Id: I953c9477e0210d395b97f5f8219eaebe4dbcd272
* distbuild: Add distbuild-list-jobs functionLauren Perry2015-04-171-0/+42
| | | | | | | | | | Add InitiatorListJobs class and list-jobs message template, add distbuild-list-jobs to morph commandlist, send running job information back to initiator, split out handling of build request and list-jobs messages to separate functions and change generating a random integer to UUID for message identification Change-Id: Id02604f2c1201dbc10f6bbd7f501b8ce1ce0deae
* Add dns option to simple-network conf extRichard Ipsum2015-04-151-1/+8
| | | | Change-Id: I0f4490d76caca802536b21085ea0d770fb8c0798
* Give more helpful error when stratum artifacts in cache are corruptedSam Thursfield2015-04-141-10/+24
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | I had a stratum artifact in my artifact cache which for some reason was 0 bytes long. When building a system that included this stratum, `morph build` gave me the following output 2015-04-13 13:48:57 ERROR Traceback (most recent call last): File "/src/morph/morphlib/builder.py", line 539, in build_and_cache self.unpack_strata(fs_root) File "/src/morph/morphlib/builder.py", line 600, in unpack_strata chunks = [ArtifactCacheReference(c) for c in json.load(f)] File "/usr/lib/python2.7/json/__init__.py", line 290, in load **kw) File "/usr/lib/python2.7/json/__init__.py", line 338, in loads return _default_decoder.decode(s) File "/usr/lib/python2.7/json/decoder.py", line 366, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode raise ValueError("No JSON object could be decoded") ValueError: No JSON object could be decoded With this patch, I get a better error: ERROR: Corruption detected: No JSON object could be decoded while loading /src/build/cache/artifacts/8b4422c58ecb2a085b142fbba74b760f501f65d4b2885bf707994973230e0c58.stratum.build-essential-minimal Change-Id: I0ad359901c5da75bd26d5a1a8108ef4e6f1d7708
View Lorry Controller Status