summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* Don't mount /dev/shm in builds.baserock/richardmaw/parallelism-improvementsRichard Maw2014-10-225-42/+12
| | | | | | | This was the final mount command that was causing problems. Now that it's gone, we don't need all the infrastructure that had grown up to do the mounts before running commands independently of what linux-user-chroot already does.
* stagingarea: Mount proc and ccache inside the namespaceRichard Maw2014-10-152-10/+18
| | | | | | | | | | | This works towards allowing multiple concurrent builds in the same system, which has the same problem as deployments. This is the easy bit, since linux-user-chroot has support for bind mounts and /proc mounts. We also need to get rid of the /dev/shm mount to be able to build in parallel though.
* deploy extensions: Don't crash if someone builds at the same timeRichard Maw2014-10-151-1/+3
| | | | | | | | | | | | | | If a build happens, it creates a new network namespace, and if this happens while you have a disk image mounted, then you can't remove the mount-point, because the other namespace is using it. We can avoid the other namespace keeping this mount-point open by creating the disk image in a private mount namespace, so it never sees it. The nicest way to do this is to have every extension run in a private mount namespace, since you'd have to have extensions re-exec themselves, since the appropriate system calls aren't exposed very well.
* Merge branch 'baserock/richardipsum/arfix'Richard Maw2014-10-131-74/+51
|\ | | | | | | | | | | Reviewed-by: Richard Maw Reviewed-by: Richard Ipsum Reviewed-by: Sam Thursfield
| * Simplify _resolve_system_artifactsRichard Ipsum2014-10-101-12/+4
| |
| * Replace resolver queue with loopRichard Ipsum2014-10-101-62/+57
| | | | | | | | | | | | Since we're no longer adding anything to the queue we can replace it with a loop. This patch also splits the code up a little bit.
| * Remove queueingRichard Maw2014-10-101-16/+6
| | | | | | | | | | | | | | Artifact resolution is no longer recursive, so there's no need to requeue sources, removing this queue provides a significant speed up. Fixing an issue where artifact resoltuion was taking too long on some systems (several minutes in some cases).
* | Remove out of date commentRichard Ipsum2014-10-101-3/+0
| |
* | Merge branch 'sam/fix-list-artifacts'Sam Thursfield2014-10-101-6/+7
|\ \ | | | | | | | | | | | | | | | Reviewed-By: Adam Coldrick <adam.coldrick@codethink.co.uk> Reviewed-By: Richard Ipsum <richard.ipsum@codethink.co.uk> Reviewed-By: Richard Maw <richard.maw@codethink.co.uk>
| * | Fix the `morph list-artifacts` commandSam Thursfield2014-10-101-6/+7
| |/ | | | | | | It was broken by the per-source building changes.
* | Merge branch 'baserock/richardmaw/fix-ssh-rsync-destroying-versions'Richard Maw2014-10-101-77/+105
|\ \ | |/ |/| | | | | | | Reviewed-by: Sam Thursfield Reviewed-by: Jim MacArthur Reviewed-by: Richard Ipsum
| * ssh-rsync: Don't delete version if it existsRichard Maw2014-10-101-77/+105
|/ | | | | | | | | | | | | | | | A quirk in the resource cleanup code meant that if you gave the same version label when deploying a new version, then it would fail, then remove the old version, as it had assumed that it was the one to create those directories. This patch fixes this issue by making short context managers for all the resource allocation, so cleanup is done by walking up the context managers, so only the mount and the temporary directory need to be cleaned up if the `mkdir "$VERSION_ROOT"` fails. I've tested this with a deploy of a version that doesn't already exist, and the version I'm currently running, so I can conclusively say it's fixed that problem.
* Merge remote-tracking branch 'origin/baserock/ps/log-chunk-build-times'Richard Maw2014-10-091-0/+8
|\ | | | | | | | | Reviewed-by: Pedro Alvarez Reviewed-by: Richard Maw
| * Log elapsed time for each chunk buildbaserock/ps/log-chunk-build-timesPaul Sherwood2014-09-271-0/+7
| |
* | Update artifact resolver and ckc testsRichard Ipsum2014-10-092-9/+9
| | | | | | | | | | | | The original resolve_artifacts method is essentially unchanged, it's just private now. This means that aside from this name change the tests remain the same.
* | Tidy up artifact resolvingRichard Ipsum2014-10-092-34/+24
| | | | | | | | | | | | | | | | This patch started off as an attempt to address the comment in find_root_artifacts, though this patch doesn't impose an ordering on the list of artifacts as the comment suggested. The artifact resolver now returns a list of root artifacts which may make it easier to add multi-system builds to morph.
* | Merge branch 'sam/handle-missing-version-label'Sam Thursfield2014-10-082-2/+7
|\ \ | | | | | | | | | | | | Reviewed-By: Pedro Alvarez <pedro.alvarez@codethink.co.uk> Reviewed-By: Francisco Redondo Marchena <francisco.marchena@codethink.co.uk>
| * | deploy: Make ssh-rsync upgrade extension handle unset VERSION_LABELSam Thursfield2014-10-082-2/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | It now gives an error message. Previously it would fail with a backtrace like this: 2014-10-08 09:51:37 [systems/genivi-baseline-system-armv7lhf-jetson.morph][self]Removing temporary mounts Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 190, in _run self.process_args(args) File "/src/morph/morphlib/exts/ssh-rsync.write", line 54, in process_args self.upgrade_remote_system(location, temp_root) File "/src/morph/morphlib/exts/ssh-rsync.write", line 107, in upgrade_remote_system location, ['btrfs', 'subvolume', 'delete', orig_dir]) UnboundLocalError: local variable 'orig_dir' referenced before assignment
* | | Merge branch 'baserock/richardmaw/fix-distbuild-v3'Richard Maw2014-10-0874-2431/+1436
|\ \ \ | |/ / |/| | | | | | | | | | | Reviewed-by: Sam Thursfield Reviewed-by: Richard Ipsum Reviewed-by: Pedro Alvarez
| * | distbuild: serialize dependent sources of graphRichard Maw2014-10-082-24/+46
| | |
| * | Allow distbuilds to choose where to put logsRichard Maw2014-10-083-1/+14
| | |
| * | Only attempt to run distbuild yarns if possibleRichard Maw2014-10-082-0/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | You need both bottle and flup for this to work, and it would wait forever for a cache-server that is never going to start unless you have bottle and flup. So we now check whether the cache server is functional before attempting distbuild tests.
| * | distbuild: yaml-encode messages before json encodingRichard Maw2014-10-081-2/+8
| | | | | | | | | | | | | | | | | | JSON can only handle unicode strings, but commands can write anything to stdout/stderr, so we do the same trick as for the serialise, and json encode yaml.
| * | Allow distbuilding morphologies with binary data embeddedRichard Maw2014-10-081-9/+10
| | | | | | | | | | | | | | | | | | The horrible json.dumped, yaml dump is because we need it to be both binary safe (which yaml gives us) and one line per message (which json gives us).
| * | Test that morph can handle binary garbage output in buildsRichard Maw2014-10-081-1/+11
| | |
| * | Add echo to the test shellRichard Maw2014-10-081-0/+6
| | |
| * | Fix issues with distbuild caused by moving to building per-sourceRichard Maw2014-10-086-25/+31
| | |
| * | Add distbuilding yarnRichard Maw2014-10-023-7/+192
| | | | | | | | | | | | Deploying after distbuild currently broken
| * | Allow morph-cache-server to be bound to an ephemeral portRichard Maw2014-10-022-5/+31
| | |
| * | Allow ephemeral ports for distbuild servicesRichard Maw2014-10-022-2/+2
| | |
| * | Merge in morph cache serverRichard Maw2014-10-014-6/+527
| |\ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is a runtime dependency of distbuild, so you can't exactly test distbuild without it. The cache server itself depends on bottle and flup, so rather than having a distbuild stratum and a trove stratum both containing the cache server, we should split morph out of the tools stratum and include it in both devel and trove systems.
| | * \ Merge branch 'baserock/richardipsum/post_artifacts5'Richard Ipsum2014-04-151-3/+31
| | |\ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Reviewed by: Lars Wirzenius Daniel Silverstone (on IRC) Sam Thursfield (v4 of patch) Richard Maw (on IRC)
| | | * | Add post request for /artifactsRichard Ipsum2014-04-151-3/+31
| | |/ / | | | | | | | | | | | | | | | | | | | | | | | | With this we can request the state of a set of artifacts in a single request. Artifacts are sent as a json array. We check whether each artifact is in the cache or not and send our findings back to the client as a json object.
| | * | Update copyright noticeRichard Ipsum2013-12-164-24/+3
| | | |
| | * | Merge branch 'baserock/tiagogomes/lighttpd'Tiago Gomes2013-09-121-3/+11
| | |\ \ | | | | | | | | | | | | | | | | | | | | | | | | | Reviewed by: Pedro Alvarez Daniel Silverstone
| | | * | Allow to use flup as the backend server for bottle.Tiago Gomes2013-09-121-3/+11
| | |/ / | | | | | | | | | | | | | | | | | | | | | | | | | | | | lighttpd does not support WSGI, so we need to be able to run the server in FCGI mode. The flup package provides one. Running with the default bottle server is still useful for testing purposes.
| | * | Merge branch 'jonathan/fix-resolve-sha1-rebase'Jonathan Maw2013-06-041-3/+8
| | |\ \ | | | | | | | | | | | | | | | Reviewed-by: Lars Wirzenius <lars.wirzenius@codethink.co.uk
| | | * | Handle requesting a sha1 of a sha1Jonathan Maw2013-06-041-3/+8
| | |/ /
| | * | Merge remote-tracking branch ↵Lars Wirzenius2013-01-141-0/+53
| | |\ \ | | | | | | | | | | | | | | | 'origin/jannispohlmann/add-batch-sha1-and-file-queries'
| | | * | Handle batch sha1/file queries using POST requestsJannis Pohlmann2013-01-021-0/+53
| | |/ / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This commit adds support for resolving multiple refs into SHA1s and requesting multiple files at once, each using a single POST request. The (repo, ref) and (repo, ref, filename) parameters are passed to the POST requests as JSON dictionaries in a list. The response to both types of requests is a JSON list with the same dictionaries again, to which a "sha1" and "data" field are added, respectively. The file contents returned by "POST /1.0/files" requests are base64-encoded and need to be decoded at the receiver's end. This is because the contents may be binary or contain quotes and therefore cause JSON syntax errors.
| | * | Fix misspelled variable nameSam Thursfield2012-12-131-1/+1
| | | |
| | * | Log exceptions instead of ignoring themSam Thursfield2012-12-131-1/+1
| | | |
| | * | Use 'git rev-parse --verify' to resolve refsSam Thursfield2012-12-131-22/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | 'git show-ref' returns multiple results where there are partial matches for the given ref, which creates the possibility that we might resolve a ref incorrectly. 'git rev-list' is also overkill for verifying that a SHA1 is valid.
| | * | Fix missing argument to rev_list(), breaking resolve_ref for SHA1sJannis Pohlmann2012-12-031-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This bug has been present since the initial commit to the cache server. Due to the missing repo_dir argument to rev_list(), resolving SHA1s rather than symbolic refs via /1.0/sha1s fails. This feature, however, is absolutely required for morph to resolve petrified system branches.
| | * | Trim leading slashes from URI element during direct-mode. rs=richardmawDaniel Silverstone2012-09-251-1/+4
| | | |
| | * | Update /fetch API to latest definitionDaniel Silverstone2012-09-171-20/+51
| | | |
| | * | Merge branch 'danielsilverstone/updates'Daniel Silverstone2012-09-113-14/+137
| | |\ \
| | | * | Add facility to delete artifactsDaniel Silverstone2012-09-071-0/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | In order to allow the artifact cache to be cleaned up, this patch allows for a /delete method which can remove artifacts from the cache. It takes the following arguments: artifact=artifactname The artifact will be deleted and a JSON object returned in the form: { "status": errno, "reason": strerror } Where errno is zero on success, 1 on EPERM, 2 on ENOENT etc. and reason is the strerror of the errno, in case the architectures differ between caller and cache.
| | | * | Support for fetching artifacts to the cacheDaniel Silverstone2012-09-071-9/+41
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Rather than pushing artifacts to the cache, this method allows the caller to specify a host and artifact which the cache server will then fetch into its local cache. It takes the following arguments: host=hostname:port artifact=artifactname This is transformed into a fetch to: http://hostname:port/artifacts?basename=artifactname Which is then fetched into the cache under the given name. The return from this is a JSON object of the form: { "filename": artifactname, "size": NBYTES_SIZE_OF_FILE, "used": NBYTES_DISK_SPACE_USED }
| | | * | Add a /list methodDaniel Silverstone2012-09-071-0/+22
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When --enable-writes is set, we provide a /list target which produces a JSON dictionary of information about the state of the artifact cache. The dictionary is of the form: { "freespace": NBYTES_OF_SPACE, "files": { "artifact-filename": { "atime": ATIME_AS_NUMBER, "size": NBYTES_SIZE_OF_FILE, "used": NBYTES_USED_ON_DISK }, ... } } This allows a controller to decide which artifacts have not been requested in some time and also how big artifacts are, not only in terms of their 'byte' size, but also the space they consume on disk. System images in particular may differ in this respect since they should be sparsely stored.
View Lorry Controller Status