summaryrefslogtreecommitdiff
path: root/morphlib
Commit message (Collapse)AuthorAgeFilesLines
* tests: fix up for change in artifact nameRichard Maw2012-07-032-6/+30
| | | | | This fixes references to a system artifact without the -rootfs appended and adds a unit test for resolving an arm system artifact.
* ArtifactResolver: systems define multiple artifactsRichard Maw2012-07-031-16/+22
| | | | | Instead of a System source producing one System artifact, it can produce multiple System artifacts, dependent on if it is an ARM System.
* Merge branch 'master' of roadtrain.codethink.co.uk:baserock/morphLars Wirzenius2012-07-031-5/+3
|\
| * morph branching: use git-config to set pushInsteadOfRichard Maw2012-07-031-5/+3
| | | | | | | | | | | | | | This is the standard way of altering config, it shouldn't be done by modifying the config file. This may allow better forward compatibility if the config keys or storage format change.
* | Merge remote branch 'origin/baserock/bugfix/S3215-bootstrap'Lars Wirzenius2012-07-031-2/+16
|\ \ | |/ |/|
| * BuildCommand: make get_recursive_deps iterativeRichard Maw2012-07-031-7/+6
| | | | | | | | This is less code and can't blow the stack.
| * build: install recursive deps in staging areaRichard Maw2012-07-031-2/+14
| | | | | | | | | | | | All of the dependencies need to be installed, so generate the transitive closiure to pass to cache_artifacts_locally and install_chunk_artifacts.
| * build: re-implement bootstrap build logic.Richard Maw2012-07-021-0/+3
| |
* | Merge remote branch 'origin/baserock/bugfix/S2528-remount-rw'Lars Wirzenius2012-07-022-2/+4
|\ \ | |/ |/|
| * CacheKeyComputer: increment compat-versionRichard Maw2012-07-021-0/+2
| | | | | | | | | | | | | | | | | | This will prevent systems that don't have the right mount options being cache hits. Having fstab be in some System configuration morphology, which is also included in the cache key would prevent this being needed if the fstab format changes again.
| * SystemBuilder: fix mounting optionsRichard Maw2012-06-251-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | fstab had errors=remount-ro, this is an ext option, btrfs doesn't support it. However it should be mounted noatime, to prevent the metadata trees being duplicated because they were read. extlinux.conf also has its mounting options changed so that the rootfs is mounted read-only initially. This should prevent the metadata trees being updated by reading them until it is remounted with noatime.
* | Normalize timestamps without filesystem operationsLars Wirzenius2012-06-291-4/+9
| | | | | | | | | | | | | | This changes the timestamp in the Python tarfile.TarInfo data structure rather than setting the timestamp in the filesystem. Suggested by Richard Maw.
* | Improve error message for missing build-depsLars Wirzenius2012-06-291-2/+4
| | | | | | | | Suggested by Richard Maw.
* | Normalize timestamps of files in chunk artifactsLars Wirzenius2012-06-282-1/+10
| | | | | | | | | | | | | | This avoids problems with clock skew between the machine that built an artifact and the machine the uses it. I ran into this problem during a test build of other changes in this patch series. We have seen it before, now it is fixed.
* | Add check for missing build depends in strataLars Wirzenius2012-06-281-0/+11
| | | | | | | | | | | | | | This will make a build fail if the morphology is missing explicit build dependencies. Also fix test causes so that ./check passes.
* | Improve status messages for clarityLars Wirzenius2012-06-271-13/+29
| | | | | | | | | | | | | | Suggested by Richard Maw. This commit also includes a bug fix, since I'm too lazy to detangle changes in git commits today.
* | Remove unnecessary variableLars Wirzenius2012-06-271-1/+1
| | | | | | | | Suggested by Richard Maw.
* | Rename build_order method to build_in_orderLars Wirzenius2012-06-271-3/+3
| | | | | | | | Suggested by Richard Maw.
* | Create class for build logic and fix the logicLars Wirzenius2012-06-271-206/+280
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The build logic was spread all across the Morph class. This patch gathers it into one class. Having it in one place is useful for general code quality reasons, but also makes it feasible to experiment with the build logic, and override only parts of it. This patch also changes how the build logic works. Earlier we kept one staging area for staging builds, and were unpacking everything in a build group at a time. This meant that the staging area would contain things that were not strictly build dependencies. This has a possiblity of changing what gets built. The new logic creates a new, fresh, clean staging area for each build. This makes building a bit slower, but we'll fix that by making it really fast to create a staging area and filling it with build dependencies.
* | Fix test suite to deal with morph output changesLars Wirzenius2012-06-261-7/+9
| | | | | | | | | | | | | | | | Also, some bug fixes. Also, when handling a BaseException, log the exception (with traceback) that we're handling, in case there is an error while handling it, because the second error will otherwise mask the first one.
* | Fix problems found by Richard in reviewLars Wirzenius2012-06-262-14/+9
| |
* | Be more usefully verbose for chunk buildingLars Wirzenius2012-06-251-2/+6
| |
* | Add status message for unpacking chunksLars Wirzenius2012-06-252-4/+5
| |
* | Replace Morph.msg with Morph.statusLars Wirzenius2012-06-252-55/+134
| | | | | | | | | | | | | | | | | | The new method takes a list of keyword arguments. This is more useful than the old way of giving just a string, since now the presentation layer may transmogrify the status update. For example, it can usefully translate the message to another language. Add --verbose option to allow more control over what the user sees.
* | Remove optional msg argument from Submodule classLars Wirzenius2012-06-251-3/+2
| | | | | | | | | | Nobody was using this, so it is obviously useless. Instead, log at the warning level.
* | Merge remote branch ↵Lars Wirzenius2012-06-252-9/+10
|\ \ | |/ |/| | | 'remotes/origin/baserock/feature/S2938-system-artifact-splitting'
| * make-patch: fix artifact name to include suffixRichard Maw2012-06-211-1/+4
| |
| * SystemBuilder: create a new artifact for kernelRichard Maw2012-06-211-8/+6
| | | | | | | | | | | | | | The kernel should be considered a full artifact, not just metadata. The rootfs is now suffixed by -rootfs and the kernel is named $name-kernel. This is similar to how chunk splitting works.
* | Use shutil.copyfileobj to avoid excessive memory useLars Wirzenius2012-06-221-1/+1
| | | | | | | | | | | | Thanks to Richard Maw for pointing this out! I need to get out of my ancient habits and use the modern Python stdlib and rember that files in these degenerate modern times can be larger than 64 KiB.
* | Handle only the urllib2.URLError exceptionsLars Wirzenius2012-06-222-4/+9
| | | | | | | | | | | | | | | | Handling all exceptions in a way that assumes they're all related to the URL fetching hides unrelated errors, such as missing imports, or bad types, or such. Also, add more logging for debugging.
* | Add logging, for debuggingLars Wirzenius2012-06-221-0/+1
| |
* | Prettify str(RemoteArtifactCache) to be the server URLLars Wirzenius2012-06-221-0/+3
| | | | | | | | | | This makes error messages that print the artifact cache object be useful to the reader.
* | Include cache id in error messageLars Wirzenius2012-06-221-2/+3
| | | | | | | | For usefulness when tracing why artifact isn't being found.
* | Change install_artifacts to fetch missing artifactsLars Wirzenius2012-06-221-3/+9
|/ | | | | | | | | | | | This is a behavioral change, and is useful when building and having access to an artifact cache server. Previously, nothing was actually fetching artifacts from the remote server when the staging area was being crafted. Morph could do with a bit of cleanup where we separate more cleanly the acts of fetching stuff to the local artifact cache from the remote one, when possible, and then using artifacts from the local cache. Currently this stuff is spread around a bit too much. I did not do that in this patch, however, to keep undiscussed changes small.
* ChunkBuilder: mount /dev/shm as wellRichard Maw2012-06-201-19/+24
| | | | | | This required refactoring mount_proc and unmount_proc into more generic mounting functions. do_mounts returns the paths that were mounted instead of just one
* app: rename setup_proc to setup_mountsRichard Maw2012-06-202-13/+13
| | | | | This change is purely cosmetic, but reflects the fact that we need to mount more than just proc sometimes.
* Remove dead code (dependencies in Source objects)Lars Wirzenius2012-06-182-34/+0
|
* Return list of built artifacts from builderLars Wirzenius2012-06-151-2/+9
|
* Merge branch 'richardmaw/strata-chunk-lists'Richard Maw2012-06-147-88/+190
|\
| * fix tabs and line widths so ./check passesRichard Maw2012-06-141-11/+11
| |
| * morphlib.builder2: check in local cache firstRichard Maw2012-06-141-2/+2
| | | | | | | | | | | | It's possible to work without a remote artifact cache, sadly there is not an explicit check for a None remote artifact cache to prevent it downloading dependencies.
| * morphlib.builder2: write stratum metadataRichard Maw2012-06-142-9/+45
| | | | | | | | | | | | Stratum metadata was getting lost because it is not part of the chunks. Now metadata is written to the cache, then when the system builder assembles the strata it copies the metadata in.
| * morphlib.builder2: fix unit testsRichard Maw2012-06-142-5/+23
| |
| * morphlib.builder2: write overlaps to cacheRichard Maw2012-06-141-15/+38
| | | | | | | | Rather than just log the overlaps, write them to the cache as well
| * morphlib.bins: remove create_stratumRichard Maw2012-06-142-35/+0
| | | | | | | | create_stratum is no longer used, since strata are now chunk lists
| * CacheKeyComputer: add stratum-format-versionRichard Maw2012-06-141-0/+2
| | | | | | | | | | | | Since strata aren't tarballs any more, it's simpler to make tarball strata cache misses than have the system build code behave differently for tarballs and json chunk lists.
| * builder2: write json to cache instead of tarballRichard Maw2012-06-141-33/+53
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Note that this has no way to deal with an old cached stratum being a tarball yet. Strata are now json files listing basenames, which get converted into ArtifactCacheReferences. SystemBuilders now have to ensure they download the chunks as well, since having the strata doesn't imply that the chunks are also available. Overlap checking is slightly more complicated by strata not being tarballs any more, but this is dealt with by looking at the chunks' tarballs.
| * ArtifactCacheReference: remove from_artifactRichard Maw2012-06-141-4/+0
| | | | | | | | | | | | | | | | This convenience method is pointless, since it's trivial to make an ArtifactCacheReference from an Artifact with ref = ArtifactCacheReference(artifact.basename()) and it's actually shorter than ref = ArtifactCacheReference.from_artifact(artifact)
| * morphlib.artifactcachereference: store basenameRichard Maw2012-06-141-15/+13
| | | | | | | | | | | | | | | | This is not to make it easier, but to make it not miss if the algorithm for generating cache names changes. Strictly speaking it should store the format string for accessing metadata as well to be compatible, but missing an artifact's metadata is less important.
| * morphlib: add artifact cache reference classRichard Maw2012-06-132-0/+44
| | | | | | | | | | | | | | The artifact cache doesn't need to know the whole of an artifact object to be able to retrieve something from the cache, just the basename. This can be generated from the contents, or just saved itself.
View Lorry Controller Status