summaryrefslogtreecommitdiff
path: root/morphlib/builder2.py
Commit message (Collapse)AuthorAgeFilesLines
* morphloader: Get commands when loading morphologyAdam Coldrick2014-08-141-1/+1
| | | | | | Rather than having a `get_commands` method to obtain missing commands from the build system when they are needed, get the commands when loading a morphology.
* Fix not including symlinks in chunks if they exist on the hostbaserock/richardmaw/bugfix/symlink-includeRichard Maw2014-07-291-1/+1
| | | | | | | | | | | | | | | | | With os.walk, if the target of the link doesn't exist, or it is a link to a file, it ends up in the basenames list. If it is a link to a directory, it goes in the subdirs list. There's a bug in the subdirsymlinks check, in that it checks if the wrong file is a symlink, so it never returns them. This was missed, since we did not have the cross bootstrap in CI. This is not eligible for our yarn tests, since to trigger this would require changes to the host system's rootfs, so it's a system-level test. To test this properly requires putting the cross bootstrap in CI.
* Make our use of json binary path safebaserock/richardmaw/bugfix/unicode-safe-jsonRichard Maw2014-07-111-8/+13
| | | | | | | | | | | | | | | | | | | | | json only accepts unicode. Various APIs such as file paths and environment variables allow binary data, so we need to support this properly. This patch changes every[1] use of json.load or json.dump to escape non-unicode data strings. This appears exactly as it used to if the input was valid unicode, if it isn't it will insert \xabcd escapes in the place of non-unicode data. When loading back in, if json.load is told to unescape it with `encoding='unicode-escape'` then it will convert it back correctly. This change was primarily to support file paths that weren't valid unicode, where this would choke and die. Now it works, but any tools that parsed the metadata need to unescape the paths. [1]: The interface to the remote repo cache uses json data, but I haven't changes its json.load calls to unescape the data, since the repo caches haven't been made to escape the data.
* Fix: use end of options option with run-partsRichard Ipsum2014-06-181-1/+6
| | | | | | We want to supply the end of options option before the SYSTEM_INTEGRATION_PATH so that the system integration path doesn't get interpreted as an option if it happens to begin with a -
* Use logfile kwarg to generate build logRichard Ipsum2014-05-141-24/+47
| | | | | | | | | We could just set stdout to subprocess.PIPE then read from the pipe, but then we won't get the output till the command's finished and some commands take a long time. Using the logfile kwarg a file will be created by tee and the output will be written to it in 'real time'.
* Run the system integration commands when buildingPedro Alvarez2014-03-201-0/+46
|
* Generate scripts with the system-integration-commands of chunk morphologies.Pedro Alvarez2014-03-201-3/+42
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The scripts will be created into the folder: /baserock/sytem-integration/ The system integration commands have the following syntax: name: linux kind: chunk ... install-commands: ... system-integration: linux-libs: 00-depmod: - depmod -a 70-more-integration: - touch /baserock/FILE - | for FOLDER in $(ls /) do echo "$FOLDER" done linux-misc: 70-more-integration: - echo "Hello world" In this concrete example, the following files will be created: $DESTDIR/baserock/system-integration/ 00-depmod-linux-libs-0000: #!/bin/sh set -xev depmod -a 70-more-integration-linux-libs-0000 #!/bin/sh set -xev touch /baserock/FILE 70-more-integration-linux-libs-0001 #!/bin/sh set -xev for FOLDER in $(ls /) do echo "$FOLDER" done 70-more-integration-linux-misc-0000 #!/bin/sh set -xev echo "Hello world"
* Don't create a blank /etc/fstabSam Thursfield2014-03-041-25/+0
| | | | | | | | This messes up the baserock-system-config-sync tool. Systemd does not require /etc/fstab to exist in any case. I have bumped the 'system-compatibility-version' field in this commit to trigger rebuilding all system artifacts.
* Merge artifact splitting workRichard Maw2014-01-171-33/+51
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Rationale ========= This patch series implements the concept of stratum splitting. For a long time we've had code to split a chunk into multiple artifacts, however there's not been a way to split strata up, or systems select a subset of the produced stratum artifacts to be included in the system. This patch series implements the ability to split strata and have systems include them in a way which still has the same behaviour if no rules are specified, but with default rules that split chunk artifacts up into various components, strata into runtime and development versions and has systems include everything by default, but can be told to include less. The default rules have chunk foo split up into -bins, -libs, -devel, -doc, -locale and -misc. These rules can be overridden in the chunk morphology by adding the new 'products' field, which lists match rules like the following: products: - artifact: libudev include: - (usr/)?lib(32|64)?/lubg?udev\..* - artifact: udev include: - (usr/)?s?bin/udev* - (usr/)?lib(32|64|exec)?/systemd/systemd-udevd Strata are by default split into -runtime and -devel. -devel by default contains chunks ending with -devel and -doc, -runtime contains everything else. Extra match rules can be added to a stratum similarly to chunks, but instead of matching file names, they match artifact names. products: - artifact: core-python include: - "cpython-.*" # lazy shortcut to put all of cpython in this stratum - "python-.*" # lazy shortcut to include all python chunks in Additionally, in chunk specs, chunk artifacts may be assigned to stratum artifacts, this takes precedence over products match rules in the stratum and the default match rules. Assigning the chunk to `null` will discard the chunk. chunks: ... - name: systemd ... artifacts: libudev: foundation-runtime udev: foundation-runtime systemd-doc: null By default a system includes every produced artifact of every stratum listed. Instead a subset can be specified in the stratum spec as follows: name: tiny-system strata: - name: build-essential ... artifacts: - build-essential-runtime
| * Use given artifact when constructing StrataRichard Maw2014-01-161-8/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | For some reason we used to create a new Artifact object with the same name as the Stratum morphology for our cache key. This is non-sensical, since we already have an Artifact object and it breaks splitting strata. NOTE: cmdtest tests do not pass, since they list files and artifacts produced, which has changed since the new default splitting rules were added. The next patch fixes this, but was kept as a separate commit for readability.
| * Split chunk morphologies according to new rulesRichard Maw2014-01-161-23/+41
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Filenames are now matched before chunks are constructed, so bins.create_chunk now takes a list of relative file names. bins.chunk_contents is gone, since this is now handled by passing source.split_rules.partition the file names. We now don't consider it to be a problem for directories to remain in the DESTDIR after artifacts have been removed, since we need to handle file matches implying their parent directories, and explicit matches against directories. NOTE: The bins_tests were broken in this patch, and are fixed in the next. This was done to try and aid readability of the patch series. Full functionality is still broken until stratum splitting is fixed.
| * Replace chunk 'chunks' field with 'products'Richard Maw2014-01-151-2/+2
|/ | | | | | | | | | | | | | | | | | | I think that it's confusing for both strata and chunk morphologies to have a 'chunks' field, with the former listing sources and the latter listing rules for splitting this source into artifacts. The design for splitting strata has roughly the same idea, but operating on chunk artifact names, rather than file names, so a name that can be used for both was chosen. Splits and artifacts weren't satisfactory names, so they're now called 'products'. It was decided to break backwards compatibility of chunk morphologies being able to specify 'chunks', since the format has changed, so extra code would be required to translate the format, and the only users of the 'chunks' field was the test suite, since there was no way to select from the system, which chunk artifacts were included.
* Added chunkname prefix to some elements of the debug log.Dan Firth2013-10-111-5/+9
|
* morphologyfactory: move empty stratum check hereRichard Maw2013-08-071-9/+0
| | | | | | | | | | | | | | | It is better to test whether a straum is empty here, since it will be noticed earlier in the build, as soon as the morphologies are loaded, rather than after they have all been parsed. It is also conceptually nicer to put it here, since the morphologyfactory was written to perform this kind of validation. On a more practical note, the validation is moved here so that the test for this error isn't masked by the test for no build dependencies. To ensure tests still pass, we alter the stratum morphology used by other unit tests to no longer be empty, and add an empty one to test.
* stagingarea: move to failed on construction failRichard Maw2013-08-061-4/+1
| | | | | | | | | This uses the same logic as when a build fails, so it's been consolidated into `StagingArea.abort()`. You could argue that if a build fails before any commands are run, then there's nothing interesting to see, but it will be useful if the hardlink/tarball extract algorithm fails in some corner case.
* Add morph cross-bootstrapSam Thursfield2013-07-301-63/+54
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Cross-bootstrap is a way to build baserock on an architecture that does not currently have Baserock. It can be used by `morph cross-bootstrap <ARCH> <REPO> <REF> <MORPH>`, and will build an artifact that can be used as a root filesystem with a basic build environment with a script named `native-bootstrap` which will build and install every chunk in the system. If done with a devel system, this will give you a suitable environment for building a proper Baserock system. This does not currently provide a kernel for the target architecture. Apart from adding the cross-bootstrap plugin, it also makes the following changes: * Moves the lit of valid_archs into morphlib (instead of locally-scoped in MorphologyFactory) * BuildCommand takes an extra argument, build_env * split BuildCommand's get_artifact_object into create_source_pool and resolve_artifacts (plus changes things that use get_artifact_object to use the new way) * setup_mounts finds out whether to do so by whether build_mode is 'staging', instead of by whether the setting 'staging-chroot' is true. * Makes ChunkBuilder's get_sources use the morphlib.builder2.extract_sources() method, and moved set_mtime_recursively into morphlib.builder2, since it's not currently used anywhere else. * moved ChunkBuilder's get_commands into the Morphology class (plus changes to anything that used get_commands)
* Save more than one failed buildRichard Maw2013-07-031-1/+1
| | | | | | I blithely replaced runcmd(['mv', src_dir, dest_dir]) with os.rename(src_dir, dest_dir), without realising that mv does extra work when the target is a directory that already exists.
* Make /etc/os-release have the easy infoJonathan Maw2013-07-021-4/+6
|
* Fix permissions of /etc/os-releaseLars Wirzenius2013-06-201-2/+4
| | | | | Also clarify how the directory containing it gets created to avoid a bit of code duplication.
* Fix warnings and test suiteLars Wirzenius2013-06-201-4/+0
|
* Remove SystemKindBuilderFactoryLars Wirzenius2013-06-201-45/+52
| | | | | | We now only support one system-kind, so there's no need for the factory stuff, and at this point it only serves to obfuscate and complexify.
* Require system-kind to be rootfs-tarballLars Wirzenius2013-06-201-0/+3
|
* Delete system-kind plugins except for rootfs-tarballLars Wirzenius2013-06-201-161/+0
| | | | | Also delete the infrastructure for disk-based system-kinds from builder2.py.
* builder: replace shelling out to mv with os.renameRichard Maw2013-06-061-1/+1
| | | | It's just 1 system call instead of at least 3.
* Change the structure of the temporary directory used by morphTiago Gomes2013-06-051-0/+4
| | | | | | Now, inside the temporary directory we will have the following subdirectories: chunks, staging, failed and deployments. The failed directory will contain the staging areas of failed builds.
* Merge branch 'liw/kill-bare-except'Lars Wirzenius2013-05-281-2/+7
|\ | | | | | | | | | | | | Reviewed-by: Richard Maw I, Lars, note that we have an old version of CoverageTestRunner in Baserock. The new version hides the spurious logging messages.
| * Stop using bare except: statementsLars Wirzenius2013-04-161-2/+7
| | | | | | | | | | | | | | | | | | It is almost never a good idea to catch all exceptions, and then do nothing about them. This patch logs all caught exceptions so that the user has some possibilty to debug what is happening. Also, make ./check check for bare excepts and fail the test suite if it finds anything.
* | Remove excessive debugging messagesRichard Dale2013-05-241-3/+2
| |
* | Include the artifact metadata-version in the cache id keyRichard Dale2013-05-241-1/+0
| |
* | Add contents list to chunk and stratum metadataRichard Dale2013-05-241-6/+13
| |
* | Create an empty fstab if one is missingLars Wirzenius2013-05-021-7/+10
|/
* Corrections to previous commitsRichard Holland2013-04-081-1/+1
| | | | | Removed trailing comma, corrected copyright year, used #pragma: no cover to exclude the exception class form coverage and allow ./check full to run.
* Changed empty stratum warning to errorRichard Holland2013-04-081-1/+7
| | | | | Added new cliapp.AppException class to raise error when attempting to build an empty stratum (no dependencies) and raised it appropriately.
* Don't include bootstrap chunks in output strataSam Thursfield2013-03-131-3/+8
| | | | | This is necessary so that the build-essential stratum doesn't end up containing three different compilers.
* Refactor build processSam Thursfield2013-03-131-12/+9
| | | | | | | | | | | | | | | | | | Reorganise the build_artifact() and build_artifacts() functions to allow more complex work when setting up chunk builds in build_artifact(). The staging area now holds the BuildEnvironment object (the environment variables that should be set during build). This makes sense because all build commands should be run inside the staging area and therefore through the StagingArea object. The BuildEnvironment object is now considered immutable after it is created. The environment is used in cache key computation when computing what artifacts are required; if it changes after that point we risk either computing different artifact keys for the same artifact or missing data in the cache key that should be included in the hash. Better to force changes into a separate 'extra_env' variable.
* Remove features used by old bootstrap methodSam Thursfield2013-03-131-0/+3
| | | | | | That means that bootstrapping Baserock is currently not possible with this branch of Morph, but there's no reason it cannot be bootstrapped using an older version of Morph instead.
* Merge branch 'liw/pre-commands' into stagingSam Thursfield2013-03-071-4/+14
|\ | | | | | | | | Conflicts: morphlib/morph2.py
| * Add pre- and post-commands for chunksLars Wirzenius2013-02-191-4/+14
| | | | | | | | | | | | | | | | We already have configure-commands. Add pre-configure-commands and post-configure-commands. Likewise for build-command, test-commands, and install-commands. Added-to-pacify: Rob Kendrick
* | Add the ability for chunk morphs to specify devicesJonathan Maw2013-02-281-0/+24
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Within a linux-user-chroot, we do not want to allow arbitrary code to create device nodes, but still want it to be possible to create a device node. This commit creates and handles the 'devices' field in a chunk morphology, which takes: * filename. A string, e.g. "/dev/null" * uid. The ID of the user the file belongs to, e.g. 0 for root. * gid. The ID of the group the file belongs to, e.g. 0 for root. * type. A string of either 'c' for a character device or 'b' for a block device. * major. The device's major number. * minor. The device's minor number. * permissions. A string of the octal number that would be passed to chmod e.g. '0777'
* | Hardlink files into the staging-areaJonathan Maw2013-02-281-64/+6
|/ | | | | | Instead of repeatedly unpacking tarballs into a succession of staging- areas, it will unpack each tarball only once, then hardlink all the files into the staging-area instead.
* Make metadata include the repo-alias.Richard Maw2013-02-111-0/+1
| | | | | | | | | This is required for reproducing the morphologies that created the artifacts. It is possible, but expensive, to `unexpand` a url, provided your repository aliases are the same as the morphology's, but this may not always be the case.
* Avoid confusing error when staging area is emptySam Thursfield2013-02-061-2/+3
| | | | | | | | | | | | | | | | | | For Morph to work it's necessary for the staging area to contain a full FHS heirarchy. However, if the staging area is empty we want to give the user an obvious error. Previously we would say something like: OSError: [Errno 2] No such file or directory: '/tmp/ccache' This makes it seem like the problem is with ccache. With this patch, we will instead get as far as: /bin/sh: No such file or directory This should make the real problem clearer. Also, aborting while trying to mount the ccache directory causes the /proc and /dev/shm mounts to persist which is fixed by this patch.
* Include morph-version in artifact metadataRichard Maw2013-01-281-0/+7
| | | | | | This can be used to rebuild an artifact with the exact same version of morph. Eventually there may be a tool to do this, but it allows for expert diagnosis until that point.
* Make ccache use the same directory for repos that are clonesJonathan Maw2013-01-031-4/+15
| | | | | | | | | This patch makes morph store its ccaches by the url's basename instead of using the whole url. This fixes the problem of morph choking when it tries to mount a directory with a colon in its path, and also makes all clones of one repository use the same cache (probably). This will cause projects with the same name to use the same cache, but that is not likely to cause serious problems.
* Add support for ccacheJonathan Maw2012-12-171-0/+24
| | | | | | | | | | | | | | | | | | | This is set up so that each individual project repository has its own ccache, all under one defined directory. The top-level ccache directory is added as the setting 'compiler-cache-dir', and defaults to $cachedir/ccache. When a build is performed, this will bind-mount a project's ccache into the /tmp/ccache of the staging-area and set up the environment variables so that ccache will be used (if appropriate executables are installed to /usr/lib/ccache in the staging-area). In addition, this removes code for ccache-remotedir, as it is unrelated to this implementation of ccache. Reviewed-by: Jannis Pohlmann <jannis.pohlmann@codethink.co.uk> Reviewed-by: Lars Wirzenius <lars.wirzenius@codethink.co.uk>
* Revert "Merge branch 'liw/hardlink-staging-area-pre-rebase'"Jannis Pohlmann2012-11-301-5/+27
| | | | | | | | | | | | | | This reverts commit c859d70d86423a52bc7053abf64e9ca21f62a487, reversing changes made to 3085a672d1e8b5177be33f0463385de72a0ef5bf. Unfortunately, hardlinking and linux-user-chroot both break builds in various ways. Hardlinking breaks the bootstrap process by creating symlinks like /usr/libexec that can then not be overwritten with real files by install scripts from morphologies like gcc. linux-user-chroot caused problems by breaking privileged operations such as chgrp and CAP_SETFCAP. As a consequence, chunks like util-linux and libcap can no longer be built.
* Use the builder runcmd, to get build environment rightLars Wirzenius2012-11-291-6/+4
|
* Reformat code, plus adapt for non-staging-area buildsLars Wirzenius2012-11-291-1/+2
|
* Hardlinked chroot done with linux-user-chrootJoe Burmeister2012-11-291-31/+10
| | | | | | | | | | | | | | The patch gives two things. Improves morph build time by reusing decompressed files of chunks/stage-fillers with hardlinks from the chroot. Rather than decompressing each time into each chroot. Original: real 5h 17m 47s Hardlink: real 2h 52m 27s It uses linux-user-chroot to create the chroot and make all but the basics readonly.
* Don't store original filename in compressed image filesSam Thursfield2012-11-161-1/+2
| | | | | | | gzip files can store the name and date of the uncompressed original, and this is displayed in some GUI archive managers, or when gunzip -N is used. In our case the original filename is something like 'tmpXsgaY15', which is useless information and serves only to confuse.
View Lorry Controller Status