summaryrefslogtreecommitdiff
path: root/morphlib/builder.py
Commit message (Collapse)AuthorAgeFilesLines
...
* Builder.build and build_single also do not need to return values anymoreLars Wirzenius2012-02-281-6/+1
|
* Remove unused needs_built methodLars Wirzenius2012-02-281-4/+0
|
* Set sensible defaults for git-base-url, bundle-server, cachedir, max-jobsLars Wirzenius2012-02-281-4/+1
| | | | | | | | | | | | | | Also, change builder.py to always obey the --max-jobs setting, unless a morphology has a max-jobs field. The defaults have been chosen so that they work for everyone equally well. It may be useful to have a local mirror and then set the options to point there, but it's not reasonable to try to guess such things, so the defaults can be adapated to that. Collect the defaults into one place so they're easier to overview. The cliapp interface for adding settings is verbose enough that the defaults were getting buried.
* Make ./check check for long lines (with excptions); fix long linesLars Wirzenius2012-02-271-1/+2
|
* Create chunks, strata in cache via temporary filesLars Wirzenius2012-02-271-5/+13
| | | | | This avoids problems with files with the right names but partial content, if morph is killed in the middle of writing the file.
* Hide git output from unit test outputLars Wirzenius2012-02-271-1/+1
| | | | | | | | | | | | This makes it easier to check the output with vgrep for errors or problems. Voluminous git output made that hard. The key is that "git bundle create" has no --quiet optio (meh), so we have to redirect stderr to /dev/null. However, that would hide real errors, which we want to catch. Luckily, we can just replace the call to subprocess.call with a call to subprocess.check_call, and then we'll at least abort if there's an error, instead of silently ignoring it.
* morph: drop special case .git in mtime settingRichard Maw2012-02-271-6/+4
| | | | | | | | | | | | | Either all .gits need to be left alone, including submodules, or all of them get changed. Doing all gets surprisingly complicated, as bottom-up traversal is needed for directories to have their mtimes set, which means that either we check the path for a .git component, or we do our own traversal function, to top-down exclude .git and bottom-up set mtimes. This is more effort than it is worth when git probably doesn't care so let's just drop it entirely.
* morph: use right dir for set mtimeRichard Maw2012-02-271-1/+1
|
* morph: set mtime after submodule extractRichard Maw2012-02-271-1/+1
| | | | | | | | Setting mtimes can fail if it is a symbolic link and it points to a non-existant file. This has happened when we had symbolic links pointing to files in a submodule, so we set mtimes after submodules have been checked out.
* Exclude the .git directory from mtime changing, just in caseLars Wirzenius2012-02-241-5/+7
|
* Set mtime of all files after git checkout of source tree for buildingLars Wirzenius2012-02-241-0/+19
|
* Use "cp -a" and "git checkout" to unpack sources in the build tree.Jannis Pohlmann2012-02-221-2/+9
| | | | | | | | | This is done to keep the git repository intact with history and all that. cp + checkout turns out to be faster than a regular "git clone". In order to avoid roundtrips to the internet whenever a chunk tries to do something with submodules, we force the submodule URLs to point to our locally cached repos.
* builder.py: use sh -c instead of writing a scriptRichard Maw2012-02-211-11/+2
| | | | | | | | | | | | | | | | Putting the commands in a file made it more difficult to tell what happened in scripts, until the self.msg was added. After then it made the commands more difficult to reproduce So if it can safely be done in a single command it should be more readable. This approach uses sh -c to cd then evaluate the remaining options The command is 'cd "$1" && shift && eval "$@"', then passes the command as arguments to the shell script, so the program does not need to do any escaping of strings. If the commands were a list of strings instead of a quoted string then it would be a bit safer, but less flexible for use.
* Add Assert to make sure the destdir muckery is safeLars Wirzenius2012-02-171-0/+1
|
* Remove staging path prefix from $DESTDIR when we run things in chrootLars Wirzenius2012-02-171-0/+7
|
* Show the commands to be run in chrootLars Wirzenius2012-02-171-0/+1
|
* Merge branch 'master' of gitorious.org:baserock/morphLars Wirzenius2012-02-161-1/+2
|\
| * build-system: autotools now runs autoreconfRichard Maw2012-02-161-1/+2
| |
* | Fix how ldconfig is runLars Wirzenius2012-02-161-2/+2
| | | | | | | | Now it should actually work.
* | Build non-bootstrap chunks inside a staging chrootLars Wirzenius2012-02-161-4/+27
|/ | | | | | This adds options --staging-filler and --staging-chroot. The wisdom of these options needs to be re-considered at some point, but for now they're OK.
* Fix overriding of build system commandsLars Wirzenius2012-02-151-1/+15
| | | | | This time, with testing. For a five line method, this was surprisingly tricky to get right. Should sleep properly at night...
* Allow *-commands to override commands provided by build systemLars Wirzenius2012-02-151-18/+25
| | | | | | | | | | | | | This allows us to do this: "build-system": "autotools", "configure-commands": [ "./configure --prefix=$PREFIX --disable-nls --enable-foo" ] The explicit command is used for configure, but the commands provided by the autotools build-system will be used for build-commands, test-commands, and install-commands.
* Revert "Add --prefix option, adjust autotools system and env variables."Jannis Pohlmann2012-02-141-26/+6
| | | | This reverts commit 63cab3e0e2f550a0bfe470d018063a8b91a759bd.
* Revert "Set environment variables needed to build in a different prefix."Jannis Pohlmann2012-02-141-21/+19
| | | | This reverts commit e0fb97f08156af97ec2341ec8061fe514ae358e5.
* Set environment variables needed to build in a different prefix.Jannis Pohlmann2012-02-111-19/+21
| | | | This only includes PKG_CONFIG_PATH and LD_LIBRARY_PATH at the moment.
* Add --prefix option, adjust autotools system and env variables.Jannis Pohlmann2012-02-111-6/+26
|
* Extract chunk submodule repos into the build directory directly.Jannis Pohlmann2012-02-081-18/+22
| | | | | | | | | | | | | This change requires the latest version of cliapp to be installed. Rather than creating an archive from the cached git repositories, moving this tarball archive into the cache and then extracting it from there into the build directory, we now directly extract the sources into the build directory. We also recursively extract all submodules into the corresponding paths inside the build directory. (Recursing has not been tested yet, however.)
* Set GIT_DIR when running git archive. Drop --remote.Jannis Pohlmann2012-02-021-1/+2
| | | | | This change is necessary because upstream git no longer allows --remote to be combined with an arbitrary SHA1.
* Save the build log of each blob as $cache_prefix.log.Jannis Pohlmann2012-01-311-2/+24
| | | | | This may not be the optimal way of doing it but it is a start towards making debugging failed builds a little easier.
* Properly unpack dependencies into the staging area in build-single.Jannis Pohlmann2012-01-311-21/+24
| | | | | | | | | | | | This requires build-single to take a dependency context tuple when building chunks of a stratum. This context tuple is the surrounding stratum which is used to construct the dependency graph in the worker and then do a breadth-first search to collect all dependencies that need to be added to the staging area. Implementing this required a few hash/eq changes in Blob, Morphology and Treeish as well as a few adjustments in the corresponding unit tests.
* Add a "build-single" command to build a blob without its dependencies.Jannis Pohlmann2012-01-261-2/+35
| | | | | ...while at the same time making sure that all dependencies are marked for staging prior to building the blob itself.
* Disable logging in SourceManager (too verbose), save build times again.Jannis Pohlmann2012-01-231-1/+3
|
* Port everything to using Treeish objects instead of (repo, ref).Jannis Pohlmann2012-01-201-39/+34
| | | | | | | | | | | This affects pretty much every part of morph, so this might not be fully working and stable yet. This commit also introduces the "update-gits" command that can be used to update all cached repositories from the list of base URLs. The tree walk when resolving the Treeish objects in Builder.get_cache_id() is a bit similar to what we do in BuildDependencyGraph, maybe we can merge that one day.
* Fix coding style, update copyrights, drop unused imports.Jannis Pohlmann2012-01-191-0/+1
|
* Add support for system images in BuildDependencyGraph and new builder.Jannis Pohlmann2012-01-191-10/+8
|
* Apply code review changes and fix a bug when clearing dependencies.Jannis Pohlmann2012-01-191-13/+12
|
* Initial work on integrate the build order work into builder.Jannis Pohlmann2012-01-191-158/+185
|
* Fix cmdtests for SourceManager changesRob Taylor2012-01-191-1/+4
|
* Introduce the "show-dependencies" command and BuildDependencyGraph.Jannis Pohlmann2012-01-161-4/+1
| | | | | | | | The "show-dependencies" command takes a series of build tuples and dumps the resulting dependency graph (including strata and chunks at the moment) to the standard output. It also dumps the resulting build order which is a list of groups. These groups indicate which chunks and strata can be built in parallel and are not dependent on each other.
* make a Stopwatch usable with the "with" statementLars Wirzenius2012-01-121-121/+105
| | | | | | | | | | | | | | | | | | This simplifies its use. Previously: w.start('foo') ... w.stop('foo') Now: with w('foo'): ... With "with", it is immediately clear from the code, and its indentation, what is being measured. With the old code, you have to hunt for the start and stop method calls.
* Add MorphologyLoader and use it in the Builder class.Jannis Pohlmann2012-01-121-12/+5
|
* add ldconfig location(s) to PATH before running itLars Wirzenius2012-01-111-0/+3
| | | | | | | | Previously, we required the caller to add /usr/sbin (or other possible locations for ldconfig) to PATH before invoking morph. This was inconvenient, so now morph adds the FHS-compliant possible locations itself, to save the caller a bit of pain.
* stop morph from using fakeroot and sudoLars Wirzenius2012-01-111-33/+28
| | | | | | | | | | | | | Instead, assume the whole build will be run, by the user, under fakeroot or sudo (the latter for system image builds). This allows us to run all non-system-image-build tests without root access at all. We now always create the cache directory, if missing, even if we're running as root. We no longer run ldconfig if ld.so.conf is missing. It is missing during our tests, but is (now) created by fhs-dirs for real builds.
* run ldconfig when it's necessaryLars Wirzenius2012-01-091-0/+28
| | | | | | | Actually, run it whenever it _may_ be necessary. It's fast enough that it's worth running even when not strictly necessary, such as when unpacking a chunk that does not contain shared libraries.
* set morph.filename to be the whole pathLars Wirzenius2011-12-121-1/+2
|
* make a test subcommand that builds and gets the morph, image filenameLars Wirzenius2011-12-121-2/+2
|
* Merge branch 'richardmaw/login'Richard Maw2011-12-121-16/+22
|\
| * Add terminal login supportRichard Maw2011-12-091-1/+1
| | | | | | | | Alter extlinux.conf so that /sbin/init is used instead of /bin/sh
| * Made some sudo commands into fakeroot commandsRichard Maw2011-12-081-26/+21
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Some commands can remain as fakeroot if they share the same fakeroot This can be done with the -i and -s parameters to fakeroot Now the Execute objects contain a reference to the file that stores their state, so fake commands' fake state is recognised by later commands The creating a system image definitely needs to be sudo though. Morph needs to create some files as part of the process, so unfortunately the equivalent to 'echo foo | sudo tee bar >/dev/null' is being used. They could be created with sufficient permissions, written then have better permissions set, but it seemed neater this way A better solution may be to have a subprogram that does the system image creation Putting files into the staging area for building other chunks is currently sudo, but shouldn't have to be. Rather than having a fakeroot per Execute object, it may be better to have a global one, though this would prevent concurrent use of fakeroot.
| * Make the install parts of morphs run as sudoRichard Maw2011-12-081-5/+16
| | | | | | | | | | | | | | | | Then fix all the places where this broke things because they didn't have the required permissions. This is a potential security risk of course, it would be preferable if the install commands were run in a fakeroot session and only the final image building was run as root, but it needs to work first
View Lorry Controller Status