| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
json only accepts unicode. Various APIs such as file paths and environment
variables allow binary data, so we need to support this properly.
This patch changes every[1] use of json.load or json.dump to escape
non-unicode data strings. This appears exactly as it used to if the
input was valid unicode, if it isn't it will insert \xabcd escapes in
the place of non-unicode data.
When loading back in, if json.load is told to unescape it with
`encoding='unicode-escape'` then it will convert it back correctly.
This change was primarily to support file paths that weren't valid
unicode, where this would choke and die. Now it works, but any tools
that parsed the metadata need to unescape the paths.
[1]: The interface to the remote repo cache uses json data, but I haven't
changes its json.load calls to unescape the data, since the repo
caches haven't been made to escape the data.
|
| |
|
|
|
| |
This rips out any remaining order-preserving code and instead uses the
yaml dumper from morphloader.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
I think that it's confusing for both strata and chunk morphologies to
have a 'chunks' field, with the former listing sources and the latter
listing rules for splitting this source into artifacts.
The design for splitting strata has roughly the same idea, but operating
on chunk artifact names, rather than file names, so a name that can be
used for both was chosen.
Splits and artifacts weren't satisfactory names, so they're now called
'products'.
It was decided to break backwards compatibility of chunk morphologies
being able to specify 'chunks', since the format has changed, so extra
code would be required to translate the format, and the only users of
the 'chunks' field was the test suite, since there was no way to select
from the system, which chunk artifacts were included.
|
| |
|
|
|
| |
This it convenient, as it allows the new validation code to validate
the old morphology class during the transition period.
|
| |\
| |
| |
| |
| | |
Reviewed-by: Lars Wirzenius
Reviewed-by: Richard Maw
|
| | | |
|
| |\ \
| |/
|/|
| | |
Changed the error (exception) to list all obsolete fields.
|
| | | |
|
| | | |
|
| |/ |
|
| |
|
|
| |
Add the necessary tests to keep CoverageTestRunner happy.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Cross-bootstrap is a way to build baserock on an architecture that
does not currently have Baserock. It can be used by `morph
cross-bootstrap <ARCH> <REPO> <REF> <MORPH>`, and will build an artifact
that can be used as a root filesystem with a basic build environment
with a script named `native-bootstrap` which will build and install
every chunk in the system.
If done with a devel system, this will give you a suitable environment
for building a proper Baserock system.
This does not currently provide a kernel for the target architecture.
Apart from adding the cross-bootstrap plugin, it also makes the
following changes:
* Moves the lit of valid_archs into morphlib (instead of locally-scoped
in MorphologyFactory)
* BuildCommand takes an extra argument, build_env
* split BuildCommand's get_artifact_object into create_source_pool and
resolve_artifacts (plus changes things that use get_artifact_object to
use the new way)
* setup_mounts finds out whether to do so by whether build_mode is
'staging', instead of by whether the setting 'staging-chroot' is true.
* Makes ChunkBuilder's get_sources use the
morphlib.builder2.extract_sources() method, and moved
set_mtime_recursively into morphlib.builder2, since it's not currently
used anywhere else.
* moved ChunkBuilder's get_commands into the Morphology class (plus
changes to anything that used get_commands)
|
| |
|
|
|
|
|
|
|
| |
Morph no longer supports setting the prefix using the --prefix
argument / setting. This was only used in tests and during bootstrap.
If a chunk build-depends on a chunk within a stratum which has
a custom prefix, that prefix is appended to the PATH in the build
environment.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Allowed values:
staging: build with a staging chroot (default)
test: build with the host's tools
bootstrap: build with the host's tools, and do not include this
chunk in the final stratum artifact
In the past, 'normal mode' has been used to describe building a chunk
with the host's tools. We don't want that mode to ever be used,
because it is a huge hole in reproducability, but we need to keep it
around to avoid making Morph's cmdtest suite depend on Baserock.
Hopefully naming it 'test' should discourage potential abusers.
It is unfortunate that the build tests now take a separate code path
compared to real-world usage of Morph. However, this is necessary to
avoid a circular dependency between Morph's test suite and the
build-essential stratum in Baserock.
We do whole-build testing of Baserock, too, so the 'staging' code path
is still tested outside of Morph. However, testing a staging area
requires populating it with at minimum a working shell, and this is a
bit too complex to go in Morph's test suite.
|
| |\
| |
| |
| |
| | |
Conflicts:
morphlib/morph2.py
|
| | |
| |
| |
| |
| |
| |
| |
| | |
We already have configure-commands. Add pre-configure-commands and
post-configure-commands. Likewise for build-command, test-commands,
and install-commands.
Added-to-pacify: Rob Kendrick
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Within a linux-user-chroot, we do not want to allow arbitrary code to
create device nodes, but still want it to be possible to create a device
node. This commit creates and handles the 'devices' field in a chunk
morphology, which takes:
* filename. A string, e.g. "/dev/null"
* uid. The ID of the user the file belongs to, e.g. 0 for root.
* gid. The ID of the group the file belongs to, e.g. 0 for root.
* type. A string of either 'c' for a character device or 'b' for a block
device.
* major. The device's major number.
* minor. The device's minor number.
* permissions. A string of the octal number that would be passed to chmod
e.g. '0777'
|
| | | |
|
| | |
| |
| |
| |
| |
| | |
Remove the special case hacks we had and do a proper comparison
between original and new in-memory dict when writing updates to
user morphologies.
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| | |
If given an empty string to parse yamlparse.load() will return None,
but this breaks code further down that expects to be dealing with a
dict. Raise an exception to avoid crashing.
Also, avoid catching 'Exception' when we only want to catch JSON
parse errors.
|
| |/ |
|
| |
|
|
|
|
|
| |
This saves a check (and an indentation) in the deployment plugin,
making the code a tiny bit simpler.
Suggested-By: Sam Thursfield
|
| |
|
|
|
| |
This can go away when we have made a release with yaml in it, and its
staging filler.
|
| |
|
|
|
| |
Tests are currently broken, one because invalid JSON
can be valid YAML, and coverage is incomplete.
|
| |
|
|
|
|
|
|
| |
This way everyone who needs json or an OrderedDict doesn't have to worry
about it.
Currently morph2 is the only one that needs this complex behaviour,
other users of the json module don't need object_pairs_hook.
|
| |\
| |
| |
| |
| |
| |
| |
| |
| |
| | |
ssh://git.baserock.org/baserock/baserock/morph
This includes the following fixups:
- altering the bootstrap script to install ordereddict and simplejson.
- Adding a comment to clarify that it is intentional to use simplejson
if collections does not have OrderedDict
- Amending the copyright years to include 2013
|
| |/ |
|
| |
|
|
|
| |
'morph edit' is for editing source code, not built artifacts, so we
should use the source morphology name.
|
| |
|
|
|
|
|
|
|
|
| |
This requires disabling the feature that retains the original order of
fields in a morphlogy when it gets overwritten. The implementation relies
on features that are not available in Python 2.6. We need to support
Morph on Debian squeeze, for bootstrapping purposes, and therefore need
to have it work with Python 2.6. However, the morphology rewriting is
only relevant for system branching and merging, and that isn't needed
for bootstrapping, so we disable the affected tests on Python 2.6.
|
| | |
|
| |
|
|
|
| |
This helps us avoid writing out meaningless fields when we want to
edit a morphology and write it back to disk.
|
| |
|
|
|
|
| |
This is needed for being able to write morphologies to disk
programmatically after making automatic changes to them using
morph.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
In order to modify morphologies in place and then write them back to
disk in system branches (e.g. when running "morph edit"), we need to
have access to the dicts that store references to strata in system
morphologies and chunks in stratum morphologies, respectively.
Therefor, the previous triplet-returning child lookup is replaced
with a new internal method to ensure uniqueness of names in
morphologies and a new method to lookup children in this commit.
The unit tests are adjusted to cover everything in appropriate ways.
|
| |
|
|
|
|
| |
This requires that we enforce uniqueness.
New method: Morphology.lookup_morphology_by_name()
|
| |
|
|
| |
Rename "sources" field of stratum morphologies to "chunks".
|
| |
|
|
|
|
|
|
|
| |
This was done with the aid of the pep8 script, available by running
`easy_install pep8`.
It may be worth making this part of ./check, but that will require
putting pep8 into the development tools stratum.
This should be easy, given pep8 has no external dependencies.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This is an ugly, ugly way to do this, but time is pressing.
SystemBuilder checks what arch is defined in the morphology, if
it is an x86 (or None for compatibility) then it will do the syslinux
install stuff.
This hack is needed because syslinux is x86 specific and arm
often has different requirements for where the kernel must be loaded
from, sometimes it is flash, sometimes it is a different partition.
This will likely become board specific, but for a qemu-system-arm,
the kernel should be a separate file, to be passed on the command line.
Having a different 'kind' for each architecture would be a nicer way,
but would require more changes, since there are various checks for
morphology['kind'] == 'system'
|
| | |
|
| |
|
|
|
|
|
|
|
|
| |
Builder checks whether commands need to be chosen from the build
system by if the morphology has no commands, the check is whether
the commands are None, rather than an empty list, so that it is
possible to override a build system's commands with an empty list.
However this means that the default of an empty list means that the
build system is ignored and commands aren't run.
|
| | |
|
| |
|
|
|
|
|
|
|
| |
We will almost always want to look up sources based on the data we find
in morphologies (e.g. chunk sources found in a stratum or strata found
in a system). For that we need to remember the original_ref in addition
to the resolved SHA1 and look up sources using this original ref. The
original ref is therefore also used as part of the hash key in
SourcePool now.
|
| | |
|
|
|
The old morphlib.morphology.Morphlogy class is entangled by having
treeish and requires reading from an open file. This is a bad design
for the class: the treeish is unnecessary coupling, and the open file
makes is harder than necessary to parse a morphology which we don't
have as a file on disk.
The new class gets the text of the morphology and does not care about
treeishes at all. It also acts more like a dictionary, giving more
uniform access to the various items, while reducing the amount of
code we have in the class.
The old class will remain until all uses of it have been eradicated.
|
