| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Now that we don't support system branches, we don't need to iterate over
a list of repos to create temporary build branches, commit their local
changes and push those branches. We only need to do this for the
definitions repository itself.
A bug is also fixed where the local build branch was not being deleted
due a missing call to _register_cleanup() when commiting the changes.
This commit also renames some functions to more clear names, and moves
the logic on pushed_build_branch() to the definitions_repo module, the
only place where it is used.
Change-Id: Id86240d0c189245bed36bc46355be13d46498dbc
|
|
|
|
| |
Change-Id: Ifb2bc2e2084806fe70b6db96828a390d3082288f
|
|
|
|
|
|
|
| |
The conversion was made so that the build command was used instead of
the build-morphology command.
Change-Id: I67c43d765ed603ecdd806bc649815526243b7b87
|
|
|
|
|
|
| |
The build-morphology command will be removed in a future commit.
Change-Id: I1048062e5d8556622a71276c4b8fae2d3283af19
|
|
|
|
|
|
| |
Also, add integration tests to prevent regressions.
Change-Id: I80421112382168a71ca69e4bafe3c3a174e7f9ef
|
|
|
|
|
|
| |
Also, move the print-architecture contents to noncore-plugins.yarn.
Change-Id: I6a78c8d49c7ad70144845f6ce313fa6d44fa8fd2
|
|
|
|
| |
Change-Id: I9a37ff80a43f8930a55f466a053fb54de8a02570
|
|
|
|
|
|
|
| |
Rename the git repository where the morphologies used for testing are
kept, from morphs to definitions.
Change-Id: Icd5fa6609ddec3993dd4c772cd916359843ef500
|
|
|
|
| |
Change-Id: I185377090ba837805b492f8dd1b31360843e9548
|
|
|
|
| |
Change-Id: Ibf1edd1b0e58c600167eb2f1394e464d1f0041c1
|
|
|
|
| |
Change-Id: I853d5c3f7be920835694f390838870634b16b147
|
|
|
|
| |
Change-Id: I7766d3eac28cf34aedb7f3edbc0ff3dda26ac79a
|
|
|
|
| |
Change-Id: Id470c7a77a47c89118a5d9d0d23b2206d8a839e4
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
With `morph edit` removed, there is no need to load all the
morphologies, check whether a chunk was `morph edited`, change the
current ref to the build ref and write back the morphologies to a
temporary branch.
That is a lot of work as code profiling demonstrated. With this patch
applied, morph execution finishes around 10 seconds sooner on my
machine. This is not a big achievement when a full build is performed
where the wall clock time is dictated by the actual build commands, but
it will provide a much quicker feedback when the build artifacts were
already cached, or the semantic validation of morphologies fail.
We add the option `--untracked-files` to _get_status() in the GitIndex
class so that uncommitted morphologies on a uncommitted directory are
considered. Previously this was being done by calling the following
call in inject_build_refs():
self._root_index.add_files_from_index_info(
self._hash_morphologies(self._root, morphs.morphologies))
This commit also removes some now unused code.
Change-Id: I14215db5c06ab06045ce901131e4e341271a039d
|
|
|
|
|
|
| |
This class was only used on the now removed branch-from-image command.
Change-Id: I75a0b4618b16a24bd9f3b3ea7b3a6228db723715
|
|
|
|
|
|
| |
RIP.
Change-Id: I6aac995415c5d67c60687367697173be52cd2bde
|
|
|
|
| |
Change-Id: Ie66b543eb67282c08a2651062727a4583057ee2d
|
|
|
|
|
|
|
| |
Remove instructions for creating a workspace and update the cluster
example.
Change-Id: If0aae9cb5c8ba94af44c0468905e0d2a395eb62e
|
|
|
|
| |
Change-Id: Ie190d444346448f928a1bb6aaf0863f2fd4521b8
|
|
|
|
|
|
|
| |
This should help clear the way for the eventual removal of deprecated
workspace code and related commands: morph checkout etc
Change-Id: I66298c4f9f790848962bd70ab434bef96cf96cb9
|
|
|
|
|
|
| |
These must have been missed when we removed the edit command.
Change-Id: Ia0eca7eab50dec34f2e02193d3ff676b0e78b33c
|
|
|
|
|
|
|
| |
Static analysis showed this code is not being used, and we can always
bring it back if necessary through git.
Change-Id: Id8bf7d73436b5c3d0dfe050befaae034a05afc86
|
|
|
|
|
|
|
|
|
|
|
| |
fixes
ERROR: [Errno 2] No such file or directory:
'/usr/lib/python2.7/site-packages/morphlib/schemas/defaults.json-schema'
when running morph build
Change-Id: I440a274f9eff4fe86450064538ebc1ed36945ea2
|
|
|
|
|
|
|
| |
Tests were broken since https://gerrit.baserock.org/#/c/1418/ was
merged.
Change-Id: I4e5a139d84946514634214938990cca3c86f4512
|
|
|
|
|
|
|
| |
YBD has been doing this for a while and seems like it is faster as a
result.
Change-Id: I8f95a53195cdbc2c75c06a8abe9eb089a84b1c1b
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Currently Morph enforces that chunk A must be defined before anything
that build-depends on it. YBD doesn't enforce that.
Definitions format at <http://wiki.baserock.org/definitions/current/>
doesn't mention ordering currently. I propose that we make Morph be
permissive about ordering, like YBD is, and update the spec to mandate
no restrictions on ordering.
Since behaviour was previously undefined, making Morph be more
permissive about this shouldn't require a new version number of the
definitions format.
I still think we need to make sure stratum .morph files are ordered
logically, but that is in the realm of 'code style', it shouldn't be
being enforced by a build tool.
Change-Id: I425f2e5b9dfb62e4a26ed11f5c50e3978a0dd1a6
|
|
|
|
|
|
|
|
|
|
|
| |
Accordingly to [1], the listdir function (called by os.walk) returns a list of
filepaths on an arbitrary order. Sort the list of file paths to produce
always the same list.
[1]: https://docs.python.org/2/library/os.html
Change-Id: I4bb9842b1722f27a8becb9c50391cda089bb0a33
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When attempting to get a repo with no commit at the ref we want,
we get a 'No such file or directory' error, which is confusing,
this is because we attempt to remove the directory we plan to clone
into, but if there's no commit at the ref we want then we don't
get as far as running git clone, so the directory doesn't exist,
and we fail to remove it.
This commit improves the error message by catching the InvalidRefError
separately.
Change-Id: Ibca20dd995d858713e56ab834daa97a8297cafc8
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This commit makes it an error for a system to contain duplicate chunk sources,
example error message below,
ERROR: Multiple `syslinux' chunks detected:
upstream:syslinux|d715b39c0801ecea5e52f9029cea7c76320f93cf|strata/bsp-x86_both-tools/syslinux.morph|syslinux
upstream:syslinux|2aab8555987b547b617cbb887e61083fece01541|strata/bsp-x86_64-generic/syslinux.morph|syslinux
Multiple `nasm' chunks detected:
upstream:nasm|78bdad3d14fb875d5f2062957e326ba2a9e4ccb0|strata/bsp-x86_64-generic/nasm.morph|nasm
upstream:nasm|78bdad3d14fb875d5f2062957e326ba2a9e4ccb0|strata/bsp-x86_both-tools/nasm.morph|nasm
Change-Id: I1d1539a46ce6eb098d3a559295ab9a08d6d2865c
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This adds a new 'Defaults' class to represent definitions defaults
The Python 'jsonschema' module is used to validate the contents of the
Defaults file. This module is already included in Baserock 'build' and
'devel' reference systems by way of the 'openstack-common' stratum.
This commit embeds a copy of the JSON-Schema schema for the DEFAULTS
file. I think the canonical location of this schema should be in the
reference definitions.git, for now. In future, the schemas should maybe
have their own repos. Either way, Morph should embed a copy for the time
being so that we are sure the schema matches how Morph expects to parse
the file.
Morph's automated tests are all updated to use definitions version 7.
I removed most of the tests for built-in build systems, because the
built-ins themselves are no longer part of Morph. Only the mechanism for
defining them needs to be tested now.
Change-Id: I65f8f1c967683ef605852bfae5c68518e53f9981
|
|
|
|
| |
Change-Id: I493fced8cf2664283923f6f41097ca991d3fc3de
|
|
|
|
|
|
|
|
| |
Some tests were removed in commit f4d3cbaf1d6e5c3bdbb1b2e2 but it seems
there are still code remnants that we don't need any more. There may be
more -- I haven't done a thorough job of this.
Change-Id: I0d7151402365f9e6f23a93a1d30df0198de9ef02
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When preparing the build directory for a chunk, Morph clones the Git repo
being built (and any submodules) into the staging area.
Instead of using `git clone --no-hardlinks`, the
morphlib.git.copy_repository() function uses 'cp -a' plus some faffing
to achieve the same thing a bit faster.
The 'cp -a' command will preserve the ownership and permissions of the
cached copy. These may not match up with which user is doing the build.
I found that files in my Git cache were owned by UID 1002. This caused
chunks that needed to access the .git directory at build time to fail
with strange errors. Worse, it would trigger a bug in Git[1] that leads
to a fork bomb, which would cause Linux to freeze up completely. This
occured even though I was building as `root`, because of the way
`linux-user-chroot` drops certain privileges: presumably, setting
SECBIT_NOROOT leads to CAP_DAC_OVERRIDE being unset.
To avoid this bug, the code now ensures the copied .git repos are owned
by the user and group who ran `morph`.
Another way to fix this would be to change the
morphlib.gits.copy_repository() function to use `git clone
--no-hardlinks`. This is what YBD does. I found that there is a slight
speed benefit to using the current code ... these are results of cloning
repos from the cache with the 2 methods:
With 'cp -a' plus a chown:
upstream:bison: 4.54 seconds average (10 results)
upstream:ybd: 0.13 seconds average (10 results)
upstream:linux: 40.51 seconds average (10 results)
With 'git clone --hardlinks':
upstream:bison: 6.23 seconds average (10 results)
upstream:ybd: 0.11 seconds average (10 results)
upstream:linux: 43.36 seconds average (10 results)
Test code is: <https://gist.github.com/ssssam/833e0ef8d04fb1fb6ff3>.
Ideally we would fix `git clone --no-hardlinks` to be faster, but we
may as well keep the existing code for the time being.
[1]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=750687
Change-Id: Ieea87322ea7b7f62975b9480f877755665656217
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Loading .morph files is becoming a bit more complicated, as we need to
deal with the VERSION file, and possibly soon with a DEFAULTS file as
well.
The logic of loading and parsing .morph files is done either in the
sourceresolver module, or the morphloader module. This change means that
all users of the latter module can use the get hold of a
MorphologyLoader instance with VERSION already parsed. If DEFAULTS is
added then it is also simple to parse DEFAULTS.
Change-Id: Ib33756e9dbd078e38f12dd7f776c89584b178959
|
|
|
|
|
|
|
| |
Using app.status() produces nicer logs as the messages will have the
status prefix prepended.
Change-Id: I0573d7066784afdbfb878bed85af704e82bb9797
|
|
|
|
|
|
|
|
|
| |
Replace cliapp.shell_quote() with pipes.quote() to produce the chroot
scripts. Despite pipes.quote() being deprecated, it produces much less
escaping than cliapp.shell_quote(). This method should be replaced with
shlex.quote() when migrating to Python 3.
Change-Id: I4b8e8eefe0cd321458bb3fae72f6d0552680f84f
|
|
|
|
| |
Change-Id: I276a00bb9117aaa2d4fcc7e187b846eedae65591
|
|
|
|
|
|
| |
... instead the more common autogen[.sh]
Change-Id: I3d797581708ffaefd1a1d4880253cfc2a3f5aa85
|
|
|
|
|
|
|
|
|
|
| |
Commit a4d8098b229592db40565747b0444b518bf8a6eb removed the `morph
foreach` command, which broke some `morph edit` tests. I don't think
we are ready to remove `morph edit` yet (documentation still mentions
it) but we can remove these tests, which require `morph foreach` to
work.
Change-Id: Id372760f69f31a46d1d872ee36ec6fdfb324ce34
|
|
|
|
|
|
|
|
|
| |
Looks like,
2015-07-05 16:08:10 [Build 1/304] [stage1-binutils] Fetching to local cache: artifact stage1-binutils-misc
stage1-binutils-misc[##################### ] 51.9/73.0 MB
Change-Id: Ib10f1cfaa0c1df80ae605ecfeb5b706c8d46c4a4
|
|
|
|
| |
Change-Id: Iedefd1745e85e1f30d45ee8d26b315760a542a0b
|
|
|
|
|
|
|
|
|
|
|
|
| |
Pass the Source to the staging area constructor so that we don't need to
pass it as a parameter when we call some StagingArea methods. Also move
the creation of the build and destdir directories to the constructor so
that we can get rid of the chroot_open() and chroot_close() methods.
Also, provide API to retrieve the relative locations for buildir and
destdir.
Change-Id: I6e8085392e19ff3d8df807f260acf90eec9e0901
|
|
|
|
| |
Change-Id: Iad95af65bd5c528d2e72f5b2ffa80a01152f50ff
|
|
|
|
|
|
|
|
| |
We're beginning a transition away from workspaces,
foreach in particular is possibly one of the least used commands,
and is only mentioned in tutorials explicitly marked as 'old'.
Change-Id: I2c95ce689bb5550bf50feb6b55be6c58671c4c4a
|
|
|
|
| |
Change-Id: Icec7e948c60decefe5a8bab6141bc89603c265b7
|
|
|
|
| |
Change-Id: I6c45763fb4d9f304695c1894625ced36bda36f89
|
|
|
|
|
|
|
|
|
| |
This change, suggested in story #16, causes the testing suite to use
the user-defined tempdir in /etc/morph.conf if it exists or continue
using /tmp if not. This will prevent the test suite from filling drives
that are not intended to be used for development work.
Change-Id: I94edabe4ea1eca7a55ece7424ece7bbd4b68c513
|
|
|
|
| |
Change-Id: I9e5b1a70946a87eb29009df2def3bd98ecc7ad2a
|
|
|
|
|
|
|
|
| |
ExtUtils::MakeMaker is preferred, Module::Build was meant to replace it
but essentially wasn't good enough, some projects still use Module::Build
though.
Change-Id: I124ee7b33f32167302e9bcb5299f6422f4fc346e
|
|
|
|
|
|
| |
Install to default locations rather than overriding.
Change-Id: Ieccd06afdf623eee0d09b8d29d40d147dc8c5e7c
|