| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
| |
Do not recalculate the shared part of cache keys. In particular, do not
call get_unique_key() multiple times for a single element or source.
Fixes #167
|
| |
|
|
|
|
|
|
|
|
| |
This adds the _update_state() method to the Element class to keep track
of element state and avoid calculating the same cache key multiple
times. This also consolidates the different get_cache_key
methods into a single method that always returns the cache key
calculated by _update_state(), if available.
Fixes #149, #173
|
| | |
|
| |
|
|
|
|
|
|
|
| |
We have a policy that subprocesses can return error messages, but not
actual exceptions.
In particular this allows GLib.Error exceptions to be returned, which
would otherwise be silently lost due to
https://gitlab.gnome.org/GNOME/pygobject/issues/145
|
| |
|
|
|
| |
This reduces the differences between local and remote artifact
repositories, increasing code coverage of tests.
|
| |
|
|
|
| |
Closing stdout will cause an exception when used with multiprocessing as
that calls sys.stdout.flush() as well.
|
| |
|
|
|
|
| |
initializing cache
This fixes issue #141
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
If you have an artifact remote cache in your config using the ssh://
protocol and some SSH issue prevents us from connecting, we now show
the user the stderr output from the `ssh` program, e.g.
[--:--:--] WARNING Failed to fetch remote refs from ssh://artifacts@172.17.0.2:22200/artifacts/: BuildStream did not connect successfully to the shared cache ssh://artifacts@172.17.0.2:22200/artifacts/: Permission denied (publickey,keyboard-interactive).
Previously we would just show the error message raised by the protocol
code, which was less useful, e.g.:
[--:--:--] WARNING Failed to fetch remote refs from ssh://artifacts@172.17.0.2:22200/artifacts/: BuildStream did not connect successfully to the shared cache ssh://artifacts@172.17.0.2:22200/artifacts/: Expected reply, got none
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
| |
This adds a new test for parsing artifact cache configuration, which
calls the helper function from the 'artifactcache' module directly
rather than trying to assert based on blind push and pull commands
whether or not we got the complex precedence rules exactly right.
This means frontend push/pull tests no longer need to be so thorough
about testing precedence but they are instead expanded to assert that
multiple caches work correctly.
|
| |
|
|
|
|
|
|
|
|
| |
The initial multiple cache support patch implemented a rather fragile
logic where we would push to the first cache in the list that used the
ssh:// protocol, if any. If we implement push-over-https:// in future
then this will become totally unworkable.
This patch alters the logic so that each remote has a 'push' option,
and BuildStream will push to any remote that has 'push: true' set.
|
| |
|
|
|
|
|
|
| |
This allows pushing and pulling from a specific cache, ignoring what is
configured.
If we choose to add a --remote option to `bst build` in future that
would now be simple to do.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This extends the 'artifacts' configuration block such that a list of
`url` mappings can be given instead of a single entry. For example:
artifacts:
- url: http://example.com/artifacts1
- url: ssh://ostree@example.com/artifacts2
The OSTreeCache class is updated to set up multiple remotes and query
remote refs from all of them.
There are no automated tests for this yet.
Empty URLs ('') now raise an exception. They cause breakages internally
if we allow them through, and they can only occur if the user or our
tests are misconfiguring things somehow.
We report failure to fetch from the cache by printing a message to
stderr for now. This is because BuildStream's actual logging
functionality can't be used during frontend init -- see issue #168.
|
| |
|
|
|
| |
Minor tweak to hopefully make the test cases a bit shorter and more
readable, in preparation for adding more.
|
| |
|
|
|
|
|
|
| |
Added app.print_error() to consistently print an error while
running the pipeline stuff.
Also make bst push & pull commands more consistent with
the rest, and print the summary at the end as expected.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
o Now print the message itself on the same line in place of the abbreviated log file
o When reading back the last lines of the log file, prefix that
with an informative message about whence the log lines come,
showing the full unabbreviated log file.
o Fixed the behavior of read_last_lines() so that it works as
intended, problems were:
- Was prepending an additional newline when the log contains
more lines than we want to print
- Was not printing anything at all in the case we want to
print more lines than exist in the log
|
| | |
|
| | |
|
| |
|
|
|
|
|
| |
Now any BuildStream exception can potentially add detail to
the errors they raise.
Allow detail strings already in PluginError()
|
| |
|
|
|
| |
Dont make the ERROR message explicitly anymore, leave that
to the frontend to sort out.
|
| | |
|
| |
|
|
|
|
|
| |
This is already problematic as this particular error is
reported but fails to fail the session correctly.
Documented this in issue #187
|
| |
|
|
|
|
|
|
|
|
| |
using it.
Errors are fatal, as such plugins must raise either SourceError() or ElementError().
This method was never used in plugins so far, lets remove it even
though we are technically breaking (unused) API for the overall
betterment of the API.
|
| | |
|
| |
|
|
|
|
|
|
| |
This works around an inconsistent behavior with setuptools.
Newer versions of setuptools fail to preserve symbolic links when
creating a source distribution, meaning that tests run from the
dist tarball will fail.
|
| |
|
|
|
|
|
| |
We had a cache key instability issue with local sources generating
inconsistent cache keys due to iterating over their files in
a random order. Tests did not catch this previously due to the local
source sample only using one file to iterate over.
|
| |
|
|
|
|
|
|
| |
Technically this breaks cache keys for the local source, but as
this comes in a branch which fixes local source cache keys to be
stable (they were random before this branch), we wont bother with
considering this enhancement a separate API break, the cache key
breakage was inescapable anyway.
|
| |
|
|
|
|
|
|
|
|
|
| |
This is needed because plugins make use of this function to
generate a cache key which must be stable.
In addition to the above, this patch also mitigates the performance
hit of sorting, and allows _process_list() to function to iterate
through the generator when additional sorting is not needed.
This patch is an enhanced version of Sam Thursfield's patch on MR !216
|
| |
|
|
| |
libfuse
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
This fixes issue #183.
Also move the `format-version` related documentation to the first
section "Essentials", beside the project name and element path, since
this is a quite global option it belongs here and not hidden away
with the plugin loading documentation.
Also adjust the main index.rst to include the plugins sub-section
as an adjacent sub-point of the project configuration (consistent
with other project configuration sections).
|
| |
|
|
|
| |
Also removed some things which are either of little importance, not
worth mentioning, or covered already by the buildstream user documentation.
|
| |
|
|
|
| |
It's more suitable to add this note here than to have a separate section
in the HACKING.rst for this.
|
| |
|
|
| |
error time
|
| |
|
|
|
| |
If a plugin raises an error, prepend the plugin identifier to the
error message and raise PipelineError.
|
| |
|
|
|
|
|
| |
This also makes main.py less redundant, for some reason there was a stray
call to app.pipeline.initialize() beside app.initialize() in every function,
instead move that into app.initialize() where we now handle the error
and exit gracefully.
|
| | |
|
| |
|
|
|
|
| |
Enhanced the error checking Result() methods to always assert that
the CLI actually exited, there are no cases worth testing for where
buildstream would be expected to exit on an unhandled exception.
|
| | |
|
| |
|
|
|
|
|
|
|
| |
It looks like this newly added test assumes the user has a very recent
version of pytest, which supports treating the `tmpdir` fixture like
a string.
A reasonable alternative to this patch would be require at minimum
a version of pytest which supports this newly introduced API.
|
| | |
|
| |
|
|
|
| |
Instead add a comment about why this is really there,
and invoke sphinx python modules with python3 directly.
|
| |
|
|
|
|
|
|
|
|
| |
Some of the warnings from sphinx-build are really just warnings,
but a lot of the things we want to avoid and really break documentation,
like broken internal references and some invalid rst directives should
really be errors.
Now we treat all warnings as errors, this should ensure that
any commits landing upstream never break the docs.
|
| |
|
|
|
|
| |
This makes writes done by `bst track` atomic, meaning a crash or power
loss while overwriting a .bst file should never cause the the contents
of that file to be lost or partially written.
|
| |
|
|
|
| |
This avoids a circular dependency between the 'utils' and '_yaml'
modules.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
| |
This is a context manager that can be used to divert file writes into a
temporary file, which is then renamed into place once writing is
complete. It is primarily intended for use by source plugins which
download files, so they can ensure that their downloads appear atomic
and there is no risk of leaving half-downloaded files in the cache.
So far this is not used in the core, but it is needed by the Docker
source plugin that is proposed for the bst-external plugins repo. See:
https://gitlab.com/BuildStream/bst-external/merge_requests/9
|
| |
|
|
|
|
| |
Fixes the warning:
WARNING: /path/to/buildstream/doc/source/install.rst:45: (WARNING/2) Title underline too short.
|
| |
|
|
|
|
|
|
| |
Recently I added the `reason` member which can be used to set
machine readable error reason strings for the purpose of testing.
Forgot to add the necessary `*` argument, forcing `reason` to be
a keyword-only argument.
|
| | |
|
| |
|
|
| |
Some basic module metadata before publishing the 1.0 release.
|
