| Commit message (Collapse) | Author | Age | Files | Lines |
... | |
|
|
|
| |
staging artifacts
|
|
|
|
|
|
|
|
|
| |
* Adds the 'overlap-whitelist' field to elements' public data. This is a
list of globs that match files that the element is allowed to overlap
other elements with.
* Adds the project-wide 'fail-on-overlaps' field. If set,
non-whitelisted overlaps will raise an error instead of printing a
warning.
|
| |
|
|
|
|
|
|
|
| |
In case a source with an open workspace is tracked and it's ref gets
updated, BuildStream should inform the user that the new ref will not be
picked up so long as the workspace is open. To start using the updated
ref, the existing workspace will have to be closed.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This functionality is only supported for sources which have an open
workspace. When such sources are present, the workspace directory will
be mounted directly inside the sandbox. As opposed to the default
behavior, which is to copy files inside the sandbox.
This will save time when building large projects as only those files
will need be re-compiled that have been modified during two consecutive
builds (assuming the underlying build system supports such behavior).
A few things to note regarding this behavior:
- If there are any `configure-commands` present, they will run only once
for each open workspace. If an element has multiple workspaces and any
one of them is opened/closed, they will be executed again on the next
run. But, modifying the contents of a workspace will not trigger the
`configure-commands` to be executed on the next run.
- Workspaced builds still leverage the cache. So, if no changes are made
to the workspace, i.e. no files are modified, then it will not force a
rebuild.
Fixes #192.
|
|
|
|
| |
Do not depend on caller calling it on its own.
|
| |
|
| |
|
| |
|
|
|
|
|
| |
Do not consider pull as pending if pull failed. This is required to
allow fallback to fetch and build after pull failure.
|
|
|
|
|
|
| |
This fixes strong cache key handling when the local cache contains an
artifact that matches the weak cache key and the remote cache contains
an artifact that matches the strict cache key.
|
|
|
|
|
|
|
| |
This moves the pull status logic from PullQueue to Element.
As PullQueue.process() simply calls Element._pull(), it doesn't make
sense to keep the status logic in PullQueue. This is also required to
use the pull status in other places without code duplication.
|
|
|
|
|
| |
Wait for cache to become queryable instead of simply fetching the
sources.
|
|
|
|
|
| |
Wait for cache to become queryable instead of simply skipping the
element.
|
|
|
|
|
|
| |
Queue sometimes called ready() before skip() and sometimes it didn't.
This change consolidates the two methods to ensure we always determine
the correct status.
|
|
|
|
|
|
|
|
|
|
|
|
| |
The strict cache key is preferred when pulling or extracting artifacts
in non-strict mode, before falling back to the weak cache key. This
means that the strict cache key must be available before the cache is
queried.
The strict cache key is also sufficient for querying caches using strong
cache keys. Do not defer those cache queries.
Fixes #202
|
|
|
|
| |
Fixes: bc492fa8 ("Use explicit element state updates")
|
|
|
|
|
|
|
|
| |
TrackQueue must call _set_ref() for all sources, otherwise the source
will never get out of the inconsistent state. This fixes --track when
using local sources.
Fixes: 03412a9e ("Use explicit source state updates")
|
|
|
|
|
|
|
| |
Source.track() may return None when tracking is not available. Handle
this identical to the case where track() returns the current ref.
Fixes #201
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This is intended to make things easier to find for the
various types of people searching for stuff.
o The installation instructions remain on the main page.
o Three main separate pages have been created
- Using BuildStream
- Authoring BuildStream projects
- Core API reference, for plugin authors
o The "Authoring projects" section swallows the
previous plugin index; so one can find the plugin one
is looking for on the same page as the rest of the format
docs
o The plugin authoring section has been swallowed by the
core API reference section, with a note that this is useful
especially for plugin authors.
|
|
|
|
|
| |
This adds the _update_state() method to the Source class, similar to the
corresponding method in the Element class.
|
|
|
|
|
| |
_force_inconsistent is too low level. Keep that detail contained in the
Source class.
|
|
|
|
|
| |
Do not mark target element as inconsistent for `bst source-bundle` and
`bst workspace open` if tracking is disabled.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
Before:
[--:--:--] WARNING Failed to fetch remote refs from ssh://ostree@ostree.baserock.org:22000/cache: BuildStream did not connect successfully to the shared cache ssh://ostree@ostree.baserock.org:22000/cache: SSH error: ssh: connect to host ostree.baserock.org port 22000: Connection refused
After:
[--:--:--] WARNING Failed to fetch remote refs from ssh://ostree@ostree.baserock.org:22000/cache. ssh: connect to host ostree.baserock.org port 22000: Connection refused
|
|
|
|
|
| |
This does not introduce a performance gain alone, but importing pkg_resources
is costly at the moment and can be avoided until it is actually needed.
|
|
|
|
|
|
|
|
| |
A LocalPath was being passed into os.path.isabs, which failed when it
tried to call the path's startswith method.
Converting it to a string before using _yaml.dump seemed to solve the
problem.
|
|
|
|
|
|
|
| |
Do not recalculate the shared part of cache keys. In particular, do not
call get_unique_key() multiple times for a single element or source.
Fixes #167
|
|
|
|
|
|
|
|
|
|
| |
This adds the _update_state() method to the Element class to keep track
of element state and avoid calculating the same cache key multiple
times. This also consolidates the different get_cache_key
methods into a single method that always returns the cache key
calculated by _update_state(), if available.
Fixes #149, #173
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Originally this was created with also `bootstrap-commands` and
`test-commands` but these were never documented or used.
For `bootstrap-commands`, these originated in baserock but are
basically a part of the `configure-commands` stage, prepending
commands to `configure-commands` is more suitable here since we
now have the prepend/append list directives.
For `test-commands`, these were never used and it's unclear at
this time if it's the correct place for it. It would be interesting
to implement `test-commands` as a separate operation which can run
in parallel with reverse dependency builds (no need to block the
build of a reverse dependency on failing tests, we can still fail
the build as a whole based on a failing test without blocking
builds).
|
| |
|
| |
|
|
|
|
|
|
|
|
|
| |
We have a policy that subprocesses can return error messages, but not
actual exceptions.
In particular this allows GLib.Error exceptions to be returned, which
would otherwise be silently lost due to
https://gitlab.gnome.org/GNOME/pygobject/issues/145
|
|
|
|
|
| |
This reduces the differences between local and remote artifact
repositories, increasing code coverage of tests.
|
|
|
|
|
| |
Closing stdout will cause an exception when used with multiprocessing as
that calls sys.stdout.flush() as well.
|
|
|
|
|
|
| |
initializing cache
This fixes issue #141
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
If you have an artifact remote cache in your config using the ssh://
protocol and some SSH issue prevents us from connecting, we now show
the user the stderr output from the `ssh` program, e.g.
[--:--:--] WARNING Failed to fetch remote refs from ssh://artifacts@172.17.0.2:22200/artifacts/: BuildStream did not connect successfully to the shared cache ssh://artifacts@172.17.0.2:22200/artifacts/: Permission denied (publickey,keyboard-interactive).
Previously we would just show the error message raised by the protocol
code, which was less useful, e.g.:
[--:--:--] WARNING Failed to fetch remote refs from ssh://artifacts@172.17.0.2:22200/artifacts/: BuildStream did not connect successfully to the shared cache ssh://artifacts@172.17.0.2:22200/artifacts/: Expected reply, got none
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
This adds a new test for parsing artifact cache configuration, which
calls the helper function from the 'artifactcache' module directly
rather than trying to assert based on blind push and pull commands
whether or not we got the complex precedence rules exactly right.
This means frontend push/pull tests no longer need to be so thorough
about testing precedence but they are instead expanded to assert that
multiple caches work correctly.
|
|
|
|
|
|
|
|
|
|
| |
The initial multiple cache support patch implemented a rather fragile
logic where we would push to the first cache in the list that used the
ssh:// protocol, if any. If we implement push-over-https:// in future
then this will become totally unworkable.
This patch alters the logic so that each remote has a 'push' option,
and BuildStream will push to any remote that has 'push: true' set.
|
|
|
|
|
|
|
|
| |
This allows pushing and pulling from a specific cache, ignoring what is
configured.
If we choose to add a --remote option to `bst build` in future that
would now be simple to do.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This extends the 'artifacts' configuration block such that a list of
`url` mappings can be given instead of a single entry. For example:
artifacts:
- url: http://example.com/artifacts1
- url: ssh://ostree@example.com/artifacts2
The OSTreeCache class is updated to set up multiple remotes and query
remote refs from all of them.
There are no automated tests for this yet.
Empty URLs ('') now raise an exception. They cause breakages internally
if we allow them through, and they can only occur if the user or our
tests are misconfiguring things somehow.
We report failure to fetch from the cache by printing a message to
stderr for now. This is because BuildStream's actual logging
functionality can't be used during frontend init -- see issue #168.
|
|
|
|
|
| |
Minor tweak to hopefully make the test cases a bit shorter and more
readable, in preparation for adding more.
|
|
|
|
|
|
|
|
| |
Added app.print_error() to consistently print an error while
running the pipeline stuff.
Also make bst push & pull commands more consistent with
the rest, and print the summary at the end as expected.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
o Now print the message itself on the same line in place of the abbreviated log file
o When reading back the last lines of the log file, prefix that
with an informative message about whence the log lines come,
showing the full unabbreviated log file.
o Fixed the behavior of read_last_lines() so that it works as
intended, problems were:
- Was prepending an additional newline when the log contains
more lines than we want to print
- Was not printing anything at all in the case we want to
print more lines than exist in the log
|
| |
|
| |
|
|
|
|
|
|
|
| |
Now any BuildStream exception can potentially add detail to
the errors they raise.
Allow detail strings already in PluginError()
|
|
|
|
|
| |
Dont make the ERROR message explicitly anymore, leave that
to the frontend to sort out.
|
| |
|