| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
| |
Call directly the relevant methods from the stream to the scheduler
|
| |
|
|
|
| |
This removes all notifications left coming from the scheduler, and
replaces them by callbacks
|
| |
|
|
|
| |
The stream is itself calling the `run` method on the scheduler, we don't
need another indirection
|
| |
|
|
|
| |
We are calling the scheduler, and it returning correctly already tells
us this.
|
| |
|
|
|
| |
We are calling the scheduler, and it returning correctly already tells
us this.
|
| |
|
|
|
| |
Stop using 'Notifications' for retries, the state is the one handling
the callbacks required for every change in status of elements
|
| |
|
|
|
| |
This moves all implementations of 'start_time' into a single place
for easier handling and removing roundtrips of notifications
|
| |
|
|
|
| |
The State is the interface between both, there is no need to do multiple
round-trips to handle such notifications
|
| |
|
|
|
| |
The messenger should be the one receiving messages directly, we don't
need this indirection
|
| |\
| |
| |
| |
| | |
All: Update all references to the mailing list.
See merge request BuildStream/buildstream!1980
|
| |/
|
|
|
|
| |
Update any references to the mailing list to now point to:
https://lists.apache.org/list.html?dev@buildstream.apache.org
|
| |\
| |
| |
| |
| | |
_artifactcache.py: Skip push only if server has identical artifact
See merge request BuildStream/buildstream!1976
|
| | |
| |
| |
| |
| | |
This is a regression test to skip push only if server has identical
artifact.
|
| |/
|
|
|
|
|
|
|
|
|
|
| |
The existing artifact on a server may refer to a previous build and the
referenced blobs may no longer exist. As we push blobs referenced by the
local artifact proto, we can skip pushing the artifact proto only if the
artifact proto on the server is identical.
It would be more efficient to always push the artifact proto without
checking what artifact proto is currently on the server. However, we
need the extra information to mark push jobs as skipped. This is clearer
to users and also required by some test cases.
|
| |\
| |
| |
| |
| | |
bst-artifact-server: Fix error handling for requests forwarded to casd
See merge request BuildStream/buildstream!1972
|
| | | |
|
| |/ |
|
| |\
| |
| |
| |
| |
| |
| | |
Explicit control of junction configuration overrides
Closes #1326
See merge request BuildStream/buildstream!1901
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This commit:
* Redocuments the `junction.py` element almost in it's entirity
* Describes the `overrides` feature
* Describes how to use `link` elements to derive junction configuration
from subprojects.
* Adds project.conf documentation for junction `duplicates`
* Adds project.conf documentation for junction `internal`
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This commit adds support for the `internal` list of the project's
`junctions` configuration section.
_loader/loadcontext.py is updated with this commit to consider
the internal markers and avoid raising errors for collisions with
internal junctions, as well as being enhanced to list internal
junctions in any conflicting junction error messages.
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This adds some measure of tolerance to duplicate projects by
accepting cases where multiple projects are loaded but whitelisted
to be loaded as duplicates by the project.conf.
This change does the following:
* _project.py: Add `junction_is_duplicated()` API and parse `duplicates`
The project now maintains a table of junctions which are considered
"duplicated" per project, and exposes an API for the loader to query
whether a given loader is explicitly listed as a duplicate (allowing
multiple "duplicated" instances of that project to coexist in the same
pipeline).
This new API searches for the passed Loader instance by way of it's
project relative search paths specified in the duplicates. Using the
`Loader.get_loader()` codepath ensures that any links get resolved
and that any overrides along the way will be considered correctly
while attempting to match the duplicate.
* _loader/loader.py: Few changes here.
- Allow Loader.get_loader() to operate in a mode which does not
load subprojects.
- Add new function Loader.ancestors(), this is a generator which yields
every ancestor loader which is capable of addressing the given loader,
including ancestors for which their junction definitions to this loader
have been overridden.
To this end, we now record `_alternative_parents` when searching for
overrides.
- Now implements __str__() in order to print an identifying string
for a loader in error messages
* _loader/loadcontext.py: Enhanced to now ask projects whether they are
marked as duplicates in order to avoid erroring out on project collisions
if those are explicitly marked to not be an error.
We now also split out the new function LoadContext.assert_loaders(), which
is to be called once at the end of loading the whole pipeline recursively.
This reduces the complexity of searching the loaded graph while loading
the graph, and also reports more complete errors (all instances of a
conflicting project will be reported in the error at once).
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
* Test various scenarios of overriding junctions, including
deep paths as junctions to override, and as junctions to use
to override.
* Test conflicting junction configurations, ensuring that we
report both provenances of where the junctions were declared.
* Test circular references in element paths while declaring overrides,
for instance when trying to override a subproject using a deeper
definition of the same subproject.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This patch removes the functionality of coalescing junction
configurations on junction element names, and replaces it with
an explicit format which allows junction declarations to override
junctions in subprojects with junction declarations from the local
project.
Changes:
* plugins/elements/junction.py: Load the overrides dictionary
* exceptions.py: Added new CIRCULAR_REFERENCE to LoadErrorReason.
* _loader/loadcontext.py: Add new register_loader() function which
is called by the Loader to register all loaders with the context,
delegating the task of detecting conflicting junctions to the
load context.
* _loader/loader.py: When loading a junction, check if there is
an override from the parent project and use that instead.
Register with the LoadContext, and pass the LoadContext
and provenance along when creating a Project
Further, we now protect against circular element path references
with a new _loader_search_provenances table.
* _project.py: Pass new `provenance` member through the constructor
and onwards into the child loader.
|
| | | |
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This commit refactors the junctions test to use more parameterization
and to remove copy_subproject(), using statically committed data as
much as possible.
This is because copy_subproject() causes data to be shared among tests
in such a way that when such data get's modified, it easily causes
unintended side effects on adjacent test cases, better to keep this
data separate.
Overview of changes:
* Remove copy_subproject()
* Split up test data into case specific directories, sometimes
reusing a directory among various related tests
* Use @pytest.mark.parameterize() as much as possible for better
coverage and more clearly expressed test cases
* Adds update_project() to modify a project.conf inline, as is
done in some other tests like tests/plugins/loading.py
* Removes tests related to junction name coalescing, this feature
will be removed later in this branch and other tests related
to junction overrides will replace these.
* Removes some redundant tests
* Removes a comment about how junction name coalescing can cause
errors when trying to open a junction but the project.conf is
missing. We continue to test missing project.conf files in a
variety of scenarios, but the comment will be rendered irrelevant
with the removal of junction name coalescing.
* Change the git related tests to use tar instead, this serves
the same purpose, but tar will remain a core plugin in BuildStream 2.
|
| |/
|
|
|
|
|
|
|
|
|
|
| |
When a junction element has dependencies, we were not even specifying
which junction element was guilty of having dependencies and only
saying that such a thing would be forbidden (wherever it might be in
your project).
Fix this by using the provenance of the dependency in the junction,
specifying exactly where the invalid dependency exists.
This required carrying the provenance onto the other Dependency() object.
|
| |\
| |
| |
| |
| | |
Small cleanups
See merge request BuildStream/buildstream!1966
|
| | |
| |
| |
| |
| | |
Otherwise some of BuildStream's config will fail and it is therefore
impossible to just run `pytest tests/`
|
| |/ |
|
| |\
| |
| |
| |
| | |
_loader: Adding LoadContext
See merge request BuildStream/buildstream!1964
|
| |/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Instead of passing around many details though calling signatures
throughout the loader code, create a single LoadContext object
which holds any overall loading state along with any values which
are constant to a full load process.
Overall this patch does:
* _frontend/app.py: No need to pass Stream.fetch_subprojects() along anymore
* _loader/loadelement.pyx: collect_element_no_deps() no longer takes a task argument
* _loader/loader.py: Now the Loader has a `load_context` member, and no more
`_fetch_subprojects` member or `_context` members
Further, `rewritable` and `ticker` is no longer passed along through all
of the recursing calling signatures, and `ticker` itself is finally removed
because this has been replaced a long time ago with `Task` API from `State`.
* _pipeline.py: The load() function no longer has a `rewritable` parameter
* _project.py: The Project() is responsible for creating the toplevel
LoadContext() if one doesn't exist yet, and this is passed through
to the Loader() (and also passed to the Project() constructor by the
Loader() when instantiating subprojects).
* _stream.py: The `Stream._fetch_subprojects()` is now private and set
on the project when giving the Project to the Stream in `Stream.set_project()`,
also the Stream() sets the `rewritable` state on the `LoadContext` at the
earliest opportunity, as the `Stream()` is the one who decides this detail.
Further, some double underscore private functions are now regular single
underscores, there was no reason for this inconsistency.
* tests/internals/loader.py: Updated for API change
|
| |\
| |
| |
| |
| | |
Completely abolish job pickling.
See merge request BuildStream/buildstream!1965
|
| |/ |
|
| | |
|
| |\
| |
| |
| |
| | |
Remove a not very important message from the cli
See merge request BuildStream/buildstream!1963
|
| | |
| |
| |
| |
| |
| | |
Add some comments clearly separating public and private methods,
and moving the _validate_node() private method into the private
section, renaming it _validate_toplevel_node() for better clarity.
|
| | |
| |
| |
| | |
This is no longer needed.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This API was only used to issue a not very useful message in the CLI
when removing a workspace the user happened to have invoked BuildStream
from.
The API also didn't really make any sense, being documented as:
"Checks whether the workspace belonging to element_name is
required to load the project"
This in fact meant whether having that workspace open was required
for BuildStream to be invoked, due to the metadata encoded into the
workspace directory.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| | |
When closing a workspace while the `workspace close` command was issued
from the closed workspace, we go to great lengths to inform the user
that they can no longer run bst commands from the closed workspace directory.
This is not worthwhile, the user will know this the next time they run
`bst` anyway.
|
| |/
|
|
|
|
| |
Remove tests which check for a user message to be issued upon closing
a workspace who's metadata was used to launch BuildStream and find
the BuildStream project directory.
|
| |\
| |
| |
| |
| | |
Reorginize some __init__() functions
See merge request BuildStream/buildstream!1961
|
| | |
| |
| |
| |
| |
| |
| |
| | |
Put the private members in the private section and
the public members in public section.
The exception is `_context` which is resolved early, and
this is required by other code which runs in __init__().
|
| |/
|
|
| |
Instead of mixing it up with the directories
|
| |\
| |
| |
| |
| | |
.gitlab-ci.yml: Update BuildBox components to 0.0.9
See merge request BuildStream/buildstream!1962
|
| |/ |
|
| |\
| |
| |
| |
| | |
cascache.py: Fix file modes in checkout()
See merge request BuildStream/buildstream!1959
|
| | |
| |
| |
| |
| | |
Do not copy file mode from casd object file. Do not change
non-executable mode bits if file is executable.
|
