summaryrefslogtreecommitdiff
path: root/src/buildstream/_loader/loader.py
Commit message (Collapse)AuthorAgeFilesLines
* Completely remove MetaElementTristan van Berkom2020-08-131-149/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This dramatically affects the load process and removes one hoop we had to jump through, which is the creation of the extra intermediate MetaElement objects. This allows us to more easily carry state discovered by the Loader over to the Element constructor, as we need not add additional state to the intermediate MetaElement for this. Instead we have the Element initializer understand the LoadElement directly. Summary of changes: * _loader/metaelement.py: Removed * _loader/loadelement.py: Added some attributes previously required on MetaElement * _loader/loader.py: Removed _collect_element() and collect_element_no_deps(), removing the process of Loader.load() which translates LoadElements into MetaElements completely. * _loader/init.py: Export LoadElement, Dependency and Symbol types, stop exporting MetaElement * _loader/metasource.py: Now take the 'first_pass' parameter as an argument * _artifactelement.py: Use a virtual LoadElement instead of a virtual MetaElement to instantiate the ArtifactElement objects. * _pluginfactory/elementfactory.py: Adjust to now take a LoadElement * _project.py: Adjust Project.create_element() to now take a LoadElement, and call the new Element._new_from_load_element() instead of the old Element._new_from_meta() function * element.py: - Now export Element._new_from_load_element() instead of Element._new_from_meta() - Adjust the constructor to do the LoadElement toplevel node parsing instead of expecting members on the MetaElement object - Added __load_sources() which parses out and creates MetaSource objects for the sake of instantiating the element's Source objects. Consequently this simplifies the scenario where workspaces are involved. * source.py: Adjusted to use the new `first_pass` parameter to MetaSource when creating a duplicate clone.
* _loader: Use only one Dependency() classTristan van Berkom2020-08-101-9/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This moves the Dependency() implementation from the _loader.types module into the _loader.loadelement module, replacing the duplicate Dependency() definition there. Instead of creating _loader.loadelement.Dependency objects on the fly during the load process, we simply fill in the new LoadElement member on the original Dependency objects. This refactor should make the code easier to work with, since any originally loaded state from the YAML is preserved in the intermediate LoadElement data model and does not need to be manually handed over to a separate object. Summary of changes: * _loader/types.pyx: Removed Dependency() implementation from here, and renamed to _loader/types.py * _loader/loadelement.py: Replaced the second Dependency object implementation with the original one from _loader/types.py, adding a new LoadElement member which is resolved during the load process via the new Dependency.set_element() API * _loader/loader.py: Instead of creating a second kind of Dependency object on the fly during the load process, simply resolve the existing dependencies with Dependency.set_element() * setup.py: Fixed up cython build instructions, now that _loader/types.py no longer contains any cython code.
* Extract ElementSources classjuerg/element-sourcesJürg Billeter2020-08-061-1/+1
| | | | An ElementSources object represents the combined sources of an element.
* loader.py: use the full name of the element when looking up a workspaceAbderrahim Kitouni2020-07-161-1/+1
| | | | | This fixes a bug where buildstream would ignore the opened workspace on a cross-junction element.
* _loader: Treat `link` elements like junctionsTristan van Berkom2020-07-151-2/+2
| | | | | | | link elements need to be treated the same way as junctions in the load process. This fixes issue #1359
* _loader: Add support for duplicate junctionsTristan van Berkom2020-06-241-15/+110
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This adds some measure of tolerance to duplicate projects by accepting cases where multiple projects are loaded but whitelisted to be loaded as duplicates by the project.conf. This change does the following: * _project.py: Add `junction_is_duplicated()` API and parse `duplicates` The project now maintains a table of junctions which are considered "duplicated" per project, and exposes an API for the loader to query whether a given loader is explicitly listed as a duplicate (allowing multiple "duplicated" instances of that project to coexist in the same pipeline). This new API searches for the passed Loader instance by way of it's project relative search paths specified in the duplicates. Using the `Loader.get_loader()` codepath ensures that any links get resolved and that any overrides along the way will be considered correctly while attempting to match the duplicate. * _loader/loader.py: Few changes here. - Allow Loader.get_loader() to operate in a mode which does not load subprojects. - Add new function Loader.ancestors(), this is a generator which yields every ancestor loader which is capable of addressing the given loader, including ancestors for which their junction definitions to this loader have been overridden. To this end, we now record `_alternative_parents` when searching for overrides. - Now implements __str__() in order to print an identifying string for a loader in error messages * _loader/loadcontext.py: Enhanced to now ask projects whether they are marked as duplicates in order to avoid erroring out on project collisions if those are explicitly marked to not be an error. We now also split out the new function LoadContext.assert_loaders(), which is to be called once at the end of loading the whole pipeline recursively. This reduces the complexity of searching the loaded graph while loading the graph, and also reports more complete errors (all instances of a conflicting project will be reported in the error at once).
* junctions: Replace coalescing with explicit overridesTristan van Berkom2020-06-231-40/+79
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch removes the functionality of coalescing junction configurations on junction element names, and replaces it with an explicit format which allows junction declarations to override junctions in subprojects with junction declarations from the local project. Changes: * plugins/elements/junction.py: Load the overrides dictionary * exceptions.py: Added new CIRCULAR_REFERENCE to LoadErrorReason. * _loader/loadcontext.py: Add new register_loader() function which is called by the Loader to register all loaders with the context, delegating the task of detecting conflicting junctions to the load context. * _loader/loader.py: When loading a junction, check if there is an override from the parent project and use that instead. Register with the LoadContext, and pass the LoadContext and provenance along when creating a Project Further, we now protect against circular element path references with a new _loader_search_provenances table. * _project.py: Pass new `provenance` member through the constructor and onwards into the child loader.
* _loader: Fix invalid junction error to include provenanceTristan van Berkom2020-06-191-3/+7
| | | | | | | | | | | | When a junction element has dependencies, we were not even specifying which junction element was guilty of having dependencies and only saying that such a thing would be forbidden (wherever it might be in your project). Fix this by using the provenance of the dependency in the junction, specifying exactly where the invalid dependency exists. This required carrying the provenance onto the other Dependency() object.
* _loader: Adding LoadContexttristan/load-contextTristan van Berkom2020-06-161-92/+49
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Instead of passing around many details though calling signatures throughout the loader code, create a single LoadContext object which holds any overall loading state along with any values which are constant to a full load process. Overall this patch does: * _frontend/app.py: No need to pass Stream.fetch_subprojects() along anymore * _loader/loadelement.pyx: collect_element_no_deps() no longer takes a task argument * _loader/loader.py: Now the Loader has a `load_context` member, and no more `_fetch_subprojects` member or `_context` members Further, `rewritable` and `ticker` is no longer passed along through all of the recursing calling signatures, and `ticker` itself is finally removed because this has been replaced a long time ago with `Task` API from `State`. * _pipeline.py: The load() function no longer has a `rewritable` parameter * _project.py: The Project() is responsible for creating the toplevel LoadContext() if one doesn't exist yet, and this is passed through to the Loader() (and also passed to the Project() constructor by the Loader() when instantiating subprojects). * _stream.py: The `Stream._fetch_subprojects()` is now private and set on the project when giving the Project to the Stream in `Stream.set_project()`, also the Stream() sets the `rewritable` state on the `LoadContext` at the earliest opportunity, as the `Stream()` is the one who decides this detail. Further, some double underscore private functions are now regular single underscores, there was no reason for this inconsistency. * tests/internals/loader.py: Updated for API change
* Completely abolish job pickling.tristan/nuke-pickle-jobberTristan van Berkom2020-06-151-30/+0
|
* _loader/loader.py: Make provenance a positional argumentTristan van Berkom2020-06-101-6/+4
| | | | | | This forces the `provenance` to `Loader.get_loader()` to be a mandatory argument, ensuring that there are never any callers which fail to provide provenance.
* _loader.py: Support full path dependencies and targetsTristan van Berkom2020-06-081-183/+214
| | | | | | | | | | | | | | | | | | | | | | This commit does the following: * Moves the `get_loader()` implementation to a private `_get_loader()` implementation again, without changing it. * Replaces the public `get_loader()` method with a simple one which parses the passed junction name and walks the `:` splits until the ultimate loader is found (by calling _get_loader() internally). * Removes an assertion in the Dependency constructor about having multiple `:` in the element name, which is now allowed, and adjusts the Dependency constructor to properly rsplit() the passed dependency target from the right, storing the full junction path in dep.junction and the target only in the dep.target. * Adds missing `provenance` argument to Loader._parse_name(), ensuring that the provenance of where a target name was found is propagated through `get_loader()` and `_load_file()` preserving the original provenance of a loaded element name.
* junctions: Remove all traces of the `target` optionTristan van Berkom2020-06-011-12/+0
| | | | | | | | | | | | | | | This removes: * The `target` feature from the junction plugin * Special case code in the loader for the `target` feature * The `target` related cases in tests/format/junctions.py This also adjusts the `target` related documentation in the `junction` element to suggest using a `link` element for the purpose of using a subproject junction configuration to access a common sub-subproject.
* link element: Adding support for new link elementTristan van Berkom2020-06-011-1/+35
| | | | | | This element acts as a symbolic link, it has no other configurations other than to specify the element (or junction) to which it refers to, either in the local project or in a subproject.
* _loader/loader.py: Make collect_element_no_deps() publicTristan van Berkom2020-06-011-75/+78
| | | | | Within the _loader module, the LoadElement will need to use this in order to resolve links.
* _loader/loader.py: Remove useless line of code.Tristan van Berkom2020-05-301-2/+1
| | | | | | | | | We were redundantly calling Loader._load_file() directly before calling Loader.get_loader(), even though Loader.get_loader() implicitly loads the file anyway. To pass tests, we needed to fix get_loader() to pass provenance along to Loader._load_file().
* loader: removing the NO_PROGRESS objecttristan/remove-loader-no-progress-objectTristan van Berkom2020-05-301-10/+5
| | | | | | | | | | | While adding a mock task object to track progress in some loader tests in commit 17144d84c2b63daf6e3aa9b42c6c773f134e8660, a new `_NO_PROGRESS` object was added as an explicit marker to denote that progress information should not be reported. This complicates the code, None should be a sufficient value for not reporting progress, while still permitting mock task objects to capture progress if desired.
* Revert "Schedule elements instead of "requiring" them"Jürg Billeter2020-05-271-0/+3
| | | | This reverts commit 14e32a34f67df754d9146efafe9686bfe6c91e50.
* _loader/loader.py: Reoganized public/private methodstristan/loader-api-fixupTristan van Berkom2020-05-241-195/+196
| | | | | | | | | | | | | * This file had `clean_caches()` documented as public but the function was actually private `_clean_caches()`: Moved this to the end of the class in the private section. * The `_get_loader()` was marked as private but is in fact public, and used by the project to load cross junction include files. This patch also updates `_includes.py` to call the public `get_loader()` method instead of sneaking in and calling the private `_get_loader()` method (while also removing one redundant line of code from the same function).
* src: Removing all pre 2.x "Since" documentation annotations.Tristan Van Berkom2020-04-211-1/+0
| | | | | | | | | | | | | | | This does not make sense to keep in the public API surface documentation. As we are heading towards a release of 2.0, this represents a "reset" in public API, and older annotations only serve to clutter the documentation with information that is not relevant to the reader. Everything which is public at the time of the 2.0 release can be considered available "Since: 2.0" implicitly (as this is going to be the starting point of this new stable API). It will make sense to start adding these annotations again for any added API in 2.2 and forward.
* Process options in includes files with the options of their junctionvalentindavid/include-options-from-junctionValentin David2020-04-151-2/+0
| | | | | | | | | Unfortunately the options from main project cannot always be processed in the include processing since project configuration might load option declarations from a separate file. For that reason the result of `Include.process` should still be passed through the option processor. But all options files included from junctioned are already evaluated.
* _workspaces: Rename `last_successful` to `last_build`Darius Makovsky2020-02-111-1/+1
| | | | | | The new incremental build approach uses the buildtree from the last build (successful or not) and no longer needs to know any information about the last successful build.
* exceptions: Expose ErrorDomain, ErrorLoadReasonThomas Coldrick2020-01-231-1/+2
| | | | | | | Plugin tests are already accessing this API, but using imports from private modules. For motivation for this to be exposed publicly, note that ErrorDomain is an argument for most things in runcli.py, and LoadErrorReason may be another.
* element.py: Remove _get_consistency and introduce explicit methodsBenjamin Schubert2020-01-161-9/+8
| | | | | | | This replaces the _get_consistency method by two methods: `_has_all_sources_resolved` and `_has_all_sources_cached` which allows a more fine grained control on what information is needed.
* element.py: Rename '_source_cached' to '_has_all_sources_in_source_cache'Benjamin Schubert2020-01-161-1/+1
| | | | | | '_source_cached' is not explicit enough as it doesn't distinguishes between sources in their respective caches and sources in the global sourcecache.
* loader.py: Optimize sorting of elements when they are multiple targetsbschubert/optimize-loading-multiple-targetsBenjamin Schubert2020-01-141-1/+5
| | | | | | | Currently, with multiple targets, we would go through all the elements in the target and sort them, even though they might already be sorted. This ensures we sort every element only once.
* Do not load the workspace.last_successful as the workspace plugin refDarius Makovsky2019-11-271-1/+1
| | | | | | Set the workspace plugin `__last_successful` attribute. loader: Use empty string as default when last_successful is missing
* Schedule elements instead of "requiring" themTristan Maat2019-11-221-3/+0
|
* element.py: Reduce calls to __update_cache_key_non_strictTristan Maat2019-11-181-2/+13
|
* Create _initialize_state() to capture an _update_state() use caseTristan Maat2019-11-181-5/+1
|
* Only run `element.__update_source_state` when necessaryTristan Maat2019-11-181-0/+4
|
* Reformat code using BlackChandan Singh2019-11-141-85/+113
| | | | | | | As discussed over the mailing list, reformat code using Black. This is a one-off change to reformat all our codebase. Moving forward, we shouldn't expect such blanket reformats. Rather, we expect each change to already comply with the Black formatting style.
* Remove unnecessary ignore_workspaces kwargDarius Makovsky2019-11-061-10/+7
| | | | | | Attempting to open a workspace for the same element without closing it now raises. This makes this kwarg unnecessary and tests should close workspaces between attempts to open.
* Remove special loading for workspacestraveltissues/notesDarius Makovsky2019-10-171-27/+20
| | | | | WorkspaceSource.init_workspace raises an exception so it is no longer necessary to retain the original source objects of the loaded element.
* Use workspace_close and workspace_open to reset workspacesDarius Makovsky2019-10-161-7/+10
| | | | | | | | | | | | | * tracking not needed in reset * support workspace opening for already open workspaces remove existing files to preserve behaviour Add ignore_workspaces kwarg to element loading via Stream().load Setting this to true will ignore special handling of sources for open workspaces and load the sources specified rather than a workspace source. This avoids having to reload elements when re-opening workspaces.
* Remove XXX comment about missing progressTristan Maat2019-10-101-2/+9
| | | | | | | | | | | | | This should be safe now - this particular point turned out to be involved in loading dependencies of junction elements, rather than anything in their projects. This meant that, yes, we were missing progress, however junction elements are not allowed to have dependencies in the first place, so we simply short-circuit their load and avoid the problem altogether. We also added more explicit progress opt-outs, since it's far too easy to end up with spurious Nones.
* testutils/context.py: Mock tasks instead of accepting NonesTristan Maat2019-10-101-4/+9
| | | | | | | | | | | | To ensure that we only disable element loading task progress reporting for very specific code paths, we need to teach the test suite to be a bit smarter. For this reason we now mock a _Task object and return it in our mock context's relevant method invocations. Other code paths that deliberately invoke the loader without task reporting now mark their loads with NO_PROGRESS.
* loader.py: Avoid loading deps of junction metaelementsTristan Maat2019-10-101-1/+19
| | | | | | | | | | | | | By avoiding this, loading metaelements of junctions becomes cheap even for junctions with erroneous dependencies, and we can ignore their task reporting. Task reporting for junction metaelement loading is confusing, since the junction element itself will never be part of the pipeline, so we'd rather not have this show up as an actual loaded element. Elements loaded from the junction are loaded separately, therefore this does not affect their progress display.
* loader.py: Remove workspace handling in element collectionDarius Makovsky2019-09-241-5/+1
|
* Remove workspace mounting for sandboxesDarius Makovsky2019-09-241-1/+1
| | | | | | | | Workspaces will be handled via the workspace source plugin methods. This does not currently support reflecting build artifacts in the open workspace. tests: Mark incremental workspace builds as xfail (strict)
* Support loading 'workspace'-kind sourcesDarius Makovsky2019-09-241-7/+28
| | | | | Add the workspace-kind source to the sources manually to prevent a user incorrectly loading the source plugin.
* Address no-else-raise pylint warningJames Ennis2019-09-061-10/+13
| | | | | | | | | | | In the latest released version of pylint (2.3.x), the "no-else-raise" (R1720) warning has been introduced. See: http://pylint.pycqa.org/en/latest/whatsnew/changelog.html#what-s-new-in-pylint-2-3-0 There are many instances of this in our codebase, which this patch addresses.
* _loader.py: Cache how many Elements we loadJames Ennis2019-09-041-0/+6
|
* Implement strict dependency semanticsTristan Van Berkom2019-08-311-2/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch allows specifying a dependency as `strict`, e.g.: build-depends: - filename: element.bst strict: true This allows finer tuning of projects which want to leverage the non-strict build mode; dependencies which are statically linked to, or who's content is otherwise included verbatim in the resulting output, should be marked `strict` to ensure these bits get reassembled if necessary when building in non-strict mode. This fixes #254 Change summary: o _loader/loadelement.pyx: Added 'strict' attribute to Dependency o _loader/types.pyx: Added 'strict' attribute to Dependency do the parsing work. o _loader/metaelement.py: Added 'strict_dependencies' list o _loader/loader.py: Resolve the 'strict_dependencies' list o element.py: Added __strict_dependencies list, and use this to conditionally use weak cache keys in place of names for the purpose of building the weak cache key (in the case of dependencies which are marked as strict).
* _message.py: Use element_name & element_key instead of unique_idtpollard/messageobjectTom Pollard2019-08-081-1/+1
| | | | | | | | | | | | | Adding the element full name and display key into all element related messages removes the need to look up the plugintable via a plugin unique_id just to retrieve the same values for logging and widget frontend display. Relying on plugintable state is also incompatible if the frontend will be running in a different process, as it will exist in multiple states. The element full name is now displayed instead of the unique_id, such as in the debugging widget. It is also displayed in place of 'name' (i.e including any junction prepend) to be more informative.
* Fix up of some spelling mistakesRebecca Grayson2019-07-261-1/+1
|
* Render progress information for loading and processing elementsJonathan Maw2019-07-261-7/+14
|
* loader: Move sort_dependencies to loadelement as a cython methodbschubert/optimize-loaderBenjamin Schubert2019-07-261-73/+2
|
* loadelement: Move loadelement to CythonBenjamin Schubert2019-07-261-6/+9
|
* loader: Don't pickle caches for child jobs, they are not needed thereDaniel Silverstone2019-07-261-0/+5
| | | | Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
View Lorry Controller Status