summaryrefslogtreecommitdiff
path: root/src/buildstream/_loader/loader.py
Commit message (Collapse)AuthorAgeFilesLines
* loader.py: Optimize sorting of elements when they are multiple targetsbschubert/optimize-loading-multiple-targetsBenjamin Schubert2020-01-141-1/+5
| | | | | | | Currently, with multiple targets, we would go through all the elements in the target and sort them, even though they might already be sorted. This ensures we sort every element only once.
* Do not load the workspace.last_successful as the workspace plugin refDarius Makovsky2019-11-271-1/+1
| | | | | | Set the workspace plugin `__last_successful` attribute. loader: Use empty string as default when last_successful is missing
* Schedule elements instead of "requiring" themTristan Maat2019-11-221-3/+0
|
* element.py: Reduce calls to __update_cache_key_non_strictTristan Maat2019-11-181-2/+13
|
* Create _initialize_state() to capture an _update_state() use caseTristan Maat2019-11-181-5/+1
|
* Only run `element.__update_source_state` when necessaryTristan Maat2019-11-181-0/+4
|
* Reformat code using BlackChandan Singh2019-11-141-85/+113
| | | | | | | As discussed over the mailing list, reformat code using Black. This is a one-off change to reformat all our codebase. Moving forward, we shouldn't expect such blanket reformats. Rather, we expect each change to already comply with the Black formatting style.
* Remove unnecessary ignore_workspaces kwargDarius Makovsky2019-11-061-10/+7
| | | | | | Attempting to open a workspace for the same element without closing it now raises. This makes this kwarg unnecessary and tests should close workspaces between attempts to open.
* Remove special loading for workspacestraveltissues/notesDarius Makovsky2019-10-171-27/+20
| | | | | WorkspaceSource.init_workspace raises an exception so it is no longer necessary to retain the original source objects of the loaded element.
* Use workspace_close and workspace_open to reset workspacesDarius Makovsky2019-10-161-7/+10
| | | | | | | | | | | | | * tracking not needed in reset * support workspace opening for already open workspaces remove existing files to preserve behaviour Add ignore_workspaces kwarg to element loading via Stream().load Setting this to true will ignore special handling of sources for open workspaces and load the sources specified rather than a workspace source. This avoids having to reload elements when re-opening workspaces.
* Remove XXX comment about missing progressTristan Maat2019-10-101-2/+9
| | | | | | | | | | | | | This should be safe now - this particular point turned out to be involved in loading dependencies of junction elements, rather than anything in their projects. This meant that, yes, we were missing progress, however junction elements are not allowed to have dependencies in the first place, so we simply short-circuit their load and avoid the problem altogether. We also added more explicit progress opt-outs, since it's far too easy to end up with spurious Nones.
* testutils/context.py: Mock tasks instead of accepting NonesTristan Maat2019-10-101-4/+9
| | | | | | | | | | | | To ensure that we only disable element loading task progress reporting for very specific code paths, we need to teach the test suite to be a bit smarter. For this reason we now mock a _Task object and return it in our mock context's relevant method invocations. Other code paths that deliberately invoke the loader without task reporting now mark their loads with NO_PROGRESS.
* loader.py: Avoid loading deps of junction metaelementsTristan Maat2019-10-101-1/+19
| | | | | | | | | | | | | By avoiding this, loading metaelements of junctions becomes cheap even for junctions with erroneous dependencies, and we can ignore their task reporting. Task reporting for junction metaelement loading is confusing, since the junction element itself will never be part of the pipeline, so we'd rather not have this show up as an actual loaded element. Elements loaded from the junction are loaded separately, therefore this does not affect their progress display.
* loader.py: Remove workspace handling in element collectionDarius Makovsky2019-09-241-5/+1
|
* Remove workspace mounting for sandboxesDarius Makovsky2019-09-241-1/+1
| | | | | | | | Workspaces will be handled via the workspace source plugin methods. This does not currently support reflecting build artifacts in the open workspace. tests: Mark incremental workspace builds as xfail (strict)
* Support loading 'workspace'-kind sourcesDarius Makovsky2019-09-241-7/+28
| | | | | Add the workspace-kind source to the sources manually to prevent a user incorrectly loading the source plugin.
* Address no-else-raise pylint warningJames Ennis2019-09-061-10/+13
| | | | | | | | | | | In the latest released version of pylint (2.3.x), the "no-else-raise" (R1720) warning has been introduced. See: http://pylint.pycqa.org/en/latest/whatsnew/changelog.html#what-s-new-in-pylint-2-3-0 There are many instances of this in our codebase, which this patch addresses.
* _loader.py: Cache how many Elements we loadJames Ennis2019-09-041-0/+6
|
* Implement strict dependency semanticsTristan Van Berkom2019-08-311-2/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch allows specifying a dependency as `strict`, e.g.: build-depends: - filename: element.bst strict: true This allows finer tuning of projects which want to leverage the non-strict build mode; dependencies which are statically linked to, or who's content is otherwise included verbatim in the resulting output, should be marked `strict` to ensure these bits get reassembled if necessary when building in non-strict mode. This fixes #254 Change summary: o _loader/loadelement.pyx: Added 'strict' attribute to Dependency o _loader/types.pyx: Added 'strict' attribute to Dependency do the parsing work. o _loader/metaelement.py: Added 'strict_dependencies' list o _loader/loader.py: Resolve the 'strict_dependencies' list o element.py: Added __strict_dependencies list, and use this to conditionally use weak cache keys in place of names for the purpose of building the weak cache key (in the case of dependencies which are marked as strict).
* _message.py: Use element_name & element_key instead of unique_idtpollard/messageobjectTom Pollard2019-08-081-1/+1
| | | | | | | | | | | | | Adding the element full name and display key into all element related messages removes the need to look up the plugintable via a plugin unique_id just to retrieve the same values for logging and widget frontend display. Relying on plugintable state is also incompatible if the frontend will be running in a different process, as it will exist in multiple states. The element full name is now displayed instead of the unique_id, such as in the debugging widget. It is also displayed in place of 'name' (i.e including any junction prepend) to be more informative.
* Fix up of some spelling mistakesRebecca Grayson2019-07-261-1/+1
|
* Render progress information for loading and processing elementsJonathan Maw2019-07-261-7/+14
|
* loader: Move sort_dependencies to loadelement as a cython methodbschubert/optimize-loaderBenjamin Schubert2019-07-261-73/+2
|
* loadelement: Move loadelement to CythonBenjamin Schubert2019-07-261-6/+9
|
* loader: Don't pickle caches for child jobs, they are not needed thereDaniel Silverstone2019-07-261-0/+5
| | | | Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
* Make ChildJobs and friends picklableAngelos Evripiotis2019-07-241-0/+25
| | | | | | | | | Pave the way toward supporting the 'spawn' method of creating jobs, by adding support for pickling ChildJobs. Introduce a new 'jobpickler' module that provides an entrypoint for this functionality. This also makes replays of jobs possible, which has made the debugging of plugins much easier for me.
* _exceptions.py: Align LoadError() parameter orderingtpollard/loaderrorTom Pollard2019-07-181-25/+19
| | | | | | All of the errors which subclass from BstError have their first positional argument as message, LoadError should follow this ordering for consistency.
* node: Add 'get_str_list' on 'MappingNode'Benjamin Schubert2019-07-171-1/+1
| | | | | | `mapping.get_sequence(...).as_str_list()` is a very common pattern seen both in plugins and the core. Adding a helper to reduce the number of operations will make usage smoother
* _pipeline.py, loader.py: Move Element._preflight() to new_from_meta()James Ennis2019-07-161-1/+0
| | | | | | | It's essential to call preflight() when loading/resolving the Elements. This patch moves the preflight call to new_from_meta, so that it is called as soon as the Element is created. This avoids the need for having multiple callsites.
* _yaml: Split Node-related parts into 'node.pyx'Benjamin Schubert2019-07-151-1/+2
| | | | | This makes the 'Node' API public, and available for use directly for plugins.
* _yaml: Remove 'node_get_provenance' and add 'Node.get_provenance'Benjamin Schubert2019-07-151-1/+1
| | | | | | | | This replaces the helper method by adding a 'get_provenance' on the node directly - Adapt all call sites - Delay getting provenance wherever possible without major refactor
* _yaml: Create 'from_dict' on Node and remove node creation methodsBenjamin Schubert2019-07-151-1/+1
| | | | | | | Using 'Node.from_dict({})' can replace new_empty_node, and the rest is not needed anymore. - Adapt all call sites
* _yaml: Remove 'node_del' and support `del mapping[key]`Benjamin Schubert2019-07-151-2/+2
| | | | | - Also add a convenience method 'safe_del' catching the exception when we don't care if the value was there or not.
* _yaml: Remove 'node_get' and migrate all remaining calls to new APIBenjamin Schubert2019-07-151-2/+0
|
* _yaml: Introduce 'get_sequence()' and 'sequence_at()'/'mapping_at()'Benjamin Schubert2019-07-151-5/+3
| | | | | | | | | - Adding 'get_sequence' on MappingNode to access sequences in a mapping - Adding 'sequence_at' on SequenceNode to access sequences in a sequence - Adding 'mapping_at' on SequenceNode to access mappings in a sequence Using "*_at" in sequences allows us to quickly understand if we are dealing with a sequence or a mapping.
* _yaml: Add 'as_str()' on ScalarNode and 'get_scalar()' on MappingNodeBenjamin Schubert2019-07-151-5/+5
| | | | | | | | | | - 'get_scalar()' allows retrieving a scalar node from a mapping. - 'as_str()' casts a ScalarNode into a string (thus removing the node information). Both together, those replace 'node_get(mapping, key, type=str)' but also allow retrieving the 'Node' itself, which will allow in the future lazier provenance computation.
* _yaml: add 'get_mapping()' to MappingNodeBenjamin Schubert2019-07-151-5/+5
| | | | | | | | This allows to get a mapping node from another 'MappingNode', replacing 'node_get(my_mapping, key, type=dict)' Also changes all places where 'node_get' was called like that by the new API.
* Refactor, use context.messenger directlyAngelos Evripiotis2019-07-051-1/+1
| | | | | | Instead of having methods in Context forward calls on to the Messenger, have folks call the Messenger directly. Remove the forwarding methods in Context.
* loader.py: Check source cache for subproject fetchingJürg Billeter2019-07-031-1/+1
| | | | | | | | If the junction source consistency is cached but the junction is not in the source cache, also use Stream.fetch_subprojects() to import sources into source cache instead of doing this implicitly in Element._stage_sources_at(). This is in preparation for buildbox-casd, which will disallow local CAS cache writes from the main process (gRPC).
* Handle subproject fetching in the Stream classJürg Billeter2019-06-251-3/+6
|
* Always fetch subprojects as neededJürg Billeter2019-06-251-34/+17
| | | | | | Treat junction element sources the same as sources of any other element and always fetch subprojects as needed. Do not ask the user to manually fetch subprojects.
* _loader: Move extract_depends_from_node from loadelement to typesBenjamin Schubert2019-06-071-4/+4
| | | | | | | | | | | extract_depends_from_node is only depending on things declared in types, loadelement doesn't have any dependency on it. It makes more sense to have it in types.pyx. Moreover, this allows us to cythonize the function, reducing its total runtime impact. - _yaml: expose node_del as public api for Cython
* _loader/loader: cythonize valid_chars_nameBenjamin Schubert2019-06-051-31/+2
| | | | | | | | | - Create a new _loader/utils.pyx cython module for functions cythonized in the loader module. - Move valid_chars_name from loader to utils and cythonize. This function is called extensively, and easy to extract
* _yaml.pyx: Forbid expected_type=Mapping, and remove isinstance checkBenjamin Schubert2019-06-031-7/+6
| | | | | | | | | | | Calls to `isinstance` can be particularily costly. Using type() is much faster. The only known case where the `isinstance` was useful is for dictionnaries where we would ask for a 'Mapping' instead. Disallowing 'Mapping' for expected_type considerably speeds up the calls to this functions. Also add into NEWS
* _loader/loade{r,lement}.py: Do not re-collect already collected elementsdanielsilverstone-ct/iterative-loader-bitsDaniel Silverstone2019-05-301-0/+7
| | | | | | | When multiple top level elements are specified we need to keep track of whether we have completed the iterative collection process. Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
* loader.py: Make _collect_element() iterativeDaniel Silverstone2019-05-301-14/+41
| | | | | | | To reduce stack usage during load, make the LoadElement to MetaElement conversion be iterative. Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
* loader.py: Make _sort_dependencies() a static iterative methodDaniel Silverstone2019-05-301-10/+13
| | | | | | | | The _sort_dependencies() method does not need to be an instance method, nor does it need to be recursive. This fixes both of those things which can get us closer to being able to cythonize the loader. Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
* loader.py: Rewrite _check_circular_deps() to be iterativeDaniel Silverstone2019-05-301-34/+38
| | | | | | | | In an effort to reduce the places where the stack might be a problem as we Cythonize things, rewrite the circular dependency checker to be iterative. Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
* loader.py: Make _check_circular_deps() staticDaniel Silverstone2019-05-301-2/+3
| | | | | | | | This did not need to be an instance method, making it static might improve performance and definitely makes it clear that it's not actually bound to the loader instances. Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
* loader: Reduce complexity of the _load_file functionBenjamin Schubert2019-05-291-27/+22
| | | | | Delegate properly everything concerning a sub-loader to the sub-loader Stop iterating twice the loop for every dependency
View Lorry Controller Status