| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
| |
WorkspaceSource.init_workspace raises an exception so it is no longer
necessary to retain the original source objects of the loaded element.
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* tracking not needed in reset
* support workspace opening for already open workspaces
remove existing files to preserve behaviour
Add ignore_workspaces kwarg to element loading via Stream().load
Setting this to true will ignore special handling of sources for open
workspaces and load the sources specified rather than a workspace
source. This avoids having to reload elements when re-opening
workspaces.
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This should be safe now - this particular point turned out to be
involved in loading dependencies of junction elements, rather than
anything in their projects.
This meant that, yes, we were missing progress, however junction
elements are not allowed to have dependencies in the first place, so
we simply short-circuit their load and avoid the problem altogether.
We also added more explicit progress opt-outs, since it's far too easy
to end up with spurious Nones.
|
|
|
|
|
|
|
|
|
|
|
|
| |
To ensure that we only disable element loading task progress reporting
for very specific code paths, we need to teach the test suite to be a
bit smarter.
For this reason we now mock a _Task object and return it in our mock
context's relevant method invocations.
Other code paths that deliberately invoke the loader without task
reporting now mark their loads with NO_PROGRESS.
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
By avoiding this, loading metaelements of junctions becomes cheap even
for junctions with erroneous dependencies, and we can ignore their
task reporting.
Task reporting for junction metaelement loading is confusing, since
the junction element itself will never be part of the pipeline, so
we'd rather not have this show up as an actual loaded element.
Elements loaded from the junction are loaded separately, therefore
this does not affect their progress display.
|
| |
|
|
|
|
|
|
|
|
| |
Workspaces will be handled via the workspace source plugin methods. This
does not currently support reflecting build artifacts in the open
workspace.
tests: Mark incremental workspace builds as xfail (strict)
|
|
|
|
|
| |
Add the workspace-kind source to the sources manually to prevent a user
incorrectly loading the source plugin.
|
|
|
|
|
|
|
|
|
|
|
| |
In the latest released version of pylint (2.3.x), the
"no-else-raise" (R1720) warning has been introduced.
See:
http://pylint.pycqa.org/en/latest/whatsnew/changelog.html#what-s-new-in-pylint-2-3-0
There are many instances of this in our codebase, which this
patch addresses.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This patch allows specifying a dependency as `strict`, e.g.:
build-depends:
- filename: element.bst
strict: true
This allows finer tuning of projects which want to leverage
the non-strict build mode; dependencies which are statically
linked to, or who's content is otherwise included verbatim in
the resulting output, should be marked `strict` to ensure these
bits get reassembled if necessary when building in non-strict
mode.
This fixes #254
Change summary:
o _loader/loadelement.pyx: Added 'strict' attribute to Dependency
o _loader/types.pyx: Added 'strict' attribute to Dependency
do the parsing work.
o _loader/metaelement.py: Added 'strict_dependencies' list
o _loader/loader.py: Resolve the 'strict_dependencies' list
o element.py: Added __strict_dependencies list, and use this
to conditionally use weak cache keys in place of names for
the purpose of building the weak cache key (in the case of
dependencies which are marked as strict).
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Adding the element full name and display key into all element related
messages removes the need to look up the plugintable via a plugin
unique_id just to retrieve the same values for logging and widget
frontend display. Relying on plugintable state is also incompatible
if the frontend will be running in a different process, as it will
exist in multiple states.
The element full name is now displayed instead of the unique_id,
such as in the debugging widget. It is also displayed in place of
'name' (i.e including any junction prepend) to be more informative.
|
| |
|
| |
|
| |
|
| |
|
|
|
|
| |
Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
|
|
|
|
|
|
|
|
|
| |
Pave the way toward supporting the 'spawn' method of creating jobs, by
adding support for pickling ChildJobs. Introduce a new 'jobpickler'
module that provides an entrypoint for this functionality.
This also makes replays of jobs possible, which has made the debugging
of plugins much easier for me.
|
|
|
|
|
|
| |
All of the errors which subclass from BstError have their first
positional argument as message, LoadError should follow this
ordering for consistency.
|
|
|
|
|
|
| |
`mapping.get_sequence(...).as_str_list()` is a very common
pattern seen both in plugins and the core. Adding a helper to reduce
the number of operations will make usage smoother
|
|
|
|
|
|
|
| |
It's essential to call preflight() when loading/resolving the Elements.
This patch moves the preflight call to new_from_meta, so that it is
called as soon as the Element is created. This avoids the need for
having multiple callsites.
|
|
|
|
|
| |
This makes the 'Node' API public, and available for use directly for
plugins.
|
|
|
|
|
|
|
|
| |
This replaces the helper method by adding a 'get_provenance' on the node
directly
- Adapt all call sites
- Delay getting provenance wherever possible without major refactor
|
|
|
|
|
|
|
| |
Using 'Node.from_dict({})' can replace new_empty_node, and the rest
is not needed anymore.
- Adapt all call sites
|
|
|
|
|
| |
- Also add a convenience method 'safe_del' catching the exception
when we don't care if the value was there or not.
|
| |
|
|
|
|
|
|
|
|
|
| |
- Adding 'get_sequence' on MappingNode to access sequences in a mapping
- Adding 'sequence_at' on SequenceNode to access sequences in a sequence
- Adding 'mapping_at' on SequenceNode to access mappings in a sequence
Using "*_at" in sequences allows us to quickly understand if we are
dealing with a sequence or a mapping.
|
|
|
|
|
|
|
|
|
|
| |
- 'get_scalar()' allows retrieving a scalar node from a mapping.
- 'as_str()' casts a ScalarNode into a string (thus removing the node
information).
Both together, those replace 'node_get(mapping, key, type=str)' but
also allow retrieving the 'Node' itself, which will allow in the future
lazier provenance computation.
|
|
|
|
|
|
|
|
| |
This allows to get a mapping node from another 'MappingNode',
replacing 'node_get(my_mapping, key, type=dict)'
Also changes all places where 'node_get' was called like that by
the new API.
|
|
|
|
|
|
| |
Instead of having methods in Context forward calls on to the Messenger,
have folks call the Messenger directly. Remove the forwarding methods in
Context.
|
|
|
|
|
|
|
|
| |
If the junction source consistency is cached but the junction is not in
the source cache, also use Stream.fetch_subprojects() to import sources
into source cache instead of doing this implicitly in
Element._stage_sources_at(). This is in preparation for buildbox-casd,
which will disallow local CAS cache writes from the main process (gRPC).
|
| |
|
|
|
|
|
|
| |
Treat junction element sources the same as sources of any other element
and always fetch subprojects as needed. Do not ask the user to manually
fetch subprojects.
|
|
|
|
|
|
|
|
|
|
|
| |
extract_depends_from_node is only depending on things declared in types,
loadelement doesn't have any dependency on it. It makes more sense to
have it in types.pyx.
Moreover, this allows us to cythonize the function, reducing its total
runtime impact.
- _yaml: expose node_del as public api for Cython
|
|
|
|
|
|
|
|
|
| |
- Create a new _loader/utils.pyx cython module for functions cythonized
in the loader module.
- Move valid_chars_name from loader to utils and cythonize.
This function is called extensively, and easy to extract
|
|
|
|
|
|
|
|
|
|
|
| |
Calls to `isinstance` can be particularily costly. Using type() is
much faster. The only known case where the `isinstance` was useful
is for dictionnaries where we would ask for a 'Mapping' instead.
Disallowing 'Mapping' for expected_type considerably speeds up the
calls to this functions.
Also add into NEWS
|
|
|
|
|
|
|
| |
When multiple top level elements are specified we need to keep track
of whether we have completed the iterative collection process.
Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
|
|
|
|
|
|
|
| |
To reduce stack usage during load, make the LoadElement to MetaElement
conversion be iterative.
Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
|
|
|
|
|
|
|
|
| |
The _sort_dependencies() method does not need to be an instance method,
nor does it need to be recursive. This fixes both of those things which
can get us closer to being able to cythonize the loader.
Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
|
|
|
|
|
|
|
|
| |
In an effort to reduce the places where the stack might be a problem
as we Cythonize things, rewrite the circular dependency checker to
be iterative.
Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
|
|
|
|
|
|
|
|
| |
This did not need to be an instance method, making it static might
improve performance and definitely makes it clear that it's not
actually bound to the loader instances.
Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
|
|
|
|
|
| |
Delegate properly everything concerning a sub-loader to the sub-loader
Stop iterating twice the loop for every dependency
|
|
|
|
|
|
|
|
|
|
| |
In order to reduce the cost of recursion in loading a project,
this commit makes a semi-iterative treatment of the _load_file()
pathway. It's only semi-iterative in the sense that whenever we
cross a loader boundary (junctions) we recurse. This should mean
that high-depth projects load more safely.
Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
|
|
|
|
|
|
|
| |
This is the canonical way to fetch sources, and now leverages
the source cache if possible, better to do this rather than
poking around into Element.sources() directly and reimplementing
fetching in the loader.
|
|
|
|
|
|
|
|
|
|
| |
Instead of looping over the sources directly, we should be using
the canonical way of checking element consistency.
This requires an additional call to Element._update_state() on the
freshly loaded junction element in order to ensure we calculate consistency.
This fixes issue #1030.
|
|
This was discussed in #1008.
Fixes #1009.
|