| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
| |
A helper function to identify if we are running in the main process
or not.
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The os.path.realpath() function is expensive and we call it many times,
to the point that os.path.realpath() calls make up around 40% of the
total time spent in Element.stage_artifact().
The cleanest way to fix this is with a `functools.lru_cache()` wrapper
that caches recently used values. None of the code in question can be
removed (as the tests added in the previous commit will demonstrate).
I tested this by running `bst shell base/base-system.bst true` in
the GNOME modulesets project.
o Without this patch there are 240,019 calls os.path.realpath()
o With this patch there are 10,379 calls to os.path.realpath()
o If we increase the cache size to 128 items, there are 10,359 calls
to os.path.realpath().
o If we reduce the cache size to 32 items, there are 10,426 calls.
o In all cases the number of *unique* calls is 10,327.
This fixes issue #174.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
This removes most of the usages of pkg_resources from the codebase, helping
the goal of getting rid of that completely.
With this change, version number is generated during install and embedded into
`__version__` which is then imported by root level `__init__`. From there,
it is used by other parts of the codebase when needed.
Generated `__version__` file is ignored and not tracked by git to prevent
unnecessary 'changes' messages and accidental commits of that file.
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This extends the 'artifacts' configuration block such that a list of
`url` mappings can be given instead of a single entry. For example:
artifacts:
- url: http://example.com/artifacts1
- url: ssh://ostree@example.com/artifacts2
The OSTreeCache class is updated to set up multiple remotes and query
remote refs from all of them.
There are no automated tests for this yet.
Empty URLs ('') now raise an exception. They cause breakages internally
if we allow them through, and they can only occur if the user or our
tests are misconfiguring things somehow.
We report failure to fetch from the cache by printing a message to
stderr for now. This is because BuildStream's actual logging
functionality can't be used during frontend init -- see issue #168.
|
| |
|
|
|
|
|
|
|
|
|
| |
This is needed because plugins make use of this function to
generate a cache key which must be stable.
In addition to the above, this patch also mitigates the performance
hit of sorting, and allows _process_list() to function to iterate
through the generator when additional sorting is not needed.
This patch is an enhanced version of Sam Thursfield's patch on MR !216
|
| |
|
|
|
| |
This avoids a circular dependency between the 'utils' and '_yaml'
modules.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
This is a context manager that can be used to divert file writes into a
temporary file, which is then renamed into place once writing is
complete. It is primarily intended for use by source plugins which
download files, so they can ensure that their downloads appear atomic
and there is no risk of leaving half-downloaded files in the cache.
So far this is not used in the core, but it is needed by the Docker
source plugin that is proposed for the bst-external plugins repo. See:
https://gitlab.com/BuildStream/bst-external/merge_requests/9
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Outline of changes to exceptions:
o Now BstError base class has a `domain` and `reason` member,
reason members are optional.
o All derived error classes now specify their `domain`
o For LoadError, LoadErrorReason defines the error's `reason`
o Now the scheduler `job` class takes care of sending the error
reason and domain back to the main process where the last
exception which caused a child task to fail can be discretely stored.
|
| | |
|
| |
|
|
|
|
|
|
|
|
| |
Tested by inflating the chance of hitting the race. First, insert a
sleep between opening the terminator context and starting the process,
then:
python3 -c 'import buildstream.utils; import os; \
buildstream.utils._call(["echo", "hello"], True); \
print(os.getpid())' & sleep 1; kill $!
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Report UtilError instead of OSError and similar python errors.
Also ensure we catch system errors and raise UtilError with
descriptive text instead; for the user experience; this is the
difference between:
o A FAILURE message with a description as to
what went wrong (exception handled with UtilError)
o A BUG message with the unhandled system error printed
with a stack trace (exception left unhandled)
Also, UtilsError and ProgramNotFoundError are now public exceptions
declared in utils.py, where they will appear in the documentation.
|
| | |
|
| | |
|
| |
|
|
|
| |
These are not used anywhere internally and are not really desirable
to use, as usually you need only care about moving a toplevel directory.
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
| |
The 'result' field is optional, and currently used only by the internals
of {copy,move,link}_files. I think it makes sense to mandate that it's
called as a keyword arg in future.
|
| |
|
|
| |
Hide all of buildstream's internal exceptions from the API surface.
|
| | |
|
| |
|
|
|
|
|
|
|
|
| |
Profiling suggested that the cumulative time spent in os.path.relpath()
was the dominant cost of utils.list_relative_paths().
Try to call this only once per directory walked.
In Python 3.5, we can optimise further by using os.scandir() and
maintaining relative paths as we go.
|
| |
|
|
|
| |
utils._force_rmtree() was overwriting it's path directory, and
then only recursively removing the last entry which was walked.
|
| | |
|
| | |
|
| |
|
|
|
| |
This little bit of code was being repeated a bunch of times
already throughout buildstream, looks sane enough to offer a utility.
|
| | |
|
| |
|
|
| |
Fetches the numeric major/minor version of the BuildStream package.
|
| | |
|
| |
|
|
|
|
|
| |
Support for prefix=None and suffix=None was added in python 3.5.
The default for suffix happens to be "", and the prefix happens to be
"tmp" (though that's an implementation detail), but it's sufficient,
and simpler than writing conditionals that omit certain args.
|
| |
|
|
|
| |
A temporary directory context manager which cleans itself out
on sigterm.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
| |
symlinks.
We want to change ownership of the symlinks, not their targets.
When doing this on an ostree checkout with symlinks to absolute paths, it
will make an attempt to modify ownership of files outside the sandbox,
and usually fail when building as the regular user (otherwise it will harm
the system when running as root).
|
| | |
|
| |
|
|
|
|
|
|
| |
By default we brutally murder subprocess trees, but sometimes we
know the nature of the tool we are using and know that it handles
sigterm gracefully.
This patch allows those cases to have a chance at graceful termination.
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Instead of using PurePath from pathlib, we use a modified version
of fnmatch.translate() to generate a regular expression.
This has multiple advantages:
* The private usage of regular expressions allows the Element
internal splitting rules to more easily adapt to the new globbing
syntax for split rules
* With the glob function under our control, we can document it more
clearly
* We have support for bash 'globstar' mode, which allows recursive
globs with double asterisk.
|
| |
|
|
| |
Fixes: 10a0b98d ("utils.py: Add _call()")
|
| |
|
|
|
|
|
|
|
| |
The glob module from the standard library only allows glob
pattern matching where there is a backing filesystem.
This utility allows glob matching on a simple list of paths,
and as such can be used to match things from an unextracted artifact
or an unextracted tarball.
|
| |
|
|
|
|
|
|
| |
Instead of returning a tuple of two lists, return the
new FileListResult object.
This is better because it will allow us to enhance the result
over time without breaking API.
|
| |
|
|
| |
This reverts commit 9ae1e1a290f67628d4e68632618b0aa9aa92b4e3.
|
