summaryrefslogtreecommitdiff
path: root/buildstream/utils.py
Commit message (Collapse)AuthorAgeFilesLines
* utils.py: Add _get_volume_size()Tristan Van Berkom2019-01-241-2/+22
| | | | We can streamline this call to os.statvfs() in a few places.
* utils.py: Added _tempnamedfile()Tristan Van Berkom2019-01-181-0/+30
| | | | | | When used in a child process in BuildStream, this should be used instead of tempfile.NamedTemporaryFile() directly, otherwise we fail to cleanup the file on SIGTERM.
* element.py: Cleanup temporary staging directories on terminationTristan Van Berkom2019-01-161-1/+1
| | | | | | | | | | | Use utils._tempdir() which in turn uses _signals.terminator() for this purpose. This also changes utils._tempdir() to use utils._force_rmtree() so that it is safe to use for cleaning up the staging directory. This patch fixes orphaned temporary directories being left behind in the artifact cache directory at forceful termination time.
* Fix stack traces discovered with ^C forceful termination.tristan/keyboard-interrupt-stack-traceTristan Van Berkom2019-01-071-0/+5
| | | | | | | | | | | | | | | | * utils.py:_kill_process_tree(): Ignore NoSuchProcess errors These are caused because we issue SIGTERM, and if the process has not exited after a timeout, we kill it. * _scheduler/jobs/job.py: Stop handling NoSuchProcess errors here redundantly, they are already ignored. It seems that we were ignoring it after sleeping when terminating tasks from the scheduler... but we were not ignoring it when performing the same pattern in the `Plugin.call()` -> `utils._call()` path, so we were still getting these exceptions at termination time from host tool processes launched by source plugins.
* buildstream/utils.py: Fix regex Deprecation WarningChandan Singh2018-12-201-2/+2
| | | | | | | | | | | | | Specify flags at the start of the expression as per the recommendation of the standard library. Without this patch, we currently get the following warning: ``` tests/examples/junctions.py::test_open_cross_junction_workspace /builds/BuildStream/buildstream/dist/buildstream/buildstream/utils.py:213: DeprecationWarning: Flags not at the start of the expression '\\/[^/]*\\Z(?ms)' regexer = re.compile(expression) ```
* utils.py: Add a helper for searching upwards for filesJonathan Maw2018-12-111-0/+31
| | | | | | | | i.e. with a given directory and filename, check parent directories until either a directory with the filename is found, or you reach the root of the filesystem. This is a part of #222
* utils.py: Add message digest functionRaoul Hidalgo Charman2018-12-051-0/+17
| | | | Get cascache.py to use this
* utils.py: improve documentation of move_atomicBenjamin Schubert2018-11-211-2/+8
|
* utils.py: Ensure move_atomic booleans are keyword only argumentsBenjamin Schubert2018-11-211-1/+1
| | | | This is to be coherent with the rest of the codebase
* Extract atomic move function to utils.pyBenjamin Schubert2018-11-191-0/+37
| | | | | | | | Moving atomically a file/directory can be tricky since different errors might be raised for the same underlying problem. Having a utility function to reduce this discrepancies will help in ensuring we have correct behavior
* utils: Fix _pretty_size() for sizes > 1024TJosh Smith2018-11-061-3/+4
|
* various: Move _sentinel from utils.py to _yaml.pyDaniel Silverstone2018-10-261-4/+0
| | | | | | | | | The `_sentinel` in `utils.py` was used only for yaml node tracking. As such, simplify matters by removing it from `utils.py` and move it to `_yaml.py` which means that we no longer need to try and avoid a circular import issue by means of runtime importing. Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
* utils.py: Group stat importsDaniel Silverstone2018-10-251-1/+1
| | | | | | | The imports of `stat` and `S_ISDIR` from `stat` should be grouped to satisfy pylint. Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
* utils.py: Disable lint warning about popenDaniel Silverstone2018-10-251-1/+2
| | | | | | | | | | We use `preexec_fn` in our use of Popen. Since there are threads in the codebase, this is not necessarily safe. However we go to great lengths to ensure that the main process doesn't spawn threads (they're used in gRPC a lot) and so it should be safe for our use. As such, we disable the lint here. Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
* Fix spelling of it's and itsRichard Dale2018-10-161-1/+1
|
* Catch correct exception from shutil.rmtreevalentindavid/rmtree_oserrorValentin David2018-10-041-1/+1
| | | | | | | | | | | | | Python documentation is not clear on what shutil.rmtree is supposed to raise. However from the source code, it is obvious that OSError are raised, but never shutil.Error. It is not easy to test in normal condition because issues happen usually in combination with a FUSE filesystem, a probably a race condition where `fstatat` returns an error just before the filesystem being unmounted. Fixes #153.
* utils.py: Document _get_dir_size() expectations.Tristan Van Berkom2018-10-031-0/+3
| | | | | This function assumes that files do not disappear while walking the given directory.
* utils.py: Give save_file_atomic() a tempdir argumentTristan Van Berkom2018-10-031-3/+4
| | | | Allow callers to decide where the temporary file will be created.
* utils.py: Reworked safe_removeknownexus2018-09-271-19/+18
| | | | | | | | | | | Non-Linux platforms don't return EISDIR when attempting to unlink a directory. Stopped safe_remove attempting to unlink dirs Previously safe_remove would attempt to unlink a path Before attempting to remove it if it was a dir Now it checks for a dir before that step
* Handle sockets when copying trees into artifactsRichard Maw2018-09-141-0/+11
| | | | | We can't include a socket in a CAS tree, but it's mostly meaningless to do so since there can't possibly be a process serving it.
* Prevent ValueError on URLs without an aliasJosh Smith2018-08-291-0/+1
| | | | Fixes #597
* Fix E305 warningsJavier Jardón2018-08-281-0/+1
|
* Catch Non Numeric versionsWilliam Salmon2018-08-221-1/+10
| | | | | | | This patch just displays a better message than the default stack trace but dose not try to fix the problem. A further patch will be created but it effects versioneer so may take longer to land as it may need to go via versioneer mainline.
* Use deterministic umask when staging sources.Valentin David2018-08-121-1/+10
| | | | | | This fix is applied to plugins bzr, git, patch. Fixes #543 #544 #555.
* utils.py: add getmtime() and magic_timestampJim MacArthur2018-08-011-5/+13
| | | | | magic_timestamp is moved into file scope so other classes can use it.
* _context.py: Cache size is now restricted to available disk spaceQinusty/491Josh Smith2018-07-271-0/+21
| | | | | | | | | | | | This address issue #491. When attempting to run buildstream with a configuration specifying a cache quota larger than your available disk space, buildstream will alert the user and exit. Note: This takes into consideration your current cache usage and therefore restricts the overall size of your artifact cache folder.
* Move _ALIAS_SEPARATOR into utilsJonathan Maw2018-07-271-0/+4
| | | | The separator is useful in source files other than _project.py
* Add cache_quota to user configTristan Maat2018-07-181-0/+70
|
* utils.py: Allow `list_relative_paths` to list directoriesTristan Maat2018-07-181-6/+8
|
* Remove shebangs from python filesGökçen Nurlu2018-06-191-1/+0
| | | | Fixes #424
* Handle missing tags in git repositories correctlyTristan Maat2018-06-111-0/+4
| | | | Fixes issue #380
* utils.py: Correcting a typo in safe_remove's commentPhillip Smyth2018-06-051-1/+1
|
* Use versioneer instead of setuptools_scmTristan Van Berkom2018-04-261-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Using setuptools_scm had a couple of bad problems: o Unexpected versioning semantics, setuptools_scm would increment the micro version by itself in the case that we derive a version number from something that is not a tag, making the assumption that we are "leading up to" the next micro version. People mostly dont expect this. o When installing in developer mode, i.e. with `pip3 install --user -e .`, then we were always picking the generated version at install time and never again dynamically resolving it. Many of our users install this way and update through git, so it's important that we report more precise versions all the time. This commit needs to make a series of changes at the same time: o Adds versioneer.py to the toplevel, this is used by setup.py for various activities. This is modified only to inform the linter to skip o Adds buildstream/_version.py, which is generated by versioneer and gives us the machinery to automatically derive the correct version This is modified only to inform the linter to skip o Adds a .gitattributes file which informs git to substitute the buildstream/_version.py file, this is just to ensure that the versioning output would work if ever we used `git archive` to release a tarball. o Modifies setup.py and setup.cfg for versioneer o Modifies utils.py and _frontend/cli.py such as to avoid importing the derived version when running bash completion mode, we dont derive the version at completion time because this can result in running a subprocess (when running in developer install mode) and is an undesirable overhead. o Updates tests/frontend/version.py to expect changed version output
* buildstream/utils.py: Fixing unused variablesTristan Van Berkom2018-04-191-2/+2
|
* utils.py: Added _is_main_process()Tristan Van Berkom2018-04-101-0/+12
| | | | | A helper function to identify if we are running in the main process or not.
* Allow 'None' as a default_value for _yaml.node_getTristan Maat2018-04-071-0/+5
|
* Allow stage_artifact to update mtimesTristan Maat2018-03-271-0/+12
|
* pylint - dealt with redefined-outer-name and redefined-built in warningsJames Ennis2018-03-141-1/+1
|
* pylint - dealt with unnecessary-pass warningJames Ennis2018-03-141-1/+1
|
* pylint - dealt with anomalous-backslash-in-string warningJames Ennis2018-03-141-1/+2
|
* utils.py: Wrap calls to os.path.realpath() in an LRU cacheSam Thursfield2018-02-121-9/+15
| | | | | | | | | | | | | | | | | | | | | | | | | | The os.path.realpath() function is expensive and we call it many times, to the point that os.path.realpath() calls make up around 40% of the total time spent in Element.stage_artifact(). The cleanest way to fix this is with a `functools.lru_cache()` wrapper that caches recently used values. None of the code in question can be removed (as the tests added in the previous commit will demonstrate). I tested this by running `bst shell base/base-system.bst true` in the GNOME modulesets project. o Without this patch there are 240,019 calls os.path.realpath() o With this patch there are 10,379 calls to os.path.realpath() o If we increase the cache size to 128 items, there are 10,359 calls to os.path.realpath(). o If we reduce the cache size to 32 items, there are 10,426 calls. o In all cases the number of *unique* calls is 10,327. This fixes issue #174.
* Get version number w/o pkg_resourcesGökçen Nurlu2018-01-311-3/+2
| | | | | | | | | | | | This removes most of the usages of pkg_resources from the codebase, helping the goal of getting rid of that completely. With this change, version number is generated during install and embedded into `__version__` which is then imported by root level `__init__`. From there, it is used by other parts of the codebase when needed. Generated `__version__` file is ignored and not tracked by git to prevent unnecessary 'changes' messages and accidental commits of that file.
* buildstream/utils.py: remove unused import 'pickle'Angelos Evripiotis2018-01-241-1/+0
|
* Update copyright years of files I have touchedSam Thursfield2018-01-111-1/+1
|
* Add support for multiple remote cachesSam Thursfield2018-01-111-0/+30
| | | | | | | | | | | | | | | | | | | | | | This extends the 'artifacts' configuration block such that a list of `url` mappings can be given instead of a single entry. For example: artifacts: - url: http://example.com/artifacts1 - url: ssh://ostree@example.com/artifacts2 The OSTreeCache class is updated to set up multiple remotes and query remote refs from all of them. There are no automated tests for this yet. Empty URLs ('') now raise an exception. They cause breakages internally if we allow them through, and they can only occur if the user or our tests are misconfiguring things somehow. We report failure to fetch from the cache by printing a message to stderr for now. This is because BuildStream's actual logging functionality can't be used during frontend init -- see issue #168.
* utils.py: Make list_relative_paths() report sorted paths.Tristan Van Berkom2018-01-101-15/+26
| | | | | | | | | | | This is needed because plugins make use of this function to generate a cache key which must be stable. In addition to the above, this patch also mitigates the performance hit of sorting, and allows _process_list() to function to iterate through the generator when additional sorting is not needed. This patch is an enhanced version of Sam Thursfield's patch on MR !216
* Move utils._generate_key() into a new 'cachekey' moduleSam Thursfield2018-01-041-19/+1
| | | | | This avoids a circular dependency between the 'utils' and '_yaml' modules.
* utils.py: Add save_file_atomic() helperSam Thursfield2018-01-041-0/+58
| | | | | | | | | | | | This is a context manager that can be used to divert file writes into a temporary file, which is then renamed into place once writing is complete. It is primarily intended for use by source plugins which download files, so they can ensure that their downloads appear atomic and there is no risk of leaving half-downloaded files in the cache. So far this is not used in the core, but it is needed by the Docker source plugin that is proposed for the bst-external plugins repo. See: https://gitlab.com/BuildStream/bst-external/merge_requests/9
* Exceptions refactoringTristan Van Berkom2017-12-291-3/+5
| | | | | | | | | | | | | | | Outline of changes to exceptions: o Now BstError base class has a `domain` and `reason` member, reason members are optional. o All derived error classes now specify their `domain` o For LoadError, LoadErrorReason defines the error's `reason` o Now the scheduler `job` class takes care of sending the error reason and domain back to the main process where the last exception which caused a child task to fail can be discretely stored.
* utils._call: rm unused assignments to exit_codeAngelos Evripiotis2017-12-221-2/+1
|
View Lorry Controller Status