summaryrefslogtreecommitdiff
path: root/buildstream
Commit message (Collapse)AuthorAgeFilesLines
...
* _gitsourcebase.py: Fetch with explicit refspecsRichard Maw2019-01-211-1/+2
| | | | | Old versions of git lack --force and --tags, but the same effect can be had by specifying refspecs.
* widget.py: Avoid "showing 0 lines" messages when there are no linesTristan Maat2019-01-181-2/+3
| | | | | | | | This happened when bst is invoked with --message-lines 0 or --error-lines 0, and was arguably a little too verbose (the user explicitly asked us not to show them any lines, after all). Fixes #779
* Derive import plugin from Element instead of BuildElementChandan Singh2019-01-181-9/+2
| | | | | | | | | | | The `import` element is not really a build element. The main purpose of the `BuildElement` class is to allow users to run `build-commands`, `install-commands` etc. But, `import` does not run such commands. Moreover, we already override all the methods provided by `BuildElement`. So it only makes it confusing to have it derived from `BuildElement` class when it is not a build element. So, derive it from the base `Element` class instead.
* _cas/cascache.py: Use utils._tempdir() and utils._tempnamedfile()Tristan Van Berkom2019-01-181-6/+5
| | | | | | | | | | | The direct usage of tempfile.TemporaryDirectory() and tempfile.NamedTemporaryFile() here causes leakage of any temporary data when the process operating on temporary data is terminated with SIGTERM. Using the utilities ensures that trash is not left behind in ~/.cache/buildstream/artifacts/tmp when the user terminates BuildStream with ^C.
* utils.py: Added _tempnamedfile()Tristan Van Berkom2019-01-181-0/+30
| | | | | | When used in a child process in BuildStream, this should be used instead of tempfile.NamedTemporaryFile() directly, otherwise we fail to cleanup the file on SIGTERM.
* _cas/cascache.py: Cleanup directories when removing refsTristan Van Berkom2019-01-181-5/+51
| | | | | With out this, empty directories in the refs/heads directory just grow unconditionally.
* cli.py: add a hint about '--' to 'bst shell' helpAngelos Evripiotis2019-01-181-0/+6
|
* bzr source plugins: Remove all of the atomic backup directory business logicTristan Van Berkom2019-01-181-74/+18
| | | | | Follow up of last commit which uses exclusive locking to protect bzr operations instead.
* bzr source plugin: Use lock files to avoid corrupting the source cacheTristan Van Berkom2019-01-181-6/+29
| | | | This patch by itself fixes #868
* _profile.py: Write binaries as well as logsjennis/profiling_outputs_binariesJames Ennis2019-01-171-2/+14
| | | | | | Private class methods which write the logs and write the binaries have been added to Profile. The binaries are able to be used by various visualisation tools.
* _profile.py: Add timestamp to the logsJames Ennis2019-01-171-4/+4
|
* sandbox/sandbox.py: Display failed commands in the detail stringtristan/error-message-regressionTristan Van Berkom2019-01-162-6/+7
| | | | | | | | | | | We should only display commands in detail strings, not in the message texts. This also updates tests/integration/sandbox-bwrap.py to expect the new message string which only contains the command exit status and not the whole command itself, this does not alter the validity of the text case which is checking that we can obtain the expected return value.
* element.py: Cleanup temporary staging directories on terminationTristan Van Berkom2019-01-162-31/+28
| | | | | | | | | | | Use utils._tempdir() which in turn uses _signals.terminator() for this purpose. This also changes utils._tempdir() to use utils._force_rmtree() so that it is safe to use for cleaning up the staging directory. This patch fixes orphaned temporary directories being left behind in the artifact cache directory at forceful termination time.
* _artifactcache/artifactcache.py: Rephrase failure messageTristan Van Berkom2019-01-161-1/+1
| | | | | | | | | | It was saying "There is not enough space to build the given element.", this makes me think the error is associated to a specific element, but this does not make sense to show up in a cleanup task. Instead say "There is not enough space to complete the build.", which should be more clear that even after cleaning up there is not enough space.
* _scheduler: Refactor of queues and resources.Tristan Van Berkom2019-01-167-239/+287
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This branch makes the following changes: * jobs/job.py: No longer stores any interested resource list Jobs are ephemeral again, they only ever exist while they are running. * queues/queue.py: Revert to only handling lists of elements Elements pass through the queues, Queue.harvest_jobs() replaces Queue.pop_ready_jobs() and now the Queue stops creating jobs as soon as there are not enough resources for the job. Also removed unused `prepare()` abstract method. * queues/buildqueue.py: Adapt the part where we launch a job This part needs to be reworked anyway, just touch it up for now so that it doesnt break with the surrounding changes. * jobs/{cachesize,cleanup}job.py: Expose uniform complete callback Allows the scheduler to manage resource deallocation for these two job completions as a custom thing, at the same phase that the Queues take care of their own resource deallocation. * resources.py: No longer has knowledge of the job Since jobs are ephemeral, they are not a suitable place to store the resource identifiers, these must be provided by the callers wherever needed. Now the main Resources object is owned by the Scheduler but shared with Queues, each take care of managing the resources of the jobs they create through the same resource API. * scheduler.py: Reverted to only creating jobs on demand This changes the flow of the scheduler such that whenever jobs complete, the queues are interrogated for as many jobs which can run at the moment but not more; and this completely removes the waiting list. For the internal cache management jobs, we handle this with a little state instead of having a waiting list and only launch when the resources permit it. By abolishing the scheduler waiting list and creating jobs on demand, we fix the order of element processing and consequently fix issue #712.
* cas: move remote only functions to CASRemoteRaoul Hidalgo Charman2019-01-164-144/+148
| | | | | | | | | | | | List of methods moved * Initialization check: made it a class method that is run in a subprocess, for when checking in the main buildstream process. * fetch_blobs * send_blobs * verify_digest_on_remote * push_method Part of #802
* casremote.py: Move remote CAS classes into its own fileRaoul Hidalgo Charman2019-01-164-257/+283
| | | | Part of #802
* _cas: Rename artifactcache folder and move that to a root moduleRaoul Hidalgo Charman2019-01-166-11/+11
| | | | | | | | | Other components will start to reply on cas modules, and not the artifact cache modules so it should be organized to reflect this. All relevant imports have been changed. Part #802
* _context.py: Add documentation to get_workspaces() commandJames Ennis2019-01-161-0/+7
|
* _context.py: Doc fix: get_toplevel_project() returns object not listJames Ennis2019-01-161-1/+1
|
* buildstream/_gitsourcebase.py: Fix case where HEAD is taggedValentin David2019-01-161-13/+18
| | | | | `git rev-list --boundary HEAD..HEAD` does not return any boundary. So in this case we need to manually tag the HEAD as a boundary.
* buildstream/_gitsourcebase.py: Reduce git history for git describe.Valentin David2019-01-161-1/+2
| | | | | | | | | | | | Found during #833. `git rev-list --boundary tag..HEAD` unfortunately gives boundaries that are deeper than should when there is a merge commit between `tag` and `HEAD`. The common ancestory of the two parents of the merge is a boundary instead of the parent of the branch that is not traversed. `--ancestry-path` fixes this issue by restricting `git` traversing those branches.
* _frontend/cli.py: Also check original args for --config in bash completionJürg Billeter2019-01-141-7/+20
| | | | | | | The path of the config file generated by testutils for completion tests is passed as regular argument, not via COMP_WORDS. Use that config file in complete_artifact() to ensure the test uses the right artifact directory.
* _scheduler/queues/queue.py: Put elements in the skip list, not jobsTristan Van Berkom2019-01-121-1/+1
| | | | | Minor correction, looks like we're not observing this queue otherwise we'd be seeing crashes.
* _pipeline.py: Fix the planner to retain the original order when depth sortingTristan Van Berkom2019-01-101-1/+2
| | | | | | | | | | | The algorithm adds elements to a dictionary and then sorts the dictionary by the discovered depths while recursing - using an OrderedDict is enough to ensure a stable order. This fixes `bst show --deps plan ...` reporting different results on each invocation. This fixes an aspect of #824
* element.py: Accept architecture aliases for sandbox configjuerg/archJürg Billeter2019-01-101-1/+7
| | | | | Accept common architecture aliases for the sandbox config for consistency with arch options.
* _options/optionarch.py: Accept architecture aliasesJürg Billeter2019-01-101-1/+30
| | | | | | | | Accept common architecture aliases for arch options instead of only accepting the canonicalized, OS-independent architecture name. This restores compatibility with existing projects and may simplify option handling for projects that only target a single OS (and thus, do not need OS-independent architecture names).
* _platform/platform.py: Add canonicalize_arch() methodJürg Billeter2019-01-101-14/+28
|
* sandboxremote.py: Rename 'instance_name' option to 'instance-name'jmac/rename_instance_nameJim MacArthur2019-01-091-2/+2
| | | | It was 'instance-name' in the documentation.
* buildstream/plugins/elements/script.py: Mark script as BST_VIRTUAL_DIRECTORYvalentindavid/script_virtual_directoryValentin David2019-01-091-0/+3
| | | | | | | ScriptElement does not use Sandbox.get_directory. It works using it with remote execution. Fixes #850
* Add support for https channel to remote execution and actions serversvalentindavid/remote_execution_configurationValentin David2019-01-091-8/+35
| | | | Fixes #780.
* Add support for user remote execution configurationValentin David2019-01-093-4/+20
| | | | Fixes #631.
* Use relative path to project directory for remote execution certificates/keysValentin David2019-01-091-0/+10
|
* sandbox/sandbox.py: Use assertions for programming errors instead of BstErrors.tristan/sandbox-programming-errorTristan Van Berkom2019-01-081-4/+4
| | | | | When code is faulty, in a plugin or in the core, we should get a stack trace and a BUG message.
* element.py: remove reference to source bundle commandphil/remove-source-bundle-referencePhil Dawson2019-01-081-1/+1
| | | | This command has been replacved by the bst source checkout command
* sandbox/sandbox.py: Fix regression of command loggingTristan Van Berkom2019-01-071-1/+1
| | | | | | | | | | Since we added batch commands, the batch commands print the text of the commands directly in the message text, but this is wrong. The detail string is the appropriate place for text of unknown lengths (the user can actually configure how many max lines of commands they want to see in their log), the message text itself should be controlled and brief enough to avoid text wrapping.
* _scheduler/jobs/job.py: Removed 'skipped' propertyTristan Van Berkom2019-01-072-20/+2
| | | | This is redundant now that we report it through the JobStatus.
* Scheduler: Introduced JobStatus instead of simple success booleanTristan Van Berkom2019-01-0713-38/+68
| | | | | | | | | | This changes the deepest callback from when a Job completes to propagate a JobStatus value instead of a simple boolean, and updates all of the effected code paths which used to receive a boolean to now handle the JobStatus values. This further improves the situation for issue #753, as now we avoid queueing cache size jobs for pull jobs which are skipped.
* _scheduler/scheduler.py: Only run one cache size job at a timeTristan Van Berkom2019-01-071-8/+45
| | | | | | | | | | | | | | | | | | | When queuing the special cache management related cleanup and cache size jobs, we now treat these jobs as special and do the following: * Avoid queueing a cleanup/cache_size job if one is already queued We just drop redundantly queued jobs here. * Ensure that jobs of this type only run one at a time This could have been done with the Resources mechanics, however as these special jobs have the same properties and are basically owned by the Scheduler, it seemed more straight forward to handle the behaviors of these special jobs together. This fixes issue #753
* _scheduler/scheduler.py: Make _schedule_jobs() privateTristan Van Berkom2019-01-071-16/+18
| | | | | | | | This is not used anywhere outside of the Scheduler, currently only the Scheduler itself is allowed to queue a job at this level. If the highlevel business logic for automatic queueing of auxiliary jobs moves to another location, we can make this public again.
* Fix stack traces discovered with ^C forceful termination.tristan/keyboard-interrupt-stack-traceTristan Van Berkom2019-01-072-10/+6
| | | | | | | | | | | | | | | | * utils.py:_kill_process_tree(): Ignore NoSuchProcess errors These are caused because we issue SIGTERM, and if the process has not exited after a timeout, we kill it. * _scheduler/jobs/job.py: Stop handling NoSuchProcess errors here redundantly, they are already ignored. It seems that we were ignoring it after sleeping when terminating tasks from the scheduler... but we were not ignoring it when performing the same pattern in the `Plugin.call()` -> `utils._call()` path, so we were still getting these exceptions at termination time from host tool processes launched by source plugins.
* _frontend/app.py: Use buildtree for interactive shell on build failureJürg Billeter2019-01-061-1/+1
| | | | Fixes: e29aea36 ("Basic options for shell --build to use buildtrees")
* _frontend/cli.py: Reinstate support for guessing targetsjuerg/source-guess-targetJürg Billeter2019-01-041-0/+10
| | | | | | Moving fetch and track to the source command group accidentally dropped the support for guessing targets for these commands when invoked from a workspace directory. This brings it back.
* Use collections.abc for Mapping, IterableThomas Coldrick2019-01-032-14/+14
| | | | | | | In _yaml.py and _frontend/complete.py we were getting pylint warnings for using collections.Mapping and collections.Iterable, which are abstract classes now provided from collections.abc. This patch just uses the classes from the right place.
* Use pycodestyle instead of pep8Javier Jardón2019-01-023-3/+3
| | | | | | | | pep8 package was renamed to pycodestyle to reduce confusion. Disabled warnings: W504 - Line break after binary operator (seems people like this) W605 - Invalid escape sequence (some of our regexes use these)
* _loader/loader.py: Add warnings about invalid characters in filenameChandan Singh2018-12-312-0/+43
| | | | | See https://mail.gnome.org/archives/buildstream-list/2018-December/msg00061.html for some related discussion.
* _loader/loader.py: Refactor warnings about element names in one methodChandan Singh2018-12-311-18/+33
| | | | | | | Currently we some duplication in the way we check for invalid filenames. To make it more robust and allow room for adding more warnings, refactor it into a separate method `_warn_invalid_elements()` that handles just this.
* _downloadablefilesource.py: Avoid crashes when HOME is unsetTristan Van Berkom2018-12-261-1/+7
| | | | | | The python netrc module will raise OSError in the case that HOME is not set, this was discovered while running tests under tox.
* buildstream/storage/_casbaseddirectory.py: Fixing new (unnecessary-pass) ↵Tristan Van Berkom2018-12-261-5/+0
| | | | linter errors
* plugins/elements/import.py: Fixing new ↵Tristan Van Berkom2018-12-261-1/+1
| | | | (duplicate-string-formatting-argument) linter error
View Lorry Controller Status