summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* tests:workspace: make deeper tests when trying to build allbschubert/pipeline-nextBenjamin Schubert2019-03-1611-16/+48
|
* tests/frontend/workspaces.py: Extended test that all elements get built in ↵Tristan Van Berkom2019-03-169-6/+41
| | | | | | | | the same session This extends the existing test which catches failures only in build-only dependency chains, with a similar test for runtime-only dependency chains.
* tests/frontend/workspace.py: Test that all elements build with workspace in playTristan Van Berkom2019-03-169-0/+74
| | | | | | | Tests that the target is still built even when a workspace is open on a runtime dependency of a build-only dependency. This adds the regression test provided by Matthew Yates for issue #919
* Test with Unique PQ for update state recursiveBenjamin Schubert2019-03-162-10/+130
|
* element.py: Add _fetch_done method and use this to update the stateBenjamin Schubert2019-03-162-1/+9
| | | | This is to be consistent will all other queues that follow this setup.
* Don't update state before summary. It should be consistentBenjamin Schubert2019-03-161-12/+1
| | | | | Since we know when an object's state can change, we don't need to update the state of all objects for consistency.
* Stop updating state in queue.status()Benjamin Schubert2019-03-164-12/+21
| | | | | | | | | | | | | | | Statuses of an element can be changed when: 1) It is pulled 2) It is fetched 3) It is workspaced and it finished building 4) One of its dependencies is tracked 5) One of its dependencies is workspaced and finished building We can therefore update the statuses at those moments and we don't need to check all the time. This reduces considerably the calls to update_states that are done
* Merge branch 'juerg/cache-buildtrees' into 'master'bst-marge-bot2019-03-1613-62/+97
|\ | | | | | | | | Tweak cache-buildtrees option See merge request BuildStream/buildstream!1208
| * Change default of cache-buildtrees option to 'auto'Jürg Billeter2019-03-167-27/+28
| | | | | | | | | | Caching buildtrees may significantly increase CAS disk usage. Change the default to cache buildtrees only when required for core functionality.
| * Rename cache-buildtrees option value 'failure' to 'auto'Jürg Billeter2019-03-167-9/+9
| | | | | | | | | | | | | | | | | | | | We anticipate other cases than build failures where buildtree caching will be required. E.g., incremental workspace build with remote execution. Or running tests in a buildtree in parallel with the build of reverse dependencies. This renames the option value 'failure' to the more generic 'auto' to cover these other cases as well.
| * userconfig.yaml: Add disclaimer to cache-buildtrees=neverJürg Billeter2019-03-161-2/+5
| |
| * _frontend/app.py: Fail on missing buildtree in failure handlerJürg Billeter2019-03-161-1/+1
| | | | | | | | | | In the interactive failure handler we should never enter the interactive shell if the buildtree is missing (due to cache-buildtrees=never).
| * _artifact.py: Do not create empty buildtree directoriesJürg Billeter2019-03-166-21/+50
| | | | | | | | | | | | | | | | | | | | Creating an empty buildtree directory when buildtree caching is disabled means cached_buildtree() can return True even if no buildtree is available, which breaks logic in the frontend and in Stream. This adds the buildtree_exists() method to indicate whether an artifact was created with a buildtree, independent of whether the buildtree is available in the local cache.
| * cascache.py: Handle missing subdir case in contains_subdir_artifacts()Jürg Billeter2019-03-161-5/+7
| | | | | | | | Return False instead of raising an error.
| * _stream.py: Fix buildtree error message formatting in shell()Jürg Billeter2019-03-161-2/+2
| | | | | | | | This removes extraneous spaces.
| * tests/integration/artifact.py: Fix empty directory checksJürg Billeter2019-03-161-2/+2
|/ | | | os.listdir() never returns None.
* Merge branch 'aevri/direct_load_junction_projects' into 'master'bst-marge-bot2019-03-154-7/+96
|\ | | | | | | | | project: don't do _find_project_dir if we're a junction See merge request BuildStream/buildstream!1231
| * _project: don't _find_project_dir if a junctionAngelos Evripiotis2019-03-153-4/+85
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | If we're creating a project for a junction, then don't go looking for the project directory - the path has already been given to us. This means that we can now detect when junctions are missing a 'project.conf', and importantly we don't accidentally start resolving elements in the enclosing project. Add tests to cover workspaced, local, and git repo cases. Note that this is also the first test coverage for the INVALID_JUNCTION path. In later work we might extract the _find_project_dir magic out of the Project class, so that there are no surprises when instantiating it.
| * runcli.run(): coerce project and args to stringsAngelos Evripiotis2019-03-151-0/+4
| | | | | | | | | | | | | | | | It would be convenient if we could use datafiles directly, instead of fussing about with os.path.join(). The standard library supports path-like things, I see no reason why we shouldn't. In later work we can refactor our tests to take advantage of this.
| * loader: clearer error if no junction project.confAngelos Evripiotis2019-03-151-3/+7
|/ | | | | | | | | | | | | | | | | | | | | | | | | | | I found the original message confusing - it would talk about wanting a project at '.', without a frame of reference. After re-reading the code and junction docs it made more sense, I think this version would save folks the trouble. Removing all mention of it, if it's not specified. If it is specified then explain it a bit more, i.e. Before: Could not find the project.conf file for junction element at hello-junction.bst [line 1 column 0]. Expecting a project at path 'project' After: Could not find the project.conf file in project referred to by junction element 'hello-junction.bst'. Was expecting it at path 'project' in the junction's source. If no 'path' specified in the junction: Could not find the project.conf file in project referred to by junction element 'hello-junction.bst'.
* Merge branch 'aevri/tmpdir_for_cache_size' into 'master'bst-marge-bot2019-03-151-1/+1
|\ | | | | | | | | cascache: atomically save size via tmpdir instead See merge request BuildStream/buildstream!1226
| * cascache: atomically save size via tmpdir insteadAngelos Evripiotis2019-03-151-1/+1
|/ | | | | | | | | | Instead of doing the atomic 'mv into place' using a temporary file in the 'cas' directory, use the sibling 'tmp' directory instead. This will avoid the situation where CacheSizeJob encounters a temporary file in the cas directory. This can happen because CacheSizeJob does not always run with an exclusive lock on the cas directory, and BuildQueue updates the cache size whenever a BuildJob completes.
* Merge branch 'jjardon/bst-external-table' into 'master'Javier Jardón2019-03-151-0/+7
|\ | | | | | | | | README.rst: Add table with package status of bst-external See merge request BuildStream/buildstream!1218
| * README.rst: Add table with package status of bst-externalJavier Jardón2019-03-141-0/+7
|/
* Merge branch 'aevri/doc_artifact_log' into 'master'Jürg Billeter2019-03-141-2/+37
|\ | | | | | | | | 'artifact log': document the 'artifacts' argument See merge request BuildStream/buildstream!1223
| * 'artifact log': document the 'artifacts' argumentaevri/doc_artifact_logAngelos Evripiotis2019-03-141-2/+37
|/ | | | | | | | | | I found it quite difficult to guess what format the 'bst artifact log' command would like artifact references in. Put them in the help for that command, to save folks time. In future work, we probably want to document the behaviour of 'Stream._classify_artifacts' globally - most commands take them and the results can be surprising if you don't know about artifact refs.
* Merge branch 'aevri/nodefaultsset' into 'master'bst-marge-bot2019-03-141-16/+11
|\ | | | | | | | | element.__init_default: treat `None` plugin_conf as if missing file + refactor See merge request BuildStream/buildstream!1210
| * element.py: remove redundant `__defaults_set`Angelos Evripiotis2019-03-141-4/+2
| | | | | | | | | | | | Use the conventional `None` value to indicate that the class variable `__defaults` is not yet initialized. This lets us remove `__defaults_set`.
| * Element.__init_defaults: init if no plugin_confAngelos Evripiotis2019-03-141-10/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Still initialize `Element.__defaults` as usual, even if a `plugin_conf` is not specified. As part of the work on !1101 to introduce `ArtifactElement`s, a special case for `__init_defaults` was made, such that none of the usual defaults would be loaded. This seems to be because it has no config and none of the defaults are required by ArtifactElement, although they don't seem to adversely affect it. Instead, treat a 'None' `plugin_conf` in the same way as the existing case of a missing file. This avoids the appearance of having plugin-specific behaviour in the base-class, and is perhaps less puzzling to new contributors.
| * element: refactor, rm unused temp varAngelos Evripiotis2019-03-141-2/+1
|/
* Merge branch 'jennis/remove_node_chain_stuff' into 'master'Jürg Billeter2019-03-145-161/+26
|\ | | | | | | | | Rip out ChainMap(), node_chain_copy(), node_list_copy() See merge request BuildStream/buildstream!1209
| * _yaml.py: Rip out ChainMap(), node_chain_copy(), node_list_copy()jennis/remove_node_chain_stuffJames Ennis2019-03-145-161/+26
|/ | | | | | | | | | | | | This class and these two functions exist as they were intended to bring efficiency. Benchmarking this patch against the debian-stack.bst element in the debian-like project [0] showed that although this took 15M more RAM (peak usage), there was a ~20s gain in the time taken to 'show' the stack. Thus the class and functions have been removed. This also has the advantage of removing a lot of duplicate and unnecessary code. [0] https://gitlab.com/jennis/debian-stretch-bst
* Merge branch 'juerg/cas' into 'master'bst-marge-bot2019-03-142-8/+8
|\ | | | | | | | | _casbaseddirectory.py: Use variable-length argument list See merge request BuildStream/buildstream!1230
| * _casbaseddirectory.py: Use variable-length argument list for _exists()Jürg Billeter2019-03-141-1/+1
| | | | | | | | This matches Directory.descend() and _objpath().
| * _casbaseddirectory.py: Use variable-length argument list for _objpath()Jürg Billeter2019-03-142-7/+7
|/ | | | This matches Directory.descend().
* Merge branch 'phil/consolidate-repo-tests' into 'master'bst-marge-bot2019-03-1444-974/+1489
|\ | | | | | | | | Consolidate templated source tests See merge request BuildStream/buildstream!1215
| * tests: move templated tests from workspace.py into separate filephil/consolidate-repo-testsPhil Dawson2019-03-146-6/+203
| | | | | | | | | | | | | | Move the templated tests in tests/frontend/workspace.py into sources/generic/workspace.py This is part of the preparation work for #944
| * tests/frontend/workspaces/py: Fix typo in WorkspaceCreator classPhil Dawson2019-03-141-7/+7
| |
| * tests: move templated tests from source-determinism.py into separate filePhil Dawson2019-03-143-74/+137
| | | | | | | | | | | | | | Move the templated tests in tests/frontend/source-determinism.py into sources/source_determinism.py This is part of the preparation work for #944
| * tests: move templated source tests from track.py into separate filePhil Dawson2019-03-142-368/+413
| | | | | | | | | | Move the templated tests in tests/frontend/track.py into sources/generic/track.py
| * tests: move templated tests from track_cross_junction.py into separate filePhil Dawson2019-03-141-0/+23
| | | | | | | | | | | | | | Move the templated tests in tests/frontend/track_cross_junction.py into sources/generic/track_cross_junction.py This is part of the preparation work for #944
| * tests: move templated tests from buildcheckout.py into basic_functionality.pyPhil Dawson2019-03-142-48/+82
| | | | | | | | | | | | | | Move the templated tests in tests/frontend/buildcheckout.py into sources/generic/buildcheckout.py This is part of the preparation work for #944
| * tests: move templated source tests from mirror.py into separate filePhil Dawson2019-03-142-397/+427
| | | | | | | | | | | | | | Move the templated tests in tests/frontend/mirror.py into sources/generic/mirror.py This is part of the preparation work for #944
| * tests: move templated source tests from fetch.py into separate filePhil Dawson2019-03-1429-75/+198
|/ | | | | | | | Create tests/sources/basic_functionality.py for templated source tests and move the templated tests in tests/frontend/fetch.py into sources/generic/fetch.py This is part of the preparation work for #944
* Merge branch 'raoul/440-source-cache' into 'master'bst-marge-bot2019-03-1438-362/+1195
|\ | | | | | | | | Source cache See merge request BuildStream/buildstream!1124
| * Source cache testsRaoul Hidalgo Charman2019-03-1415-0/+429
| | | | | | | | | | | | | | Few different source cache tests have been to check that sources are staged into the local CAS and fetched where appropriate Part of #440
| * Integrate source cache with rest of buildstreamRaoul Hidalgo Charman2019-03-1418-59/+187
| | | | | | | | | | | | | | | | | | | | | | | | | | | | This involve introducing new Consistency states `STAGED` and `BOTH` that represent when the source is just in the local CAS and in both the local CAS and unstaged in the source directory. Sources are staged for each element into the local CAS during the fetch stage. If the sources are in the local consistency state `STAGED` when wanting to open a workspace, the original sources are fetched. Relavant tests this affects have been changed. Part of #440
| * _sourcecache.py: Introduce SourceCache classRaoul Hidalgo Charman2019-03-142-0/+154
| | | | | | | | | | | | | | | | Introduces source cache which allows sources to be staged into the local CAS for use in builds. Registers methods with CASQuota in order to allow for the clean method to remove unrequired sources as well. Part of #440
| * source.py: Add source cache keyRaoul Hidalgo Charman2019-03-141-0/+23
| | | | | | | | | | | | | | | | | | This adds o _key: property to access cache key o generate_key: method to generate key given previous sources o _get_source_name: method to create ref for given source Part of #440
| * utils: add deterministic_umask context managerRaoul Hidalgo Charman2019-03-142-1/+17
| | | | | | | | | | | | Useful for when exporting file from cas. Part of #870
View Lorry Controller Status