summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* fixupsjennis/rebase_fixupsJames Ennis2019-04-183-9/+24
|
* CacheKey: Add a CacheKey class which delegates logic to varying implementationsJonathan Maw2019-04-188-88/+437
| | | | | | | | | | | | | | | | | | | | | i.e. a CacheKey class, with implementations such as: * StrictCacheKey - a normal element when buildstream runs in Strict mode * NonStrictCacheKey - a normal element when buildstream runs in non-strict mode * StrictWorkspacedCacheKey - a workspaced element when Strict mode * NonStrictWorkspacedCacheKey - a workspaced element in non-strict mode (note: only CacheKey and StrictCacheKey are implemented for now) This involves: * Creating the CacheKey and StrictCacheKey classes * In Element instantiation, create a __cache_key_obj to delegate various logic to. * For the moment, extending various functions to call a __cache_key_obj's methods if they're implemented. * This includesExtending the _KeyStrength enum to include a STRICT version, so strict cache keys can be accessed through a common interface.
* element.py: Split ready_for_runtime check into an API-private methodJonathan Maw2019-04-181-3/+10
| | | | | This is done because an external caller will be calling this in certain circumstances.
* element.py: Remove _update_state from _schedule_trackingJonathan Maw2019-04-181-1/+0
| | | | Nothing can happen at this point
* element.py: _fetch_done only updates source stateJonathan Maw2019-04-181-1/+6
|
* element.py: Promote __update_source_state to API privateJonathan Maw2019-04-181-7/+8
| | | | | There are now times when we want to call _update_source_state outside of Element.
* Merge branch 'tpollard/955' into 'master'bst-marge-bot2019-04-172-295/+142
|\ | | | | | | | | | | | | Follow-up from "Artifact 'abstraction' class" Closes #955 See merge request BuildStream/buildstream!1271
| * element.py: Discard __artifact/__strict_artifact in update_state()Tom Pollard2019-04-172-10/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | When we determine that current cache keys are to be discarded, we should also discard the associated members for correctness. This should act as a safeguard against accessing Artifact instances that have discarded cache keys. A side effect of this is the querying of element buildtree state from cli methods on elements that don't have a cached artifact would lead to the artifact member methods not being reachable. Ensuring the element is cached before trying to call said accessor methods resolves this.
| * _artifact.py: Move element metadata caching to Artifact ClassTom Pollard2019-04-172-173/+53
| | | | | | | | | | | | | | | | | | | | | | | | | | As the artifact object now stores keys, it allows for the simplification of metadata methods. Transitioning the metadata members to Artifact also means the private accessor methods can be removed from Element which reduces the overhead of syncing private variables across the two classes via parameters. With the key and metadata handling now being internalised within Artifact, a temporary Artifact for the last_successful incremental build is created for correctness. This should help guarantee that an Artifact object is never used to access different artifacts.
| * element.py: Remove redundant fully private __cached_success()Tom Pollard2019-04-171-8/+5
| | | | | | | | | | | | This was being accessed via a local private method that wasn't augmenting the output as such it can be moved to match _cached_failure.
| * element.py: Remove redundant fully private __get_build_result()Tom Pollard2019-04-171-9/+5
| | | | | | | | | | This was being accessed via a local private method that wasn't augmenting the output as such it can be moved.
| * element.py: Instantiate the Artifact objects within update_state()Tom Pollard2019-04-172-109/+83
| | | | | | | | | | | | | | | | | | | | | | The Artifact instance within Element is now initialised when the relevant keys are available, with the weak & strong keys being in the constructor. It also introduces the notion of a strict_artifact, which is used to make a distinction of the Artifact keys and cached state dependant on the context defined strictness. The keys are updated when a relevant key change occurs. As the keys are now internalised within Artifact, the majority of key determinsim and parameterisation in public Artifact method calls are now redundant.
| * element.py: Assert cached in __load_public_data()Tom Pollard2019-04-172-3/+1
| | | | | | | | | | | | This moves the cached assertion to happen before calling the Artifact method, which removes the need to loopback for the same purpose.
| * _artifact.py: Use _get_subdirectory() in non meta* methodsTom Pollard2019-04-171-8/+8
|/ | | | | | Switch to using the private _get_subdirectory() in places where we are interacting directly with a subdirectory of the artifact.
* Merge branch 'phil/rename-plugintestutils' into 'master'bst-marge-bot2019-04-17159-159/+153
|\ | | | | | | | | Rename 'plugintestutils' package to 'testing' See merge request BuildStream/buildstream!1291
| * plugintestutils: Rename 'plugintestutils' package to 'testing'phil/rename-plugintestutilsPhil Dawson2019-04-16159-159/+153
|/ | | | | | | | | - Rename plugintestutils to testing. - Don't run the tests from bst-plugins-template. This imports buildstream.plugintestutils so will have to be disabled to get through CI. This can be re nabled once bst-plugins-template has been patched.
* Merge branch 'phil/fix-empty-external-tests' into 'master'bst-marge-bot2019-04-161-1/+2
|\ | | | | | | | | Fix empty external tests See merge request BuildStream/buildstream!1295
| * tox.ini Allow movement of coverage files to fail for external testsPhil Dawson2019-04-161-1/+2
|/ | | | | | | If the list of external plugin tests is empty, tox -e pyXX-external will fail attempting to move a coverage file which doesn't exist. Allow the offending mv command to fail.
* Merge branch 'tristan/fix-non-strict-push' into 'master'bst-marge-bot2019-04-152-0/+69
|\ | | | | | | | | | | | | Fix non strict push Closes #990 See merge request BuildStream/buildstream!1290
| * tests/frontend/push.py: Test pushing artifacts in non-strict modeTristan Van Berkom2019-04-151-0/+55
| | | | | | | | This is a regression test for issue #990
| * _stream.py: Mark all elements as pulled in Stream.push()Tristan Van Berkom2019-04-151-0/+14
|/ | | | | | | | | Marking all elements as pulled in Stream.push() ensures that cache keys are resolved before pushing elements, otherwise state is left unresolved in non-strict mode while elements are awaiting to download an artifact by it's strict cache key. Fixes #990
* Merge branch 'phil/expose-templated-tests' into 'master'bst-marge-bot2019-04-1574-183/+415
|\ | | | | | | | | Expose templated source tests See merge request BuildStream/buildstream!1261
| * tox.ini: Add tox env as a wrapper for running individual templated testsphil/expose-templated-testsPhil Dawson2019-04-121-0/+20
| |
| * docs: Add generated docs for buildstream.plugintestutilsPhil Dawson2019-04-122-0/+5
| |
| * Make templated source tests available in buildstream.plugintestutilsPhil Dawson2019-04-1243-43/+60
| |
| * plugintestutils._utils: Copy subset of tests.testutils.sitePhil Dawson2019-04-122-1/+47
| | | | | | | | | | | | A subset of the functionality provided by this module is not available from buildstream._site but is necessary for the templated source plugin tests.
| * plugintestutils: Add copy of testutils.junction.pyPhil Dawson2019-04-126-4/+88
| | | | | | | | | | Copy tests.testutils.junction into plugintestutils._utils. This is needed for use by the templated source tests.
| * testutils: move repo.py into buildstream.plugintestutilsPhil Dawson2019-04-1238-167/+227
|/ | | | | | | | This needs to be exposed as part of the plugin author facing API so that plugin authors can define custom repo types which will can be passed to the set of tests which iterate over multiple source types. Part of the work towards #944
* Merge branch 'jjardon/bst-plugins-experimental' into 'master'bst-marge-bot2019-04-121-1/+1
|\ | | | | | | | | .gitlab-ci.yml: master (bst2) should use the new bst-plugins-experimental repo instead bst-external See merge request BuildStream/buildstream!1275
| * .gitlab-ci.yml: bst2 should use the new bst-plugins-experimental repo ↵jjardon/bst-plugins-experimentalJavier Jardón2019-04-121-1/+1
|/ | | | instead bst-external
* Merge branch 'juerg/partial-cas' into 'master'bst-marge-bot2019-04-129-54/+240
|\ | | | | | | | | Do not fetch file blobs for build-only dependencies with remote execution See merge request BuildStream/buildstream!1278
| * cascache.py: Simplify handling of Digest objectsJürg Billeter2019-04-121-17/+7
| | | | | | | | | | Avoid unnecessary copies and use CopyFrom() instead of copying fields one by one where copies are necessary.
| * tests/remoteexecution/partial.py: Add test for partial CASJürg Billeter2019-04-122-0/+52
| |
| * _sandboxremote.py: Fetch only needed directoriesJürg Billeter2019-04-121-3/+21
| |
| * element.py: Set sandbox build directoryJürg Billeter2019-04-121-0/+9
| | | | | | | | | | | | Let the sandbox know whether the buildtree will be required. This allows the remote execution sandbox to skip buildtree download when it's not needed.
| * element.py: Set SandboxRemote.output_files_requiredJürg Billeter2019-04-121-1/+4
| | | | | | | | This enables conditional output file fetching with remote execution.
| * sandbox.py: Add _set_build_directory() methodJürg Billeter2019-04-121-0/+16
| | | | | | | | | | This will enable SandboxRemote to download the file blobs of the build directory only when needed.
| * _sandboxremote.py: Make output file fetching conditionalJürg Billeter2019-04-121-7/+23
| | | | | | | | | | Output files need to be downloaded only if they are required in the local cache or if artifact push remotes need them.
| * _artifactcache.py: Add find_missing_blobs() methodJürg Billeter2019-04-121-0/+27
| |
| * cascache.py: Add remote_missing_blobs() methodJürg Billeter2019-04-121-1/+13
| | | | | | | | Extracted from remote_missing_blobs_for_directory().
| * cascache.py: Do not fetch files in _fetch_tree()Jürg Billeter2019-04-121-5/+0
| | | | | | | | Download them separately as they are not always required.
| * _sandboxremote.py: Fetch output file blobs after pull_tree()Jürg Billeter2019-04-121-1/+9
| | | | | | | | | | This allows skipping file download in pull_tree()/_fetch_tree() and is preparation for conditional output file fetching.
| * cascache.py: Make _required_blobs() publicJürg Billeter2019-04-121-27/+32
| |
| * element.py: Do not pull file contents if not requiredJürg Billeter2019-04-121-0/+5
| |
| * _stream.py: Suport partial CAS for build sessions with remote executionJürg Billeter2019-04-121-0/+7
| |
| * _artifact.py: Use Element._artifact_files_required() in cached()Jürg Billeter2019-04-121-1/+1
| | | | | | | | | | Ensure required artifact files are available for elements that need them even if the context does not require artifact files for all elements.
| * element.py: Add _set_artifact_files_required() methodJürg Billeter2019-04-121-0/+24
| | | | | | | | | | This will be used to allow build sessions with a partial local CAS when using remote execution.
| * element.py: Drop remote execution fallback for incompatible pluginsJürg Billeter2019-04-121-7/+6
|/ | | | | | Element plugins will be required to support virtual directories before the next major version. Drop support for fallback from remote to local execution as this blocks partial CAS.
* Merge branch 'aevri/platform_uname' into 'master'bst-marge-bot2019-04-126-50/+72
|\ | | | | | | | | use uname in a win32-compatible way See merge request BuildStream/buildstream!1262
| * Use uname field names instead of indicesaevri/platform_unameAngelos Evripiotis2019-04-112-3/+3
| | | | | | | | | | | | platform.uname() returns a namedtuple of "system, node, release, version, machine, and processor". Use these slightly clearer names instead of the magic numbers.
View Lorry Controller Status