summaryrefslogtreecommitdiff
path: root/tests/cachekey
Commit message (Collapse)AuthorAgeFilesLines
* _artifact.py: Store additional metadata on the artifactTristan van Berkom2020-12-0729-29/+29
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This commit enriches the metadata we store on artifacts in the new detatched low/high diversity metadata files: * The SandboxConfig is now stored in the artifact, allowing one to perform activities such as launching sandboxes on artifacts downloaded via artifact name (without backing project data). * The environment variables is now stored in the artifact, similarly allowing one to shell into a downloaded artifacts which are unrelated to a loaded project. * The element variables are now stored in the artifact, allowing more flexibility in what the core can do with a downloaded ArtifactElement * The element's strict key All of these of course can additionally enhance traceability in the UI with commands such as `bst artifact show`. Summary of changes: * _artifact.py: - Store new data in the new proto digests. - Added new accessors to extract these new aspects from loaded artifacts. - Bump the proto version number for compatibility * _artifactcache.py: Adjusted to push and pull the new blobs and digests. * element.py: - Call Artifact.cache() with new parameters - Expect the strict key from Artifact.get_meta_keys() - Always specify the strict key when constructing an Artifact instance which will later be used to cache the artifact (i.e. the self.__artifact Artifact). * _versions.py: Bump the global artifact version number, as this breaks the artifact format. * tests/cachekey: Updated cache key test for new keys.
* sandbox/_config.py, element.py: Refactor SandboxConfigTristan van Berkom2020-12-0729-29/+29
| | | | | | | | | | | | | | | | | | | | | This commit changes SandboxConfig such that it now has a simple constructor and a new SandboxConfig.new_from_node() classmethod to load it from a YAML configuration node. The new version of SandboxConfig now uses type annotations. SandboxConfig also now sports a to_dict() method to help in serialization in artifacts, this replaces SandboxConfig.get_unique_key() since it does exactly the same thing, but uses the same names as expected in the YAML configuration to achieve it. The element.py code has been updated to use the classmethod, and to use the to_dict() method when constructing cache keys. This refactor is meant to allow instantiating a SandboxConfig without any MappingNode, such that we can later load a SandboxConfig from an Artifact instead of from an parsed Element. This commit also updates the cache keys in the cache key test, as the cache key format is slightly changed by the to_dict() method.
* tests/cachekey: Added new cachekey test for new BuildElement configurationTristan van Berkom2020-09-184-1/+16
|
* buildelement.py: Remove legacy command stepsTristan van Berkom2020-09-183-3/+3
| | | | | | | | | | | This was actually deadcode, since node.validate_keys() was called on the configure dictionary without the legacy command steps. If any element was using the legacy commands, they would have been met with a load time error anyway. This commit also updates the cache key test, since removing these legacy commands affects BuildElement internally in such a way as to affect the cache keys.
* ScriptElement: Porting to new APITristan van Berkom2020-09-182-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | Instead of relying on Element.search(), use Element.configure_dependencies() to configure the layout. Summary of changes: * scriptelement.py: Change ScriptElement.layout_add() API to take an Element instead of an element name, this is now not optional (one cannot specify a `None` element). This is an API breaking change * plugins/elements/script.py: Implement Element.configure_dependencies() in order to call ScriptElement.layout_add(). This is a breaking YAML format change. * tests/integration: Script integration tests updated to use the new YAML format * tests/cachekey: Updated for `script` element changes
* Move handling of the source `directory` configuration to ElementSourcesjuerg/element-source-cacheJürg Billeter2020-09-0328-28/+28
| | | | | | | | | | | | | | | | | The `directory` value determines where a source is staged within the build root of an element, however, it does not directly affect individual sources. With this change the sources will individually be cached in CAS independent of the value of `directory`. `ElementSources` will use the value of `directory` when staging all element sources into the build root. This results in a cache key change as the `directory` value is moved from the unique key of individual sources to the unique key of `ElementSources`. This is in preparation for #1274.
* _elementsources.py: Fix source names in unique keyjuerg/source-cache-keyJürg Billeter2020-08-2528-28/+28
| | | | | | | | | | | | | | The goal was to include the source plugin kind in the element cache key as the unique key of a source may not be unique across different source plugins. This is the source equivalent of the `element-plugin-name` value in the element cache key. However, `Source._get_source_name()` was the wrong method for this as that also includes the key itself, which may not even be set yet. This results in a cache key change. Fixes: 3953bcc6 ("element.py: clobber sources with workspace")
* buildstream/element.py: Consider "build-root" variable in the cache key ↵tristan/build-root-cache-keyTristan van Berkom2020-08-2028-28/+28
| | | | | | | | | | | | | | | | | | | unconditionally Not all elements use the "build-root" variable, but it is the standard variable to use for the build directory, and the build directory must be considered in the cache key. Handling this unilaterally in the core is safer than delegating this to element implementations, as we have less chance of plugin authors missing this detail and possibly introducing binary variance for artifacts where only the build directory differs (something which happens when the project name or element names change). This commit also updates the hard coded cache keys in the cache key test, so as to ensure every commit passes it's own tests. This fixes #1386.
* tests/cachekey/cachekey.py: Use the same project name for cache key testsTristan van Berkom2020-08-201-1/+1
| | | | | | | | In the test_cache_key_fatal_warnings() test, use the same project name in both generated project directories in order to pass the tests in the case that elements are guaranteed to have differing cache keys for differeing element/project names (which is the case when we consider the "build-root" in the cache key unconditionally).
* testing.py: Add a new `check_cache_key_stability` helperbschubert/cache-key-helperBenjamin Schubert2020-05-122-178/+5
| | | | | This allows plugin authors to implement cache keys tests more easily
* Replace format-version with min-versionTristan Van Berkom2020-04-252-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * "min-version" is specified as a <major>.<minor> point version and uses the installed BuildStream version instead of having a separate versioning number for the format. * The presence of "format-version" is now used to indicate that we might be loading a BuildStream 1 project. * For now, where parsing the version at startup is concerned, and also where `bst init` is concerned, we artificially bump the detected BuildStream version to 2.0 if we detect a version < 2.0, these exceptions can be removed once 2.0 is tagged and released. Summary of changes: _project.py: Now parse "min-version" and detect "format-version" to warn about loading a BuildStream 1 project _versions.py: Remove obsolete BST_FORMAT_VERSION numbers from here data/projectconfig.yaml: Remove old "format-version" from defaults utils.py: Added new private _parse_version() helper function, and another _get_bst_api_version() to get an adjusted API version. frontend/app.py, frontend/cli.py: Updated `bst init` implementation testing (buildstream.testing): Updated testing utilities to generate and use projects with min-version instead of format-version. tests and examples: Updated to use min-version across the board.
* Canonicalize OS nameJürg Billeter2020-02-2728-28/+28
| | | | This matches the REAPI platform lexicon.
* tests: Remove stray workspaces.yml filesJürg Billeter2020-02-111-0/+0
|
* Reformat code using BlackChandan Singh2019-11-142-86/+64
| | | | | | | As discussed over the mailing list, reformat code using Black. This is a one-off change to reformat all our codebase. Moving forward, we shouldn't expect such blanket reformats. Rather, we expect each change to already comply with the Black formatting style.
* local.py: use extended API, set BST_NO_PRESTAGE_KEYDarius Makovsky2019-10-3015-14/+17
| | | | | | | tests: * local sources are cached * use git sources instead of local when removing objects * update expected cache keys
* element.py: remove call to _source_cached()traveltissues/benchmarkDarius Makovsky2019-10-0528-28/+28
| | | | | | | Remove call to Element._source_cached() in _calculate_cache_key and do not recalculate workspace keys. tests: shell browsing of workspaces is currently broken
* element.py: always check sources for cachingDarius Makovsky2019-09-2428-28/+28
| | | | | | | | | | | | | | | | | | | | | | Check that sources are cached even if they are workspaced and do not reset workspace cache data partially reverts !1470 closes #1088 element.py: remove workspaces in cache key calc Using the workspace source plugin, workspaces should now be handled like sources for the purpose of calculating element cache keys. partially reverts !1470 works towards #1073 Since the source keys are now calculated using the unique keys of the workspace source, this change will break external tracking for open workspaces. In future attempting to track an open workspace might raise a SourceError. The test is rewritten to close the workspace before tracking.
* artifact.proto: Add the project name to the build dependency informationJames Ennis2019-08-1928-28/+28
| | | | | | | | | We also need to store the project name of dependencies in the proto in order to reconstruct the graph. This also means that we must include the project name of the dependencies in the cache key, otherwise the proto and the cache keys may fall out of sync.
* element.py: Add the build dependency names to the cache key dictJames Ennis2019-08-197-7/+7
| | | | | We store the build dependency names in the artifact proto. This should be reflected in the cache key dict.
* element.py: clobber sources with workspaceDarius Makovsky2019-07-2229-29/+28
| | | | add name to sources in cache key dict
* Make the Cli verbose for updatingtraveltissues/fix-update-scriptDarius Makovsky2019-07-181-1/+4
| | | | Return if there is no result from parsing the element
* Mock BST_TEST_SUITE env when running update.pyDarius Makovsky2019-07-181-1/+7
| | | | | If this is not present in the env the cli will throw an exception closes #1074
* tests: Change all calls to _yaml.dump to _yaml.rountrip_dumpBenjamin Schubert2019-07-151-2/+2
| | | | | Now that both are equivalent, we can skip the sanitization part before the yaml call.
* element.py: change cache key dict fieldstraveltissues/cache-key-changesDarius Makovsky2019-07-1228-28/+28
| | | | relates to #1073
* _yaml: Remove useless calls to '_yaml.node_sanitize'Benjamin Schubert2019-06-251-1/+1
| | | | | | Calling '_yaml.dump' will itself call '_yaml.node_sanitize', therefore we can remove all calls to it in places where we directly after call dump.
* test:utils/site: Consolidate IS_LINUX/WSL/Windows in a single siteBenjamin Schubert2019-06-061-2/+1
| | | | | | | | We currently have two sites file containing redundant information. - tests/testutils/site.py: Remove IS_{LINUX,WSL,WINDOWS} - rest: Change imports to import those from buildstream/tests/_utils/site.py
* test:utils/site: Consolidate MACHINE_ARCH variable in a single placeBenjamin Schubert2019-06-061-2/+2
| | | | | | | We have two different 'site' files that are redundant and both define some variables in BuildStream environment. Moving the MACHINE_ARCH to a single place.
* test:utils/site: Consolidate Git environment variables in a single placeBenjamin Schubert2019-06-061-2/+2
| | | | | | | We have two different 'site' files that are redundant and both define some variables in BuildStream environment. Moving all the git related ones in a single place.
* test:utils/site: Consolidate Bzr environment variables in a single placeBenjamin Schubert2019-06-061-1/+2
| | | | | | | We have two different 'site' files that are redundant and both define some variables in BuildStream environment. This is a first step in consolidating them in a single file
* Remove OSTree plugin; It lives now in the bst-plugins-experimental repoJavier Jardón2019-05-144-10/+2
|
* plugintestutils: Rename 'plugintestutils' package to 'testing'phil/rename-plugintestutilsPhil Dawson2019-04-162-2/+2
| | | | | | | | | - Rename plugintestutils to testing. - Don't run the tests from bst-plugins-template. This imports buildstream.plugintestutils so will have to be disabled to get through CI. This can be re nabled once bst-plugins-template has been patched.
* tests: when comparing lists/dicts, compare all at onceBenjamin Schubert2019-03-211-3/+4
| | | | | This allows pytest to show a better report of the difference between the two lists and not just tell that one entry is wrong.
* tests: str(datafiles) instead of a longer thingAngelos Evripiotis2019-03-211-2/+2
| | | | | | | | | | | | | | | | | | | Replace some popular copypasta. This important-looking invocation: os.path.join(datafiles.dirname, datafiles.basename) is equivalent to this shorter invocation: project = str(datafiles) It seems like it's very popular copypasta, replace it with the shorter one thus: # Use 'gsed' or 'sed' etc. as appropriate for your system. git config --global alias.sub '!f() { git grep --name-only --null "$1" | gxargs --null gsed --in-place --expression "s/$1/$2/g" ; }; f' git sub 'os.path.join(datafiles.dirname, datafiles.basename)' 'str(datafiles)'
* tests:lint: disable 'unused-import' checks on pytest fixturesBenjamin Schubert2019-03-201-1/+1
| | | | | Pylint can't know that pytest's fixtures are used in a file and therefore reports false positives. Silencing all those errors
* tests:lint: silence redefined-outer-name in files using fixturesBenjamin Schubert2019-03-201-0/+4
| | | | | Pylint doesn't play well with pytest fixtures, we therefore need to silence this error.
* tests:lint: fix indentationBenjamin Schubert2019-03-201-1/+1
|
* tests:lint: remove all unneccessary-parens errors from pylintBenjamin Schubert2019-03-201-1/+1
|
* tests:lint: reorder imports for consistencyBenjamin Schubert2019-03-201-4/+6
| | | | | - Remove all wrong-import-order from pylint - Order some subgroups of imports
* tests: Remove unused variablesBenjamin Schubert2019-03-011-1/+1
|
* Bump artifact version for changes in symlink handlingJürg Billeter2019-02-1429-29/+29
|
* Cache Keys: Update to use JSON rather than pickledanielsilverstone-ct/json-cache-keyDaniel Silverstone2019-02-1429-29/+29
| | | | | | | | This affects the cache key version (updated to 7) and introduces a dependency on `ujson` which is BSD licenced as of the version locked in `requirements.txt` Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
* utils.py: Return all directories in list_relative_paths()Jürg Billeter2019-02-1314-14/+14
| | | | | | | | | | | | | | | Returning only empty directories leads to inconsistencies when computing a manifest by combining results from multiple list_relative_paths() calls as done by the compose plugin. I.e., the same directory may be empty in one dependency and non-empty in another dependency. The merged file list will still contain that directory even though it's no longer empty. This inconsistency causes problems when calculating differences between manifests. Returning all directories fixes these inconsistencies. This is a change in API behavior.
* projectconfig.yaml: Consistently include directories in split rulesJürg Billeter2019-02-1329-29/+29
| | | | | | | | | Most split rules already included the relevant directories themselves in addition to the directory contents. Add the missing bin, sbin, and libexec directories. This is required to fix tests with the following commit that changes list_relative_paths() to return all directories.
* Expose basic api for testing external plugins.phil/plugin-testing-apiPhil Dawson2019-02-082-2/+2
| | | | | | | | | | | | | | | | | | We want external plugins to be able to make use of the core testing utils. This commit exposes the basic utilities which are currently in use in bst-external plugins. If necessary, more utilities could be exposed in the future. Moves the following files from tests/testutils/ to buildstream/plugintestingutils/: o runcli.py o integration.py As part of this, this commit makes the following changes to runcli.py and integration.py: o runcli.py: Fix linting errors o runcli.py: Add user facing documentation o Integration.py: Add user facing documentation
* tests/cachekey: Test cache keys are independent of target elementsphil/cache-key-stability-testPhil Dawson2019-02-016-0/+66
|
* buildstream/data/projectconfig.yaml: Remove default strip-binariesJavier Jardón2018-12-093-3/+3
| | | | | | | | | They are too specific to be included by default Recommendation is if you are building in Linux is to use the ones begin used in the freedesktop-sdk project, for example See #645
* optionarch.py: update to use same arch names as SandboxConfigRaoul Hidalgo Charman2018-12-051-1/+1
| | | | Also update tests to be consistent with this
* _config.py: Use os and architecture settingsRaoul Hidalgo Charman2018-12-0529-29/+29
| | | | | | In element, platform asks for host os and architecture to default to when SandboxConfig is initialised. This changes element cache keys so those have been updated in the tests.
* git source plugin: Track git tags and save them to reproduce a minimum ↵Valentin David2018-12-054-1/+15
| | | | | | | | | | | | | | | shallow repository Instead of tag information being fetched which can change with time, they are tracked and saved in the projects.refs/.bst. Then we re-tag automatically the closest tag so that `git describe` works and is reproducible. This new feature is opt-in with the new `track-tags` configuration, and must be used to fix modules which are broken by our new policy of omitting the `.git/` repository when staging git sources. This fixes issue #487
* Disable cachekey tests on other architectures than x86_64Valentin David2018-11-141-1/+3
|
View Lorry Controller Status