summaryrefslogtreecommitdiff
path: root/buildstream/_artifactcache
Commit message (Collapse)AuthorAgeFilesLines
* Remove unused _KeyStrength importsJürg Billeter2018-02-272-2/+0
|
* _artifactcache: Add key parameter to pull() methodJürg Billeter2018-02-272-33/+41
| | | | Contain cache key logic in Element class.
* _artifactcache: Add keys parameter to push() methodJürg Billeter2018-02-272-13/+14
| | | | Contain cache key logic in Element class.
* _artifactcache: Add key parameter to push_needed() methodJürg Billeter2018-02-272-6/+4
| | | | Contain cache key logic in Element class.
* _artifactcache: Add keys parameter to commit() methodJürg Billeter2018-02-273-25/+12
| | | | Contain cache key logic in Element class.
* _ostree.py: Use list of refs as commit parameterJürg Billeter2018-02-271-1/+1
|
* _artifactcache: Add key parameter to extract() methodJürg Billeter2018-02-273-16/+6
| | | | Contain cache key logic in Element class.
* _artifactcache: Add key parameter to contains() methodJürg Billeter2018-02-273-39/+9
| | | | Contain cache key logic in Element class.
* _artifactcache/artifactcache.py: Add pull() to abstract base classJürg Billeter2018-02-271-0/+12
|
* _artifactcache/artifactcache.py: Add push() to abstract base classJürg Billeter2018-02-271-0/+17
|
* _artifactcache/tarcache.py: Remove unused remove() methodJürg Billeter2018-02-271-14/+0
|
* Don't push artifacts to remote caches that already contain themSam Thursfield2018-02-262-5/+57
| | | | | | | | | | | | | | | | | | | | | This adds two separate checks to avoid pushing artifacts unnecessarily. First, in the OSTreeCache.push() method we now first obtain the list of remotes that contain the refs that we are about to push. We then avoid pushing to any remote that already contains that ref. Note that the internal ref map is read once on process startup, so if multiple `bst` processes are pushing to a cache they might still both push the same artifact. There is an existing issue for this: https://gitlab.com/BuildStream/buildstream/issues/179 Secondly the Element._skip_push() method now checks if all remote caches configured for pushing already have a given artifact, and will skip the push job altogether if they do. The first check would already mean that no pushes would actually happen, but without the second check the user would still see Push jobs being created for every artifact which would be quite misleading.
* _artifactcache/ostreecache.py: Rework so that all remotes for a ref are trackedSam Thursfield2018-02-261-58/+81
| | | | | | | | | | This commit adds a couple of simple classes in order to reduce the number of different dictionaries tracking the state of the remotes within the OSTreeCache object. It also extends the internal ref map to remember all the remotes that store a given ref, not just the highest priority ref, and modifies the remote_contains_key() method to expose that data.
* _artifactcache/artifactcache.py: Fix name mismatch in doc commentSam Thursfield2018-02-261-1/+1
|
* _artifactcache/ostreecache.py: Make enable_push a keyword argumentJürg Billeter2018-02-241-1/+1
| | | | Boolean positional arguments can be confusing.
* _artifactcache: Use project-specific remotes for subprojectsJürg Billeter2018-02-083-43/+110
|
* Shorten the warnings raised when remote cache initialization failsSam Thursfield2018-01-162-4/+3
| | | | | | | | | | Before: [--:--:--] WARNING Failed to fetch remote refs from ssh://ostree@ostree.baserock.org:22000/cache: BuildStream did not connect successfully to the shared cache ssh://ostree@ostree.baserock.org:22000/cache: SSH error: ssh: connect to host ostree.baserock.org port 22000: Connection refused After: [--:--:--] WARNING Failed to fetch remote refs from ssh://ostree@ostree.baserock.org:22000/cache. ssh: connect to host ostree.baserock.org port 22000: Connection refused
* Use explicit element state updatesJürg Billeter2018-01-122-14/+25
| | | | | | | | | | This adds the _update_state() method to the Element class to keep track of element state and avoid calculating the same cache key multiple times. This also consolidates the different get_cache_key methods into a single method that always returns the cache key calculated by _update_state(), if available. Fixes #149, #173
* Update copyright years of files I have touchedSam Thursfield2018-01-113-3/+3
|
* _artifactcache/ostreecache.py: Subprocesses should not return exceptionsSam Thursfield2018-01-111-10/+6
| | | | | | | | | We have a policy that subprocesses can return error messages, but not actual exceptions. In particular this allows GLib.Error exceptions to be returned, which would otherwise be silently lost due to https://gitlab.gnome.org/GNOME/pygobject/issues/145
* _artifactcache: Use pushreceive also for local artifact repositoriesJürg Billeter2018-01-112-45/+80
| | | | | This reduces the differences between local and remote artifact repositories, increasing code coverage of tests.
* _artifactcache/pushreceive.py: Flush but do not close stdoutJürg Billeter2018-01-111-1/+1
| | | | | Closing stdout will cause an exception when used with multiprocessing as that calls sys.stdout.flush() as well.
* _artifactcache/ostreecache.py: Handle ^C and shutdown child task when ↵Tristan Van Berkom2018-01-111-4/+14
| | | | | | initializing cache This fixes issue #141
* Improve error messages for SSH protocol handshakeSam Thursfield2018-01-111-15/+21
| | | | | | | | | | | | | If you have an artifact remote cache in your config using the ssh:// protocol and some SSH issue prevents us from connecting, we now show the user the stderr output from the `ssh` program, e.g. [--:--:--] WARNING Failed to fetch remote refs from ssh://artifacts@172.17.0.2:22200/artifacts/: BuildStream did not connect successfully to the shared cache ssh://artifacts@172.17.0.2:22200/artifacts/: Permission denied (publickey,keyboard-interactive). Previously we would just show the error message raised by the protocol code, which was less useful, e.g.: [--:--:--] WARNING Failed to fetch remote refs from ssh://artifacts@172.17.0.2:22200/artifacts/: BuildStream did not connect successfully to the shared cache ssh://artifacts@172.17.0.2:22200/artifacts/: Expected reply, got none
* Allow push to multiple remotes, configurable on a per-remote basisSam Thursfield2018-01-113-75/+98
| | | | | | | | | | The initial multiple cache support patch implemented a rather fragile logic where we would push to the first cache in the list that used the ssh:// protocol, if any. If we implement push-over-https:// in future then this will become totally unworkable. This patch alters the logic so that each remote has a 'push' option, and BuildStream will push to any remote that has 'push: true' set.
* Add support for multiple remote cachesSam Thursfield2018-01-113-135/+217
| | | | | | | | | | | | | | | | | | | | | | This extends the 'artifacts' configuration block such that a list of `url` mappings can be given instead of a single entry. For example: artifacts: - url: http://example.com/artifacts1 - url: ssh://ostree@example.com/artifacts2 The OSTreeCache class is updated to set up multiple remotes and query remote refs from all of them. There are no automated tests for this yet. Empty URLs ('') now raise an exception. They cause breakages internally if we allow them through, and they can only occur if the user or our tests are misconfiguring things somehow. We report failure to fetch from the cache by printing a message to stderr for now. This is because BuildStream's actual logging functionality can't be used during frontend init -- see issue #168.
* _artifactcache/tarcache.py: Fixed for new location of ProgramNotFoundErrorTristan Van Berkom2017-12-201-2/+2
|
* Remove unused importsGökçen Nurlu2017-12-072-3/+0
|
* Only initialize remote artifact cache connections if neededSam Thursfield2017-11-232-6/+16
| | | | | | | | | | | | | | | | This fixes a regression from the canonical-pull-urls branch that was just merged. The `OSTreeCache.__init__()` function was connecting to the cache, which is bad because execution would randomly freeze for several seconds when the connection was slow. We now only initialize remote connections where needed; this was already introduced in 5c2ef6d076921bc0121e61efaa7a719c34ea1912 but had regressed. I renamed the keyword arg from `fetch_remote_refs` to `use_remote_cache` because it needs to be set for any interaction with the remote cache, doesn't matter if they are fetches or pushes. The initialization stage is also moved later so that we print a message telling the user what we are up to before trying the network access.
* _artifactcache.py: Fixed stack trace regression in pushing of artifacts.Tristan Van Berkom2017-11-241-2/+2
|
* Replace 'push-url' and 'pull-url' options with just 'url'sam/canonical-push-urlsSam Thursfield2017-11-222-57/+49
| | | | | | | | This is possible now that the SSH protocol can redirect to the appropriate pull URL. Note that this commit makes no attempt at backwards compatibility. Everthing will break!
* Remove the push-port config optionSam Thursfield2017-11-223-16/+10
| | | | | | | | | | | | Ports can and should be specified by using proper ssh:// URL forms, e.g: ssh://artifacts@example.com:22200/artifacts The alternate form of artifacts@example.com:artifacts isn't a valid URL, and doesn't let you specify a different port. People are used to this form due to Git continuing to use it but we should encourage people to use proper URLs.
* Artifact push URLs must redirect to the corresponding pull URLSam Thursfield2017-11-222-14/+56
| | | | | | | | | | | | | | | | | This change is part of issue #112 ("Artifact configuration is confusing and fragile, need canonical push/pull urls.") It updates the bst-artifact-receive program to send a pull URL to clients who access it over SSH. This requires extra configuration in the artifact cache server, so that it knows the correct pull URL. Versions of BuildStream which contain this commit will not be able to communicate over SSH with artifact caches that have not been updated to the same version. Currently this is just used to double-check user configuration, but we can now simplify the user facing configuration for artifact caches completely.
* Switch old-style string formattings to new '.format()'Gökçen Nurlu2017-11-171-14/+14
|
* Refactor, remove some unused imports138-aborting-bst-push-command-causes-stack-traceAngelos Evripiotis2017-11-131-1/+0
| | | | | | | When reading element.py, I was surprised that it was importing 'inspect'. Remove that import and some other unused ones to improve readability.
* refactoring: Made Project option privateTristan Van Berkom2017-11-062-6/+6
| | | | | | | | | | | | | | This required adding two new APIs to make up for it on the Source o get_project_directory() Added here because elements should not be accessing external resources, Sources needed for local files and GPG keys and such o translate_url() Used by sources to mish-mash the project aliases and create real urls.
* Refactoring: Rename _ArtifactError -> ArtifactErrorTristan Van Berkom2017-11-063-29/+29
| | | | This is now part of the already private _exceptions module
* Refactoring: Move exceptions module to be privateTristan Van Berkom2017-11-063-3/+4
| | | | Hide all of buildstream's internal exceptions from the API surface.
* _artifactcache/ostreecache.py: Allow explicit disabling of pushesuser-namespace-fallbackTristan Van Berkom2017-10-311-1/+14
| | | | | | This is used by the platform to disable pushing explicitly in the case that we're falling back to not using user namespaces, in which case we dont entirely trust the artifacts we create.
* _artifactcache: Adapt to new Element api _get_strict()Tristan Van Berkom2017-10-272-4/+4
|
* artifactcache.py: Fix missing fetch_remote_refs interface98-ctrl-c-doesn-t-properly-kill-a-non-interactive-shellTristan Maat2017-10-191-0/+3
|
* Ensure that artifact file permissions are set in the right orderTristan Maat2017-10-131-1/+4
|
* context.py: Make _get_overrides() take a project name, not a project.Tristan Van Berkom2017-10-021-1/+1
| | | | | | | This is merely a protection against crossing the project/context wires together. The Project will gain knowledge of the context soon, so it's better that the context itself remain without any knowledge of the context.
* _yaml.py: Fixing inconsistently named API _yaml.validate_node()Tristan Van Berkom2017-10-011-1/+1
| | | | | | | | | | All of the _yaml APIs which deal with a node (dictionary loaded from YAML) are named _yaml.node_foo(), _yaml.node_bar(), _yaml.node_baz() etc, except for this one glaring exception. Also added a missing internal API documenting comment for _yaml.node_validate() Also updated all callers to _yaml.validate_node() to now call _yaml.node_validate().
* artifactcache.py: Load the overrides from the yaml properlyTristan Van Berkom2017-10-011-5/+13
| | | | | | This was incorrectly accessing the dictionaries directly without the proper _yaml.node_get() APIs, it was also missing out on validation of the artifact configuration.
* Fix artifact push capability detectionfix-artifacts-can-pushMathieu Bridon2017-09-301-1/+1
| | | | | | | | | | Two branches got developed at the same time: * Tristan's cross-platform work * my work on per-project overrides for artifacts-related options Mine got merged first, and somehow this must have been missed in a rebase.
* Fix tests for other platformsTristan Maat2017-09-281-2/+0
|
* Implement tarcacheTristan Maat2017-09-282-3/+349
|
* Create artifactcache interfaceTristan Maat2017-09-282-286/+391
|
* Let users override project artifacts optionsMathieu Bridon2017-09-191-1/+8
| | | | | | | | | | | | | | | | | | | A project can specify its artifacts cache sharing settings, and users can define a default artifacts cache to use as a fallback. With this change, users can also override the project configuration with their own. That means for a project named "libfoo", BuildStream will resolve the artifacts-related options in the following order: 1. the projects.libfoo.artifacts options from the user configuration; 2. if the above was not defined, then the artifacts options from the project configuration; 3. if the above was not defined, then the artifacts options from the user configuration; Fixes #87
View Lorry Controller Status