summaryrefslogtreecommitdiff
path: root/buildstream/_pipeline.py
Commit message (Collapse)AuthorAgeFilesLines
* Replace 'push-url' and 'pull-url' options with just 'url'sam/canonical-push-urlsSam Thursfield2017-11-221-22/+2
| | | | | | | | This is possible now that the SSH protocol can redirect to the appropriate pull URL. Note that this commit makes no attempt at backwards compatibility. Everthing will break!
* Check connectivity to remote cache on `bst push`Sam Thursfield2017-11-221-0/+2
|
* _pipeline.py: Adjust remove_elements to work on the 'intersection'Tristan Maat2017-11-181-38/+53
|
* Load excepted elements as part of the pipelineTristan Maat2017-11-181-4/+8
| | | | | | | | Causes --except specified elements to be loaded separately and the intersections of the except elements and the loaded pipeline to be calculated. This fixes issue #131
* Change origin to point at source repoDaniel Playle2017-11-111-1/+1
| | | | | | | | | | | | This changes workspaces created with the git source element so that the origin remote points to the source repository of the build element as opposed to the internal repository in the bst cache. This introduces an addition of the init_workspace method in the source API. This method, which defaults to calling stage, is for the setup of the workspace after the creation of the workspace directory. This is a part of issue #53
* Remove DummyElement hackTristan Maat2017-11-101-18/+10
|
* Refactor: Move context.py -> _context.pyTristan Van Berkom2017-11-081-1/+1
| | | | | | | | | | Consequently: o Changed Plugin.get_context() to a private Plugin._get_context() accessor. o Updated anything which imports Context to do so from private _context module o Updated docs to exclude the now private Context
* Refactoring: Renamed _BstError -> BstErrorTristan Van Berkom2017-11-061-15/+1
| | | | | | Base class for exceptions is now a part of the already private _exceptions module Also moved PipelineError from _pipeline -> _exceptions module
* Refactoring: Rename _ArtifactError -> ArtifactErrorTristan Van Berkom2017-11-061-3/+3
| | | | This is now part of the already private _exceptions module
* Refactoring: Move exceptions module to be privateTristan Van Berkom2017-11-061-2/+2
| | | | Hide all of buildstream's internal exceptions from the API surface.
* _pipeline.py: Removing archesTristan Van Berkom2017-11-061-12/+4
| | | | | Also now the integration option to `bst checkout` is explicit, as we dont have knowledge of arches anymore.
* Adjust commands to multiple targetsTristan Maat2017-11-041-24/+37
|
* _pipeline.py: Misc adjustments to multiple targetsTristan Maat2017-11-041-7/+22
|
* _pipeline.py: Fix metaelement resolutionTristan Maat2017-11-041-9/+7
|
* _pipeline.py: Adjust to new loader APITristan Maat2017-11-041-5/+6
|
* _pipeline.py: Adjust Planner.plan to multiple targetsTristan Maat2017-11-041-3/+3
|
* Make the platform object a singletonTristan Maat2017-11-011-1/+2
|
* _pipeline.py: No except_ argument in source_bundleTristan Van Berkom2017-10-301-3/+1
| | | | | | | This is deadcode, except_ is already calculated before we get here. Updated some internal comments to reflect this, and updated the frontend to not pass the useless except_ parameter.
* _pipeline.py: The toplevel message() function no longer takes an element.Tristan Van Berkom2017-10-301-35/+32
| | | | | | | | | | | | | | | | | | | Instead, use None as the unique ID. This will help with messaging when we allow invoking multiple targets and the element would have been ambiguous - this also consequently fixes issue #137. The reason for the hangs with #137 is because: o When you --except a base element, reverse dependencies cannot calculate a cache key o When you track, cache keys are intentionally reset at startup time, because we know they are going to change o At the end of tracking, we make one attempt to print the toplevel cache key This operation is insanely expensive, because we never cache a cache key because it logically cannot ever be resolved in this situation. This fix is basically a workaround to the above.
* _pipeline.py: Make fetching of remote refs optionalTristan Van Berkom2017-10-291-1/+3
| | | | Default is not to fetch them, we only need them in a few cases.
* _pipeline.py: Improve error message when --except fails.Tristan Van Berkom2017-10-261-1/+1
| | | | This is an interim step for issue #131.
* _pipeline.py: Give the correct artifact pull URL in remote tickerSam Thursfield2017-10-251-1/+1
| | | | | | | | | | I was seeing messages like this: Fetching artifact list from None However, the artifact cache did have the correct pull URL set -- the issue was that I had overridden it my own config, and the string we sent to the ticker didn't take that override into account.
* _pipeline.py: Pass the project options to the loaderTristan Van Berkom2017-10-101-0/+1
|
* project.py: More consistent api for workspacesremove-variantsTristan Van Berkom2017-10-081-1/+1
| | | | | | | | Renamed _workspaces() -> _list_workspaces(), and changed local cache of workspaces from __workspaces -> _workspaces. We only really use the double underscore when there is a concern for freeing up namespace for public subclassing.
* _pipeline.py: No more target variantTristan Van Berkom2017-10-081-7/+4
|
* Add platform factoriesTristan Maat2017-09-281-6/+4
|
* bst push: Check connectivity to cache before trying to pushsam/push-check-connectivitySam Thursfield2017-09-201-0/+2
| | | | | | On slow machines, the actual push operation can spend several minutes preparing before actually pushing, which is annoying if the push then fails.
* Let users override project artifacts optionsMathieu Bridon2017-09-191-1/+2
| | | | | | | | | | | | | | | | | | | A project can specify its artifacts cache sharing settings, and users can define a default artifacts cache to use as a fallback. With this change, users can also override the project configuration with their own. That means for a project named "libfoo", BuildStream will resolve the artifacts-related options in the following order: 1. the projects.libfoo.artifacts options from the user configuration; 2. if the above was not defined, then the artifacts options from the project configuration; 3. if the above was not defined, then the artifacts options from the user configuration; Fixes #87
* Let projects configure their artifacts pull/push optionsMathieu Bridon2017-09-191-1/+1
| | | | | | | With this commit, we first look at the artifacts options in the project configuration, then fall back on the user configuration if necessary. Relates to #87
* Give the various artifact_* attributes to the ArtifactCacheMathieu Bridon2017-09-191-1/+1
| | | | | Having them there will allow overriding the user configuration (the context) with the project configuration.
* Construct pipeline without push queue if cannot push to artifact cacheChandan Singh2017-09-121-8/+9
| | | | | | | | | Currently, if BuildStream is unable to push to the artifact cache for whatever reason, the build will just error out. Fix it so that if we are unable to push to the shared cache, we give a warning to the user and continue building. Fixes https://gitlab.com/BuildStream/buildstream/issues/90
* _pipeline.py: Raise pipeline error in pull/pushTristan Van Berkom2017-09-051-2/+2
| | | | | Instead of trying to fire a MessageType.FAIL without an elapsed parameter, and then not returning from the push()/pull() methods.
* source-bundle: fix filtering out of non-build elementsCharles Bailey2017-09-011-7/+12
| | | | | | Attempting to remove an element from the list being iterated through has undesirable results; fix this by creating a new, filtered list rather than attempting to modify the original list in place.
* workspace open: --no-checkout doesn't need --forceAngelos Evripiotis2017-08-151-1/+1
| | | | | | | | | | Remove the requirement to specify '--force' in conjunction with '--no-checkout' if there are already files in the workspace. We won't write anything when opening the workspace, so there's nothing to force. For example, when opening a workspace to an existing clone of a repository, it seems alarming to have to '--force' the workspace open. It made me wonder if it will actually be overwritten.
* _pipeline.py: Add remote_ticker for artifact list fetchingJürg Billeter2017-08-151-0/+3
|
* _pipeline.py: Fix --deps runJürg Billeter2017-08-101-1/+1
|
* Add `bst pull` commandJürg Billeter2017-08-081-0/+38
|
* Add initial `bst push` commandSam Thursfield2017-08-081-0/+38
| | | | | | | This is mainly useful for testing artifact caches and such. Most users will hopefully be able to make use of artifact caches populated by automated build machines, but right now it's unlikely that most people will be pushing artifacts around.
* Check for write access to remote artifact cache early on in the pipelinesam/artifactcache-preflight-checkSam Thursfield2017-07-211-0/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously, the first time you configured an artifact cache, you would get to the end of your first build and then BuildStream would exit because of some stupid mistake like you got the address slightly wrong or you forgot to add the host keys of the remote artifact cache to `~/.ssh/known_hosts`. To avoid surprises, if there's an artifacts push-url configured we now try to connect to it as a preflight check so that issues are raised early. On success, you will see something like this: [--:--:--][90904fe4][ main:gnu-toolchain/stage2.bst ] START Checking connectivity to remote artifact cache [00:00:00][90904fe4][ main:gnu-toolchain/stage2.bst ] SUCCESS Connectivity OK On failure, it looks like this: [--:--:--][90904fe4][ main:gnu-toolchain/stage2.bst ] START Checking connectivity to remote artifact cache [00:00:03][90904fe4][ main:gnu-toolchain/stage2.bst ] FAILURE BuildStream will be unable to push artifacts to the shared cache: ssh: connect to host ostree.baserock.org port 2220: Connection timed out As a bonus, for some reason this check causes SSH to ask about unknown host keys rather than just failing, so you may now see messages like this if the host keys are unknown rather than an error: The authenticity of host '[ostree.baserock.org]:22200 ([185.43.218.170]:22200)' can't be established. ECDSA key fingerprint is SHA256:mB+MNfYREOdRfp2FG6dceOlguE/Skd4QwnS0tvCPcnI. ECDSA key fingerprint is MD5:8f:fa:ab:90:19:31:f9:f7:f1:d4:e5:f0:a2:be:56:71. Are you sure you want to continue connecting (yes/no)?
* _pipeline.py: Continue if remote artifact repository is unavailableJürg Billeter2017-07-201-2/+6
|
* _pipeline.py: Use _remotely_cached()Jürg Billeter2017-07-201-1/+1
|
* _pipeline.py: Fetch remote refsJürg Billeter2017-07-201-0/+3
|
* _pipeline.py: Reset workspace after deletingTristan Maat2017-07-171-4/+4
|
* _pipeline.py: Add workspace command backendTristan Maat2017-07-171-4/+168
|
* _pipeline.py: Work with new project variantsTristan Van Berkom2017-07-171-5/+11
| | | | | | | | | | | | | | This means: o Load the project elements passing in the possible variants exposed by the project, allowing the loader to error on nonexisting variants and to choose a project variant. o Resolve the project variant after loading the meta elements, but before materializing the Element and Source objects o Delay creation of the plugin contexts and factories until after the project is resolved.
* _pipeline.py: Additional writable check of directory in Pipeline.checkout()Tristan Van Berkom2017-07-131-0/+3
|
* _pipeline.py: Handle exceptions when checking out elementsJonathan Maw2017-07-131-1/+4
| | | | | Previously, an OSError would come up and be handled with a generic stack trace.
* _pipeline.py: Simplified end of session reports.Tristan Van Berkom2017-07-121-23/+8
| | | | | | This was a bit contorted, and is no longer needed with the new end of session generic reports which show the status of each scheduler queue.
* Only run integration commands on checkout for native-built artifactsSam Thursfield2017-07-061-1/+9
| | | | | | | One day BuildStream will be able to run host-incompatible integration commands using a QEMU cross-sandbox, but for now we have to disable integration commands for cross-builds to avoid errors when checking them out.
* Add --host-arch and --target-arch, and 'host-arches' conditionalSam Thursfield2017-07-061-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This command adds initial cross-compilation support to BuildStream. It has been tested against a converted version of the Baserock compiler bootstrap and used to cross build sysroots for armv8l64 and ppc64l from an x86_64 host. For example, to build a sysroot for ARM v8 64-bit you can do this: bst build --target-arch=armv8b64 gnu-toolchain/stage2.bst This would cause the adapted Baserock definitions to produce a stage1 simple cross compiler that runs on the native architecture and produces armv8b64 binaries, and then cross build a stage2 sysroot that executes on armv8b64. Currently the --host-arch option does nothing of use. It will one day enable host-incompatible builds using a QEMU-powered cross sandbox. The `--arch=` option is now shorthand for `--host-arch= --target-arch=`. Elements have 2 new variables available, %{bst-host-arch} and %{bst-target-arch}. The 'arches' conditional now follows %{bst-target-arch}, while the new 'host-arches' conditional follows %{bst-host-arch}. All of --arch, --host-arch and --target-arch default to the output of `uname -a`. There's no magic here that would make all BuildStream elements suddenly able to cross compile. It is up to an individual element to support this by honouring %{bst-target-arch} in whatever way makes sense.
View Lorry Controller Status