summaryrefslogtreecommitdiff
path: root/buildstream/_pipeline.py
Commit message (Collapse)AuthorAgeFilesLines
* project.py: More consistent api for workspacesremove-variantsTristan Van Berkom2017-10-081-1/+1
| | | | | | | | Renamed _workspaces() -> _list_workspaces(), and changed local cache of workspaces from __workspaces -> _workspaces. We only really use the double underscore when there is a concern for freeing up namespace for public subclassing.
* _pipeline.py: No more target variantTristan Van Berkom2017-10-081-7/+4
|
* Add platform factoriesTristan Maat2017-09-281-6/+4
|
* bst push: Check connectivity to cache before trying to pushsam/push-check-connectivitySam Thursfield2017-09-201-0/+2
| | | | | | On slow machines, the actual push operation can spend several minutes preparing before actually pushing, which is annoying if the push then fails.
* Let users override project artifacts optionsMathieu Bridon2017-09-191-1/+2
| | | | | | | | | | | | | | | | | | | A project can specify its artifacts cache sharing settings, and users can define a default artifacts cache to use as a fallback. With this change, users can also override the project configuration with their own. That means for a project named "libfoo", BuildStream will resolve the artifacts-related options in the following order: 1. the projects.libfoo.artifacts options from the user configuration; 2. if the above was not defined, then the artifacts options from the project configuration; 3. if the above was not defined, then the artifacts options from the user configuration; Fixes #87
* Let projects configure their artifacts pull/push optionsMathieu Bridon2017-09-191-1/+1
| | | | | | | With this commit, we first look at the artifacts options in the project configuration, then fall back on the user configuration if necessary. Relates to #87
* Give the various artifact_* attributes to the ArtifactCacheMathieu Bridon2017-09-191-1/+1
| | | | | Having them there will allow overriding the user configuration (the context) with the project configuration.
* Construct pipeline without push queue if cannot push to artifact cacheChandan Singh2017-09-121-8/+9
| | | | | | | | | Currently, if BuildStream is unable to push to the artifact cache for whatever reason, the build will just error out. Fix it so that if we are unable to push to the shared cache, we give a warning to the user and continue building. Fixes https://gitlab.com/BuildStream/buildstream/issues/90
* _pipeline.py: Raise pipeline error in pull/pushTristan Van Berkom2017-09-051-2/+2
| | | | | Instead of trying to fire a MessageType.FAIL without an elapsed parameter, and then not returning from the push()/pull() methods.
* source-bundle: fix filtering out of non-build elementsCharles Bailey2017-09-011-7/+12
| | | | | | Attempting to remove an element from the list being iterated through has undesirable results; fix this by creating a new, filtered list rather than attempting to modify the original list in place.
* workspace open: --no-checkout doesn't need --forceAngelos Evripiotis2017-08-151-1/+1
| | | | | | | | | | Remove the requirement to specify '--force' in conjunction with '--no-checkout' if there are already files in the workspace. We won't write anything when opening the workspace, so there's nothing to force. For example, when opening a workspace to an existing clone of a repository, it seems alarming to have to '--force' the workspace open. It made me wonder if it will actually be overwritten.
* _pipeline.py: Add remote_ticker for artifact list fetchingJürg Billeter2017-08-151-0/+3
|
* _pipeline.py: Fix --deps runJürg Billeter2017-08-101-1/+1
|
* Add `bst pull` commandJürg Billeter2017-08-081-0/+38
|
* Add initial `bst push` commandSam Thursfield2017-08-081-0/+38
| | | | | | | This is mainly useful for testing artifact caches and such. Most users will hopefully be able to make use of artifact caches populated by automated build machines, but right now it's unlikely that most people will be pushing artifacts around.
* Check for write access to remote artifact cache early on in the pipelinesam/artifactcache-preflight-checkSam Thursfield2017-07-211-0/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously, the first time you configured an artifact cache, you would get to the end of your first build and then BuildStream would exit because of some stupid mistake like you got the address slightly wrong or you forgot to add the host keys of the remote artifact cache to `~/.ssh/known_hosts`. To avoid surprises, if there's an artifacts push-url configured we now try to connect to it as a preflight check so that issues are raised early. On success, you will see something like this: [--:--:--][90904fe4][ main:gnu-toolchain/stage2.bst ] START Checking connectivity to remote artifact cache [00:00:00][90904fe4][ main:gnu-toolchain/stage2.bst ] SUCCESS Connectivity OK On failure, it looks like this: [--:--:--][90904fe4][ main:gnu-toolchain/stage2.bst ] START Checking connectivity to remote artifact cache [00:00:03][90904fe4][ main:gnu-toolchain/stage2.bst ] FAILURE BuildStream will be unable to push artifacts to the shared cache: ssh: connect to host ostree.baserock.org port 2220: Connection timed out As a bonus, for some reason this check causes SSH to ask about unknown host keys rather than just failing, so you may now see messages like this if the host keys are unknown rather than an error: The authenticity of host '[ostree.baserock.org]:22200 ([185.43.218.170]:22200)' can't be established. ECDSA key fingerprint is SHA256:mB+MNfYREOdRfp2FG6dceOlguE/Skd4QwnS0tvCPcnI. ECDSA key fingerprint is MD5:8f:fa:ab:90:19:31:f9:f7:f1:d4:e5:f0:a2:be:56:71. Are you sure you want to continue connecting (yes/no)?
* _pipeline.py: Continue if remote artifact repository is unavailableJürg Billeter2017-07-201-2/+6
|
* _pipeline.py: Use _remotely_cached()Jürg Billeter2017-07-201-1/+1
|
* _pipeline.py: Fetch remote refsJürg Billeter2017-07-201-0/+3
|
* _pipeline.py: Reset workspace after deletingTristan Maat2017-07-171-4/+4
|
* _pipeline.py: Add workspace command backendTristan Maat2017-07-171-4/+168
|
* _pipeline.py: Work with new project variantsTristan Van Berkom2017-07-171-5/+11
| | | | | | | | | | | | | | This means: o Load the project elements passing in the possible variants exposed by the project, allowing the loader to error on nonexisting variants and to choose a project variant. o Resolve the project variant after loading the meta elements, but before materializing the Element and Source objects o Delay creation of the plugin contexts and factories until after the project is resolved.
* _pipeline.py: Additional writable check of directory in Pipeline.checkout()Tristan Van Berkom2017-07-131-0/+3
|
* _pipeline.py: Handle exceptions when checking out elementsJonathan Maw2017-07-131-1/+4
| | | | | Previously, an OSError would come up and be handled with a generic stack trace.
* _pipeline.py: Simplified end of session reports.Tristan Van Berkom2017-07-121-23/+8
| | | | | | This was a bit contorted, and is no longer needed with the new end of session generic reports which show the status of each scheduler queue.
* Only run integration commands on checkout for native-built artifactsSam Thursfield2017-07-061-1/+9
| | | | | | | One day BuildStream will be able to run host-incompatible integration commands using a QEMU cross-sandbox, but for now we have to disable integration commands for cross-builds to avoid errors when checking them out.
* Add --host-arch and --target-arch, and 'host-arches' conditionalSam Thursfield2017-07-061-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This command adds initial cross-compilation support to BuildStream. It has been tested against a converted version of the Baserock compiler bootstrap and used to cross build sysroots for armv8l64 and ppc64l from an x86_64 host. For example, to build a sysroot for ARM v8 64-bit you can do this: bst build --target-arch=armv8b64 gnu-toolchain/stage2.bst This would cause the adapted Baserock definitions to produce a stage1 simple cross compiler that runs on the native architecture and produces armv8b64 binaries, and then cross build a stage2 sysroot that executes on armv8b64. Currently the --host-arch option does nothing of use. It will one day enable host-incompatible builds using a QEMU-powered cross sandbox. The `--arch=` option is now shorthand for `--host-arch= --target-arch=`. Elements have 2 new variables available, %{bst-host-arch} and %{bst-target-arch}. The 'arches' conditional now follows %{bst-target-arch}, while the new 'host-arches' conditional follows %{bst-host-arch}. All of --arch, --host-arch and --target-arch default to the output of `uname -a`. There's no magic here that would make all BuildStream elements suddenly able to cross compile. It is up to an individual element to support this by honouring %{bst-target-arch} in whatever way makes sense.
* _pipeline.py: Restoring good old 0644 on this fileTristan Van Berkom2017-07-051-0/+0
| | | | Seems they keep getting executable !
* _pipeline.py: Fixup an error in previous commit...Tristan Van Berkom2017-07-021-1/+1
| | | | Seems I have been impatient today...
* _pipeline.py: Fixing pep8 errorTristan Van Berkom2017-07-021-2/+4
| | | | Forgot to squash this into the previous commit before pushing.
* _pipeline.py: Use PullQueue instead, and add pulled elements to build summariesTristan Van Berkom2017-07-021-10/+12
|
* _pipeline.py: Use Element.dependencies() instead of _direct_deps()Tristan Van Berkom2017-07-011-2/+2
| | | | | Using the recurse=False argument to Element.dependencies() is equal to using the private Element._direct_deps() API.
* _pipeline.py: Fixed bad except statementTristan Van Berkom2017-06-291-1/+1
| | | | Using OSError instead of the general base Exception class proposed in MR 39.
* Fix #42Tristan Maat2017-06-281-9/+10
| | | | | | - Add a `--directory` option to source-bundle - Remove the `name` argument - Rename the tempdir
* Fix accidental permission changesTristan Maat2017-06-281-0/+0
|
* _pipeline.py: Add source-bundle command APITristan Maat2017-06-281-0/+97
|
* Artifact fetch supportJürg Billeter2017-06-271-2/+8
|
* Artifact push supportJürg Billeter2017-06-271-10/+18
|
* _pipeline.py: Better error handling from Pipeline.deps_elements()Tristan Van Berkom2017-06-211-7/+15
| | | | | | | | | | Raise PipelineError in the case that specified --except elements do not exist in the selected pipeline, instead of allowing an action to continue when the user misspells an element in the --except arguments. Also make the default message for PipelineError an empty string, so that intentionally unclassified errors just print as an empty string.
* _pipeline.py: Avoid mutable default argumentsTristan Van Berkom2017-06-211-1/+5
| | | | | It might not matter in this instance, but we stay away from these as a matter of convention.
* Add --except APITristan Maat2017-06-201-1/+59
|
* _pipeline.py: Assert consistent pipelines in fetch() and build()Tristan Van Berkom2017-06-151-47/+83
| | | | | | | Unless the track option is given for either of them. Also this adds the track option for fetch(), which was previously only availble for build().
* _pipeline.py: Changed checkout API to stage runtime dependenciesTristan Van Berkom2017-06-141-14/+11
| | | | | | Instead of just brutally linking out the artifact, which was already wrong because we risk artifact cache corruption (need to copy files out into the checkout directory, never link).
* _pipeline.py: Added inconsistent initializer argumentTristan Van Berkom2017-05-291-5/+17
| | | | | | | When instantiating in inconsistent mode, the third load stage will force all elements to appear inconsistent, ensuring that cache keys can not be calculated until a later time (e.g. TrackQueue) when the source references are first resolved.
* _pipeline.py: build() method now takes 'track_first' argumentTristan Van Berkom2017-05-281-5/+15
| | | | | When specified, a TrackQueue is placed before the FetchQueue in the build process, allowing one to track and build in one go.
* Scheduler overhaul.Tristan Van Berkom2017-05-281-97/+9
| | | | | | | | | | | | | | | | | | | | | | | | | Mostly code reorganization: o _scheduler.py now in _scheduler/ submodule, split into job.py, queue.py and scheduler.py o _pipeline.py no longer defines the queues, instead they are in the scheduler submodule as trackqueue.py, fetchqueue.py and buildqueue.py o Queue initializers no longer require parameters, instead each queue implementation defines the data which was previously passed to the initializers o The scheduler now has a concept of "tokens" for managing the availability of QueueType.FETCH and QueueType.BUILD tasks. Multiple queues of the same QueueType can coexist (e.g. a TrackQueue and a FetchQueue) in the same pipeline, but still be managed by the same user defined task limit (e.g. --fetchers) o Base Queue class now keeps track of how many elements were successfully processed, how many failed and how many were safely skipped.
* _pipeline.py: Added plan() API to produce the elements of a build planTristan Van Berkom2017-05-271-10/+13
| | | | | And take a deps argument in the Pipeline.fetch() API instead of the old boolean 'needed', to match the API for track()
* _pipeline.py: Rewrite elements during track, instead of at the end.Tristan Van Berkom2017-05-271-11/+16
| | | | | | | | | | | This was originally written to be at the end because we thought it may be possible to have multiple tasks which effect the same file, but this is now untrue since we removed the include feature and the scheduler always runs things at element granularity (one element one file, one element processed at a time). Since it's more desirable to rewrite a tracked source immediately after tracking it, that's what we're doing now.
* _pipeline.py: Count total pipeline elements and number of elements in sessionTristan Van Berkom2017-05-011-5/+13
| | | | | | | | For status display purposes, this should eventually move to the scheduler when recursive pipelines are implemented. Also changed to instantiate queues with their complete names as well as their action names.
* _pipeline.py: Use the reported Scheduler elapsed timeTristan Van Berkom2017-04-301-10/+3
| | | | Instead of managing timers around Scheduler.run()
View Lorry Controller Status