summaryrefslogtreecommitdiff
path: root/buildstream/_pipeline.py
Commit message (Collapse)AuthorAgeFilesLines
...
* Add --host-arch and --target-arch, and 'host-arches' conditionalSam Thursfield2017-07-061-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This command adds initial cross-compilation support to BuildStream. It has been tested against a converted version of the Baserock compiler bootstrap and used to cross build sysroots for armv8l64 and ppc64l from an x86_64 host. For example, to build a sysroot for ARM v8 64-bit you can do this: bst build --target-arch=armv8b64 gnu-toolchain/stage2.bst This would cause the adapted Baserock definitions to produce a stage1 simple cross compiler that runs on the native architecture and produces armv8b64 binaries, and then cross build a stage2 sysroot that executes on armv8b64. Currently the --host-arch option does nothing of use. It will one day enable host-incompatible builds using a QEMU-powered cross sandbox. The `--arch=` option is now shorthand for `--host-arch= --target-arch=`. Elements have 2 new variables available, %{bst-host-arch} and %{bst-target-arch}. The 'arches' conditional now follows %{bst-target-arch}, while the new 'host-arches' conditional follows %{bst-host-arch}. All of --arch, --host-arch and --target-arch default to the output of `uname -a`. There's no magic here that would make all BuildStream elements suddenly able to cross compile. It is up to an individual element to support this by honouring %{bst-target-arch} in whatever way makes sense.
* _pipeline.py: Restoring good old 0644 on this fileTristan Van Berkom2017-07-051-0/+0
| | | | Seems they keep getting executable !
* _pipeline.py: Fixup an error in previous commit...Tristan Van Berkom2017-07-021-1/+1
| | | | Seems I have been impatient today...
* _pipeline.py: Fixing pep8 errorTristan Van Berkom2017-07-021-2/+4
| | | | Forgot to squash this into the previous commit before pushing.
* _pipeline.py: Use PullQueue instead, and add pulled elements to build summariesTristan Van Berkom2017-07-021-10/+12
|
* _pipeline.py: Use Element.dependencies() instead of _direct_deps()Tristan Van Berkom2017-07-011-2/+2
| | | | | Using the recurse=False argument to Element.dependencies() is equal to using the private Element._direct_deps() API.
* _pipeline.py: Fixed bad except statementTristan Van Berkom2017-06-291-1/+1
| | | | Using OSError instead of the general base Exception class proposed in MR 39.
* Fix #42Tristan Maat2017-06-281-9/+10
| | | | | | - Add a `--directory` option to source-bundle - Remove the `name` argument - Rename the tempdir
* Fix accidental permission changesTristan Maat2017-06-281-0/+0
|
* _pipeline.py: Add source-bundle command APITristan Maat2017-06-281-0/+97
|
* Artifact fetch supportJürg Billeter2017-06-271-2/+8
|
* Artifact push supportJürg Billeter2017-06-271-10/+18
|
* _pipeline.py: Better error handling from Pipeline.deps_elements()Tristan Van Berkom2017-06-211-7/+15
| | | | | | | | | | Raise PipelineError in the case that specified --except elements do not exist in the selected pipeline, instead of allowing an action to continue when the user misspells an element in the --except arguments. Also make the default message for PipelineError an empty string, so that intentionally unclassified errors just print as an empty string.
* _pipeline.py: Avoid mutable default argumentsTristan Van Berkom2017-06-211-1/+5
| | | | | It might not matter in this instance, but we stay away from these as a matter of convention.
* Add --except APITristan Maat2017-06-201-1/+59
|
* _pipeline.py: Assert consistent pipelines in fetch() and build()Tristan Van Berkom2017-06-151-47/+83
| | | | | | | Unless the track option is given for either of them. Also this adds the track option for fetch(), which was previously only availble for build().
* _pipeline.py: Changed checkout API to stage runtime dependenciesTristan Van Berkom2017-06-141-14/+11
| | | | | | Instead of just brutally linking out the artifact, which was already wrong because we risk artifact cache corruption (need to copy files out into the checkout directory, never link).
* _pipeline.py: Added inconsistent initializer argumentTristan Van Berkom2017-05-291-5/+17
| | | | | | | When instantiating in inconsistent mode, the third load stage will force all elements to appear inconsistent, ensuring that cache keys can not be calculated until a later time (e.g. TrackQueue) when the source references are first resolved.
* _pipeline.py: build() method now takes 'track_first' argumentTristan Van Berkom2017-05-281-5/+15
| | | | | When specified, a TrackQueue is placed before the FetchQueue in the build process, allowing one to track and build in one go.
* Scheduler overhaul.Tristan Van Berkom2017-05-281-97/+9
| | | | | | | | | | | | | | | | | | | | | | | | | Mostly code reorganization: o _scheduler.py now in _scheduler/ submodule, split into job.py, queue.py and scheduler.py o _pipeline.py no longer defines the queues, instead they are in the scheduler submodule as trackqueue.py, fetchqueue.py and buildqueue.py o Queue initializers no longer require parameters, instead each queue implementation defines the data which was previously passed to the initializers o The scheduler now has a concept of "tokens" for managing the availability of QueueType.FETCH and QueueType.BUILD tasks. Multiple queues of the same QueueType can coexist (e.g. a TrackQueue and a FetchQueue) in the same pipeline, but still be managed by the same user defined task limit (e.g. --fetchers) o Base Queue class now keeps track of how many elements were successfully processed, how many failed and how many were safely skipped.
* _pipeline.py: Added plan() API to produce the elements of a build planTristan Van Berkom2017-05-271-10/+13
| | | | | And take a deps argument in the Pipeline.fetch() API instead of the old boolean 'needed', to match the API for track()
* _pipeline.py: Rewrite elements during track, instead of at the end.Tristan Van Berkom2017-05-271-11/+16
| | | | | | | | | | | This was originally written to be at the end because we thought it may be possible to have multiple tasks which effect the same file, but this is now untrue since we removed the include feature and the scheduler always runs things at element granularity (one element one file, one element processed at a time). Since it's more desirable to rewrite a tracked source immediately after tracking it, that's what we're doing now.
* _pipeline.py: Count total pipeline elements and number of elements in sessionTristan Van Berkom2017-05-011-5/+13
| | | | | | | | For status display purposes, this should eventually move to the scheduler when recursive pipelines are implemented. Also changed to instantiate queues with their complete names as well as their action names.
* _pipeline.py: Use the reported Scheduler elapsed timeTristan Van Berkom2017-04-301-10/+3
| | | | Instead of managing timers around Scheduler.run()
* _pipeline.py: Fixed checkout to ignore conflict on '.' fileTristan Van Berkom2017-04-031-1/+4
|
* _pipeline.py: Fixed PipelineError problems from last commits.Tristan Van Berkom2017-04-021-1/+9
| | | | | | After deriving from _BstError, PipelineError wanted an unconditional message, solve this by just providing a generic one in PipelineError's constructor.
* _pipeline.py: Added checkout() methodTristan Van Berkom2017-04-021-2/+29
| | | | To checkout the pipeline target artifact content to a directory.
* _pipeline.py: Fixed track() command since project element_path was addedTristan Van Berkom2017-03-291-1/+1
| | | | This one was missed in transition
* _pipeline.py: track command now takes list of elementsTristan Van Berkom2017-03-291-11/+3
| | | | Allow the frontend to pass in which elements should be tracked.
* _pipeline.py: Load elements from the element pathTristan Van Berkom2017-03-221-1/+1
|
* Refactor: Untangling element/source namesTristan Van Berkom2017-03-181-4/+0
| | | | | | | | | | | | | | | | | | | | | | | o The metaelements and metasources now carry the name, the loader resolves source names now. o Element/Source factories dont require a name anymore as they are already in the meta objects o Pipeline no longer composes names o Element.name is now the original project relative filename, this allows plugins to identify that name in their dependencies, allowing one to express configuration which identifies elements by the same name that the user used in the dependencies. o Removed plugin._get_display_name() in favor of the plugin.name o Added Element.normal_name, for the cases where we need to have a normalized name for creating directories and log files o Updated frontend and test cases and all callers to use the new naming
* _pipeline.py: Removed a not very relevant commentTristan Van Berkom2017-03-171-5/+0
|
* element.py: Expose element sources generator function.Tristan Van Berkom2017-03-151-3/+3
| | | | | This can be useful to plugins, even if usually they will be using the stage_sources() convenience function.
* _pipeline.py: Some changes of how the pipeline api works.Tristan Van Berkom2017-03-051-66/+78
| | | | | | | | | | | The pipeline no longer owns the scheduler but is given a scheduler to operate on, this needs to be pushed a little further for recursive pipelines to work with a single application scheduler. Also consequently, this patch addresses a problem in pipeline.fetch() which was failing to filter down to only the elements which needed fetching (was resulting in noop of fetch when actually some elements required a fetch).
* _pipeline.py: Name elements with their filenamesTristan Van Berkom2017-02-271-2/+2
| | | | | This seem a bit more intuitive since dependencies and everything are referred to by their project relative filenames.
* _pipeline.py: Added various ticker callbacks for tracking progress at load timeTristan Van Berkom2017-02-271-6/+31
|
* _pipeline.py: Added writable parameter to proxy through to Loader.load()Tristan Van Berkom2017-02-261-2/+4
|
* _pipeline.py: Added global timed messagesTristan Van Berkom2017-02-241-0/+36
| | | | Outputs the time it takes for the full run.
* _pipeline.py: Handle exit code for fetch & track tasks as well.Tristan Van Berkom2017-02-211-0/+7
|
* _pipeline.py: (Issue #4) Dont mark builds as succesfull when they fail.Tristan Van Berkom2017-02-201-5/+6
|
* _pipeline.py: Bump source consistency level *after* setting a new ref.Tristan Van Berkom2017-01-251-4/+3
|
* _pipeline.py: Rework cached & consistency state cachingTristan Van Berkom2017-01-201-15/+22
| | | | | Only interrogate plugins once, after that manually update states depending on successful build steps.
* _pipeline.py: Updated for new Scheduler semanticsTristan Van Berkom2017-01-201-20/+34
|
* _pipeline.py: Changed all -> needed, also messing with cachesTristan Van Berkom2017-01-191-15/+28
| | | | | | In the fetch and refresh API, fetch/refresh all by default, use a needed flag to specify you only want to fetch/refresh what is needed to complete a build.
* _pipeline.py: Skip some elementsTristan Van Berkom2017-01-191-21/+15
| | | | | Skip cached elements at assemble time, skip consistency cached elements in fetch.
* _pipeline.py: Changed refresh() for track()Tristan Van Berkom2017-01-181-31/+24
| | | | | | | Now we lookup the sources which changed in the master process and only apply the references sent over the process boundary to update the sources in tree, and then re-dump the bst files which needed updating.
* _pipeline.py: Added build() methodTristan Van Berkom2017-01-181-0/+48
|
* _pipeline.py: Use context provided scheduler settingsTristan Van Berkom2017-01-141-5/+2
|
* _pipeline.py: A few changes...Tristan Van Berkom2017-01-121-76/+78
| | | | | | | o Removed Resolver, this is just a function now o Some changes to work with the new scheduler and fetch result plugins by their id using _plugin_lookup()
* _pipeline.py: Implement fetch() and refresh() using the SchedulerTristan Van Berkom2017-01-111-19/+137
| | | | Now use the Scheduler to parallelize source operations.
View Lorry Controller Status