summaryrefslogtreecommitdiff
path: root/buildstream/source.py
Commit message (Collapse)AuthorAgeFilesLines
* using_commands.rst: Split out top level commands and subcommand groupsJames Ennis2019-01-221-1/+1
|
* source.py: Fixing new (unnecessary-pass) linter errorTristan Van Berkom2018-12-261-1/+0
|
* source.py: Add new delegate method validate_cache()Tristan Van Berkom2018-12-061-1/+24
| | | | | | | This is guaranteed to be called only once for a given session once the sources are known to be Consistency.CACHED, if source tracking is enabled in the session for this source, then this will only be called if the sources become cached after tracking completes.
* Source: Fix typo in documentation, and make it a link to `bst track`Tristan Van Berkom2018-12-061-1/+2
| | | | | | | Now the commands have documentation linkability so we can link to them when referring to them anywhere in the docs. This reference was previously referring to a bogus non-existent `build-stream track`, maybe this dates way back to the days before we named the frontend `bst`.
* source.py: Add optional _get_local_path() methodJürg Billeter2018-11-271-0/+17
|
* source.py: don't let StopIteration propagate to silence() contextmanagerbschubert/fix-silence-stopiteration-pep-0479Benjamin Schubert2018-11-091-20/+22
| | | | | | As per PEP 0479 (https://www.python.org/dev/peps/pep-0479/), StopIteration thrown in context managers are not valid starting from Python 3.7.
* sources: Don't redundantly document directory configurationChandan Singh2018-10-251-0/+2
| | | | | | | | | | | | Instead of redundantly documenting "directory" configuration option in each source plugin, refer to the built-in functionality documentation of the Source class. * source.py: Add anchor for built-in functionality. * sources/plugins/*.py: Remove directory config option, and add reference to Source's built-in functionality. Fixes #713.
* source.py: Remove unused variableDaniel Silverstone2018-10-251-2/+0
| | | | | | Remove the `context` variable which was unused. Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
* source.py: Remove unused local variableDaniel Silverstone2018-10-251-1/+0
| | | | | | Remove `source_kind` as it was unused. Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
* source.py: Quiet several pylint issues with abstract methodsDaniel Silverstone2018-10-251-6/+6
| | | | | | | | A number of times, abstract methods are called which pylint cannot possibly understand will only ever happen in subclasses which implement the abstract methods. This silences those specific warnings Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
* Fix python warnings: Use collections.abc instead collectionsjjardon/warnings_collectionsJavier Jardón2018-10-181-1/+1
| | | | | | | | Since python 3.3, collections has been moved to collections.abc module. For backwards compatibility, they continue to be visible in this module through Python 3.7. Subsequently, they will be removed entirely. See https://docs.python.org/3/library/collections.html
* Extended the Documentation to cover out of source buildsWilliam Salmon2018-10-161-0/+13
| | | | For issue #512 in Gitlab.
* types.py: Moved Consistency definition here from source.pyTristan Van Berkom2018-10-121-24/+1
| | | | And slightly touch up the doc strings for Consistency and Scope.
* source.py: Silence messages while consuming source fetcherstristan/fix-status-messagesTristan Van Berkom2018-10-031-17/+37
| | | | | | | | | The source fetchers might be a list or a generator, when it is a generator (like the git source does), then we want to ensure that we silence the status messages which might occur as a result of consuming a source fetcher from the generator. This fixes the logs to be less verbose.
* source.py: Fix re-instantiationTristan Van Berkom2018-09-211-8/+34
| | | | | | | | | | This fixes an issue where the re-instantiated Source used with Source mirroring enabled is not completely initialized. Failing to load the ref from the project.refs file for instance, will result in a crash at `fetch` time. This fixes issue #666
* source.py: Track marked URLs and assert they are marked during ↵tristan/source-fetcher-changesTristan Van Berkom2018-09-021-0/+20
| | | | | | | | | | | | | | Plugin.configure() This cannot test for unaliased URLs, as those can be discovered later on outside of user provided element configuration; at least we assert that if an alias was used, we have seen it at load time. This will cause a BUG to occur for a plugin which marks an aliased URL (or attempts to translate one) outside of `Plugin.configure()`, if that URL was not previously seen. This is a part of #620
* source.py: Added `primary` argument to URL marking APIsTristan Van Berkom2018-09-021-12/+24
| | | | | | | | | | | | | The Source must now mention whether the marked or translated URL is "primary" or not. Even when a Source may have multiple URLs, the auxilliary URLs are derived from the primary one, not only is this true for git, but it is mandated by our tracking API which assumes there is a primary URL. This adjusts the `git` source and the test `fetch_source.py` source to behave properly and advertize it's primary URL properly. This is a part of #620
* plugin.py: Added _configure() and _get_configuring() private APIsTristan Van Berkom2018-09-021-1/+1
| | | | | | | | | Keeps track of whether the plugin is currently being configured. Adjusted Element and Source classes to call _configure() in place of calling configure() directly. This is a part of #620
* source.py: Documenting that marking download URLs is mandatoryTristan Van Berkom2018-09-021-6/+34
| | | | | | | A download URL must be interpreted by the core at `Plugin.configure()` time, even if only employed later on. This is a part of #620
* source.py: Fixing docs link formatting to be consistent.Tristan Van Berkom2018-09-021-8/+10
|
* source.py: Document Source.get_source_fetchers() to return an iterableTristan Van Berkom2018-09-021-2/+9
| | | | | | | | Also highlight the fact that the plugin can rely on the fetcher's fetch() method getting called before consuming the next item in the list, which is the magick behavior that the git plugin relies on. This is a part of #620
* source.py: Move Source.mark_download_url() to the public methods.Tristan Van Berkom2018-08-301-13/+13
| | | | | | This was sitting in the section for abstract methods, but this is most definitely not an abstract method to be implemented by Sources.
* source.py: Stylistic changes in Source.__do_fetch()Tristan Van Berkom2018-08-301-3/+15
| | | | | | Added some comments to make the flow easier to follow, and removed an annoying 'success' variabled in favor of a for / else loop statement.
* Disable round-tripping when element is not modifiedValentin David2018-08-291-2/+3
|
* Prevent ValueError on URLs without an aliasJosh Smith2018-08-291-9/+12
| | | | Fixes #597
* Fix broken indentation after tracking.Valentin David2018-08-211-1/+1
| | | | | | | | | Issue was introduced by 171e803f (include directive) and the fix was found courtesy of @Qinusty. This fixes also the include feature. Because elements are to be serialized, the included fragments need to use copy_tree when loaded. Related to #470.
* Add pip source pluginChandan Singh2018-08-151-4/+4
| | | | | | | | | | | `pip` source plugin can stage python packages that are either specified directly in the element definition or picked up from `requirements.txt` from previous sources. In order to support the latter use-case (which is also the primary motivation for this plugin), this plugin requires access to previous sources and hence is an example of a Source Transform source. Also, bump `BST_FORMAT_VERSION` as this patch adds a new core plugin.
* Allow source plugins to access previous sourcesChandan Singh2018-08-151-51/+172
| | | | | | | | | | | | | Source plugin implementations can now specify that they need access to previously staged sources by specifying `BST_REQUIRES_PREVIOUS_SOURCES_TRACK` and/or `BST_REQUIRES_PREVIOUS_SOURCES_FETCH`, corresponding to access at `track` and `fetch` times respectively. Fixes #381. Replaces !505. For relevant discussion, see this discussion: https://gitlab.com/BuildStream/buildstream/merge_requests/505#note_83780747
* Add support for include '(@)' in project.conf and .bst filesValentin David2018-08-021-21/+39
| | | | Fixes #331.
* source.py: Bring down since version for source mirroring APIsTristan Van Berkom2018-07-291-4/+4
| | | | | Since we have now backported this to `bst-1.2`, the APIs have been introduced in 1.2 and not 1.4
* doc: Add documentation for source mirroringJonathan Maw2018-07-271-0/+27
|
* Add mirrored source trackingJonathan Maw2018-07-271-1/+28
|
* source: When fetching, try to fetch from mirrors firstJonathan Maw2018-07-271-2/+116
| | | | | | | | | | | **KLUDGE WARNING**: This involves making the source store its "meta" object so that it's possible to create a copy of the source inside the fetch queue, instead of back when the pipeline was being loaded. This adds the SourceFetcher class, which is intended for sources that fetch from multiple URLs (e.g. the git source and its submodules) Fix when fetching
* source: Store the url aliases or use an overrideJonathan Maw2018-07-271-4/+55
| | | | | | | | | | | | | This is part of a later plan to implement mirroring without forcing everyone to update their sources. We use the expected calls to Source.translate_url() when running Source.configure() to extract the aliases from the URL. Multiple aliases must be extracted because sources exist that may fetch from multiple aliases (for example, git submodules) Later, we want to substitute another URI where the alias normally reads from the project - We accomplish this by re-instantiating the Source with the alias overrides passed as an argument to the constructor.
* element.py, source.py, plugin.py: Adjusting API version since annotations.Tristan Van Berkom2018-07-271-1/+1
| | | | | | Since we backported the temporary/permanent failures patch for #397 into the `bst-1.2` branch, we need to adjust the since versions in master down to 1.2.
* _exceptions.py: Modify BstError API to allow optional retryJosh Smith2018-07-271-2/+3
| | | | | | | | | | | | job.py: Changes to the logic surrounding retry attempts and child process return codes element.py, source.py: ElementError and SourceError also implement this change. These exceptions now have an optional parameter of temporary which defaults to false. This will potentially break backwards compatibility where exceptions were previously raised and a retry was intended. To trigger a retry, one must now raise their SourceError or ElementError with temporary=True. This aims to fix #397.
* Remove shebangs from python filesGökçen Nurlu2018-06-191-1/+0
| | | | Fixes #424
* doc: Overhaul of page names and titlesTristan Van Berkom2018-06-131-1/+1
| | | | | | | | | | | | | | | | o Giving main pages simple word titles This makes the main page: * About * Installing * Using * Reference * Contributing o Now named all rst files with their parent page name as a prefix. o Also changed some titles to make overall consistent titles.
* doc/source/core_framework.rst: Use a toctree instead of linksTristan Van Berkom2018-06-091-2/+2
|
* Load and save junctioned source refs from/to junction.refsTristan Van Berkom2018-04-251-6/+24
| | | | | | | | | | | | o _projectrefs.py: Additional constructor option to choose the base name o _project.py: Load two ProjectRefs objects, one for the junctions o source.py: Load and save junctioned source refs with the appropriate ProjectRefs object o tests: Updated some tests to expect junctions to be stored in junction.refs This fixes issue #361
* source.py, element.py, _pipeline.py: Streamling preflighting.Tristan Van Berkom2018-04-161-0/+9
| | | | | | | | | | | Instead of having the pipeline preflight all sources separately from elements, have the element preflight it's sources. This is in order to simplify the shared code path for the pipeline and the loader to use for instantiating elements. Also updated tests to expect the new ElementError and SourceError instead of the PipelineError which was raised for preflighting before.
* source.py: Adhere to policy on private symbolsTristan Van Berkom2018-04-141-16/+17
| | | | This is a part of issue #285
* source.py: Enhanced documentationTristan Van Berkom2018-04-131-73/+136
| | | | | Added new Abstract Methods section here, and rearranged sources such that abstract methods are on top.
* element.py, source.py: Cleanup how Source cache keys are calculated.Tristan Van Berkom2018-04-111-10/+3
| | | | | | | | Recently after a refactor we kept the Source adding workspace keys to the source keys because, now clean this up to have the workspace key added directly in the Element cache key calculation. This breaks cache keys.
* element.py, source.py: Element consumes the rest of workspace logic.Tristan Van Berkom2018-04-091-37/+13
| | | | And Source no longer has any __workspace handle.
* element.py, source.py: Removing knowledge of assemble scheduling state from ↵Tristan Van Berkom2018-04-091-33/+0
| | | | | | source And considering the workspace related edge cases in Element instead of Source.
* element.py: Manage scheduled tracking state in ElementTristan Van Berkom2018-04-091-16/+0
| | | | | | | | | | | | | | | This removes the scheduled state of tracking from Sources, as this is really an element wide thing. To be consistent with assembly, now this comes with: o Element._schedule_tracking() o Element._tracking_done() o Element.__tracking_scheduled o Element.__tracking_done Updated the TrackQueue() to call Element._tracking_done() similarly to how we have BuildQueue() call Element._assemble_done().
* element.py: Cache source consistency logic and refactor workspace logic into ↵Tristan Van Berkom2018-04-091-10/+0
| | | | | | | Element Workspaces are now element wide, so consistency edge cases must be handled at the element level instead of the source level.
* _project.py: Adhere to policy on private symbolsTristan Van Berkom2018-04-031-7/+7
| | | | | | | | | | | | | | | And adjust all surrounding sources for changed symbols. Additional details: o Added Project.get_shell_config() to report the shell configuration, instead of making those members all public o Moved assertions about project specified format versions required of plugins out of Plugin.__init__, and into Project.create_element() and Project.create_source(), so we can keep more things private This is a part of issue #285
* _context.py: Adhere to policy on private symbolsTristan Van Berkom2018-04-031-3/+3
| | | | | | And adjust all surrounding sources for changed symbols. This is a part of issue #285
View Lorry Controller Status