summaryrefslogtreecommitdiff
path: root/buildstream/source.py
Commit message (Collapse)AuthorAgeFilesLines
* source.py: Silence messages while consuming source fetcherstristan/fix-status-messagesTristan Van Berkom2018-10-031-17/+37
| | | | | | | | | The source fetchers might be a list or a generator, when it is a generator (like the git source does), then we want to ensure that we silence the status messages which might occur as a result of consuming a source fetcher from the generator. This fixes the logs to be less verbose.
* source.py: Fix re-instantiationTristan Van Berkom2018-09-211-8/+34
| | | | | | | | | | This fixes an issue where the re-instantiated Source used with Source mirroring enabled is not completely initialized. Failing to load the ref from the project.refs file for instance, will result in a crash at `fetch` time. This fixes issue #666
* source.py: Track marked URLs and assert they are marked during ↵tristan/source-fetcher-changesTristan Van Berkom2018-09-021-0/+20
| | | | | | | | | | | | | | Plugin.configure() This cannot test for unaliased URLs, as those can be discovered later on outside of user provided element configuration; at least we assert that if an alias was used, we have seen it at load time. This will cause a BUG to occur for a plugin which marks an aliased URL (or attempts to translate one) outside of `Plugin.configure()`, if that URL was not previously seen. This is a part of #620
* source.py: Added `primary` argument to URL marking APIsTristan Van Berkom2018-09-021-12/+24
| | | | | | | | | | | | | The Source must now mention whether the marked or translated URL is "primary" or not. Even when a Source may have multiple URLs, the auxilliary URLs are derived from the primary one, not only is this true for git, but it is mandated by our tracking API which assumes there is a primary URL. This adjusts the `git` source and the test `fetch_source.py` source to behave properly and advertize it's primary URL properly. This is a part of #620
* plugin.py: Added _configure() and _get_configuring() private APIsTristan Van Berkom2018-09-021-1/+1
| | | | | | | | | Keeps track of whether the plugin is currently being configured. Adjusted Element and Source classes to call _configure() in place of calling configure() directly. This is a part of #620
* source.py: Documenting that marking download URLs is mandatoryTristan Van Berkom2018-09-021-6/+34
| | | | | | | A download URL must be interpreted by the core at `Plugin.configure()` time, even if only employed later on. This is a part of #620
* source.py: Fixing docs link formatting to be consistent.Tristan Van Berkom2018-09-021-8/+10
|
* source.py: Document Source.get_source_fetchers() to return an iterableTristan Van Berkom2018-09-021-2/+9
| | | | | | | | Also highlight the fact that the plugin can rely on the fetcher's fetch() method getting called before consuming the next item in the list, which is the magick behavior that the git plugin relies on. This is a part of #620
* source.py: Move Source.mark_download_url() to the public methods.Tristan Van Berkom2018-08-301-13/+13
| | | | | | This was sitting in the section for abstract methods, but this is most definitely not an abstract method to be implemented by Sources.
* source.py: Stylistic changes in Source.__do_fetch()Tristan Van Berkom2018-08-301-3/+15
| | | | | | Added some comments to make the flow easier to follow, and removed an annoying 'success' variabled in favor of a for / else loop statement.
* Disable round-tripping when element is not modifiedValentin David2018-08-291-2/+3
|
* Prevent ValueError on URLs without an aliasJosh Smith2018-08-291-9/+12
| | | | Fixes #597
* Fix broken indentation after tracking.Valentin David2018-08-211-1/+1
| | | | | | | | | Issue was introduced by 171e803f (include directive) and the fix was found courtesy of @Qinusty. This fixes also the include feature. Because elements are to be serialized, the included fragments need to use copy_tree when loaded. Related to #470.
* Add pip source pluginChandan Singh2018-08-151-4/+4
| | | | | | | | | | | `pip` source plugin can stage python packages that are either specified directly in the element definition or picked up from `requirements.txt` from previous sources. In order to support the latter use-case (which is also the primary motivation for this plugin), this plugin requires access to previous sources and hence is an example of a Source Transform source. Also, bump `BST_FORMAT_VERSION` as this patch adds a new core plugin.
* Allow source plugins to access previous sourcesChandan Singh2018-08-151-51/+172
| | | | | | | | | | | | | Source plugin implementations can now specify that they need access to previously staged sources by specifying `BST_REQUIRES_PREVIOUS_SOURCES_TRACK` and/or `BST_REQUIRES_PREVIOUS_SOURCES_FETCH`, corresponding to access at `track` and `fetch` times respectively. Fixes #381. Replaces !505. For relevant discussion, see this discussion: https://gitlab.com/BuildStream/buildstream/merge_requests/505#note_83780747
* Add support for include '(@)' in project.conf and .bst filesValentin David2018-08-021-21/+39
| | | | Fixes #331.
* source.py: Bring down since version for source mirroring APIsTristan Van Berkom2018-07-291-4/+4
| | | | | Since we have now backported this to `bst-1.2`, the APIs have been introduced in 1.2 and not 1.4
* doc: Add documentation for source mirroringJonathan Maw2018-07-271-0/+27
|
* Add mirrored source trackingJonathan Maw2018-07-271-1/+28
|
* source: When fetching, try to fetch from mirrors firstJonathan Maw2018-07-271-2/+116
| | | | | | | | | | | **KLUDGE WARNING**: This involves making the source store its "meta" object so that it's possible to create a copy of the source inside the fetch queue, instead of back when the pipeline was being loaded. This adds the SourceFetcher class, which is intended for sources that fetch from multiple URLs (e.g. the git source and its submodules) Fix when fetching
* source: Store the url aliases or use an overrideJonathan Maw2018-07-271-4/+55
| | | | | | | | | | | | | This is part of a later plan to implement mirroring without forcing everyone to update their sources. We use the expected calls to Source.translate_url() when running Source.configure() to extract the aliases from the URL. Multiple aliases must be extracted because sources exist that may fetch from multiple aliases (for example, git submodules) Later, we want to substitute another URI where the alias normally reads from the project - We accomplish this by re-instantiating the Source with the alias overrides passed as an argument to the constructor.
* element.py, source.py, plugin.py: Adjusting API version since annotations.Tristan Van Berkom2018-07-271-1/+1
| | | | | | Since we backported the temporary/permanent failures patch for #397 into the `bst-1.2` branch, we need to adjust the since versions in master down to 1.2.
* _exceptions.py: Modify BstError API to allow optional retryJosh Smith2018-07-271-2/+3
| | | | | | | | | | | | job.py: Changes to the logic surrounding retry attempts and child process return codes element.py, source.py: ElementError and SourceError also implement this change. These exceptions now have an optional parameter of temporary which defaults to false. This will potentially break backwards compatibility where exceptions were previously raised and a retry was intended. To trigger a retry, one must now raise their SourceError or ElementError with temporary=True. This aims to fix #397.
* Remove shebangs from python filesGökçen Nurlu2018-06-191-1/+0
| | | | Fixes #424
* doc: Overhaul of page names and titlesTristan Van Berkom2018-06-131-1/+1
| | | | | | | | | | | | | | | | o Giving main pages simple word titles This makes the main page: * About * Installing * Using * Reference * Contributing o Now named all rst files with their parent page name as a prefix. o Also changed some titles to make overall consistent titles.
* doc/source/core_framework.rst: Use a toctree instead of linksTristan Van Berkom2018-06-091-2/+2
|
* Load and save junctioned source refs from/to junction.refsTristan Van Berkom2018-04-251-6/+24
| | | | | | | | | | | | o _projectrefs.py: Additional constructor option to choose the base name o _project.py: Load two ProjectRefs objects, one for the junctions o source.py: Load and save junctioned source refs with the appropriate ProjectRefs object o tests: Updated some tests to expect junctions to be stored in junction.refs This fixes issue #361
* source.py, element.py, _pipeline.py: Streamling preflighting.Tristan Van Berkom2018-04-161-0/+9
| | | | | | | | | | | Instead of having the pipeline preflight all sources separately from elements, have the element preflight it's sources. This is in order to simplify the shared code path for the pipeline and the loader to use for instantiating elements. Also updated tests to expect the new ElementError and SourceError instead of the PipelineError which was raised for preflighting before.
* source.py: Adhere to policy on private symbolsTristan Van Berkom2018-04-141-16/+17
| | | | This is a part of issue #285
* source.py: Enhanced documentationTristan Van Berkom2018-04-131-73/+136
| | | | | Added new Abstract Methods section here, and rearranged sources such that abstract methods are on top.
* element.py, source.py: Cleanup how Source cache keys are calculated.Tristan Van Berkom2018-04-111-10/+3
| | | | | | | | Recently after a refactor we kept the Source adding workspace keys to the source keys because, now clean this up to have the workspace key added directly in the Element cache key calculation. This breaks cache keys.
* element.py, source.py: Element consumes the rest of workspace logic.Tristan Van Berkom2018-04-091-37/+13
| | | | And Source no longer has any __workspace handle.
* element.py, source.py: Removing knowledge of assemble scheduling state from ↵Tristan Van Berkom2018-04-091-33/+0
| | | | | | source And considering the workspace related edge cases in Element instead of Source.
* element.py: Manage scheduled tracking state in ElementTristan Van Berkom2018-04-091-16/+0
| | | | | | | | | | | | | | | This removes the scheduled state of tracking from Sources, as this is really an element wide thing. To be consistent with assembly, now this comes with: o Element._schedule_tracking() o Element._tracking_done() o Element.__tracking_scheduled o Element.__tracking_done Updated the TrackQueue() to call Element._tracking_done() similarly to how we have BuildQueue() call Element._assemble_done().
* element.py: Cache source consistency logic and refactor workspace logic into ↵Tristan Van Berkom2018-04-091-10/+0
| | | | | | | Element Workspaces are now element wide, so consistency edge cases must be handled at the element level instead of the source level.
* _project.py: Adhere to policy on private symbolsTristan Van Berkom2018-04-031-7/+7
| | | | | | | | | | | | | | | And adjust all surrounding sources for changed symbols. Additional details: o Added Project.get_shell_config() to report the shell configuration, instead of making those members all public o Moved assertions about project specified format versions required of plugins out of Plugin.__init__, and into Project.create_element() and Project.create_source(), so we can keep more things private This is a part of issue #285
* _context.py: Adhere to policy on private symbolsTristan Van Berkom2018-04-031-3/+3
| | | | | | And adjust all surrounding sources for changed symbols. This is a part of issue #285
* source.py: Remove unused method Source._del_workspace()Tristan Van Berkom2018-04-021-9/+0
| | | | | | | | | | This was never particularly useful, there is no circumstances under which a workspace needs to be deleted, and a cache key invalidated, in the course of a session. A workspace is deleted only atomically as a part of `bst workspace close`, which does not even load a pipeline anymore, so the pipeline state need not be adjusted in this case.
* Make workspaces use objects instead of pipeline helper methodsTristan Maat2018-03-271-72/+18
|
* source.py: Source._save_ref() should return saved stateTristan Van Berkom2018-03-211-0/+5
| | | | So we can report proper processed status from tracking queues
* source.py and loading: Refactoring of origin handlesTristan Van Berkom2018-03-201-9/+5
| | | | | | | | | | | The Source object previously stored the __origin_node, __origin_toplevel and __origin_filename, this is from a time when we did not hold on to the plugin's Provenance object explicitly. Since this information comes from the same place, let's just use Plugin._get_provenance() to derive these values, instead of redundantly carrying them along separately.
* Fix #248 - Support project.refs in the core.Tristan Van Berkom2018-03-201-2/+152
| | | | | | | | | | | | | | | | | | | | | | | | | This adds a new Source.load_ref() API which is technically optional to implement, projects which make use of a project.refs file must only use source plugins which implement the new load_ref() method. * source.py: Added load_ref() API to load a ref from a specified node. This also adds _load_ref() and _save_ref() wrappers which handle the logistics of when to load and save a ref to which location. This also fixes _set_ref() to apply the ref to the node unconditionally, this must be done independantly of whether the ref actually changed. o Modifications to the loading process such that Source now can have access to the element name and source index. o _pipeline.py: Delegate abstract loading of source refs to Source._load_ref() - Print a summarized warning about redundant source references - Assert that one cannot track cross-junction elements without project.refs. o _scheduler/trackqueue.py: Delegate saving refs to Source._save_ref()
* source.py: Document ref accessors to mandate support for None values.Tristan Van Berkom2018-03-201-3/+12
| | | | | | | This is technically an API break, but will be transparant for the vast majority of the current hand full of source implementations which exist at this time. This is a lesser evil than bloating the API with new methods.
* pylint - dealt with superfluous-parens warningJames Ennis2018-03-141-3/+3
|
* Silence messages from Source cache interrogationTristan Van Berkom2018-03-071-1/+5
| | | | | | | | | | | Source interrogation usually involves calling out to host tools to quickly check if a given ref exists. This has however regressed over time when running `bst build --track`. This patch adds a new context manager to silence the messages, and silences messages while calling `Source.get_consistency()` Fixes #280
* Add a 'sources' field to project.conf to override defaultsJonathan Maw2018-02-191-1/+28
| | | | | Equivalent to the 'elements' field, but slightly different because sources don't have accompanying yaml.
* source.py: fix comment typoAngelos Evripiotis2018-02-131-1/+1
|
* source.py: Fix typo in workspace warningJürg Billeter2018-02-131-1/+1
| | | | Fixes #250
* Raise warning if workspaced source's ref is updatedincremental-buildChandan Singh2018-01-251-0/+4
| | | | | | | In case a source with an open workspace is tracked and it's ref gets updated, BuildStream should inform the user that the new ref will not be picked up so long as the workspace is open. To start using the updated ref, the existing workspace will have to be closed.
* Add support for doing incremental buildsChandan Singh2018-01-251-5/+42
| | | | | | | | | | | | | | | | | | | | | | | | | This functionality is only supported for sources which have an open workspace. When such sources are present, the workspace directory will be mounted directly inside the sandbox. As opposed to the default behavior, which is to copy files inside the sandbox. This will save time when building large projects as only those files will need be re-compiled that have been modified during two consecutive builds (assuming the underlying build system supports such behavior). A few things to note regarding this behavior: - If there are any `configure-commands` present, they will run only once for each open workspace. If an element has multiple workspaces and any one of them is opened/closed, they will be executed again on the next run. But, modifying the contents of a workspace will not trigger the `configure-commands` to be executed on the next run. - Workspaced builds still leverage the cache. So, if no changes are made to the workspace, i.e. no files are modified, then it will not force a rebuild. Fixes #192.
View Lorry Controller Status