summaryrefslogtreecommitdiff
path: root/buildstream/plugins/sources
Commit message (Collapse)AuthorAgeFilesLines
* Fixup refs to 'bst track'Angelos Evripiotis2019-01-286-6/+6
| | | | | Now that 'bst track' is obsolete, change guidance to refer to the replacement 'bst source track' instead.
* bzr source plugins: Remove all of the atomic backup directory business logicTristan Van Berkom2019-01-181-74/+18
| | | | | Follow up of last commit which uses exclusive locking to protect bzr operations instead.
* bzr source plugin: Use lock files to avoid corrupting the source cacheTristan Van Berkom2019-01-181-6/+29
| | | | This patch by itself fixes #868
* _downloadablefilesource.py: Avoid crashes when HOME is unsetTristan Van Berkom2018-12-261-1/+7
| | | | | | The python netrc module will raise OSError in the case that HOME is not set, this was discovered while running tests under tox.
* Expose base class for Git source pluginsChandan Singh2018-12-201-644/+3
| | | | | | | | | | | | | | | | | | | Add a `_GitSourceBase` class to act as the base class for deriving source plugins that work with Git. The first user of this base class is the `git` source plugin that's in core at the moment. The contents of this base class is almost identical to the existing `GitSource` class. The two notable differences are: - the private methods now have leading underscores, as some were missing it originally - the name of the class Note that we are exposing a private member here as we expect it to move to a separate package soon. See the following discussion for more details: https://gitlab.com/BuildStream/buildstream/issues/739#note_124819869 Fixes #739.
* Force updating tags when fetching git repositoryvalentindavid/git_force_fetch_tagsValentin David2018-12-111-1/+1
| | | | | | | | | When using aliases there are multiple remotes used in the cache repository. When fetching, tags are not updated if the were previously fetched from a different remote. Commits that not in a branch and only tagged do not get fetched if the tag is not fetched. Fixes #812
* git source plugin: Emmit the ref-not-in-track warning from validate_cache()Tristan Van Berkom2018-12-061-37/+30
| | | | | | | | | | Now that we have Source.validate_cache(), this is a better place to emmit the ref-not-in-track warning, since it will be emmitted at the earliest opportunity and not only at Source.stage() or Source.init_workspace(). This also allows us to remove the `self.tracked` local state, and cleanup some convoluted calling paths, removing some unnecessary parameters from the usual codepaths and making the plugin overall more readable.
* git source plugin: Implementing submodule warningsTristan Van Berkom2018-12-061-12/+75
| | | | | | | | | | | | | | | | | o Unlisted submodule warning Now the git plugin will issue a configurable warning if a submodule exists and is used (checking out the submodule is not disabled), but is not specified in the source configuration. o Invalid submodule warning Now the git source plugin will issue a warning if the configuration specified a submodule which does not exist in the underlying git repository. As a side effect, this patch also changes the flow control of the git plugin such that submodules which are explicitly set to not be checked out, are also not fetched but instead ignored completely.
* git source plugin: Rename symbol for consistency and readabilityTristan Van Berkom2018-12-061-2/+3
| | | | | Change 'INCONSISTENT_SUBMODULE' for 'WARN_INCONSISTENT_SUBMODULE' to make things a bit more readable and comprehensive.
* git source plugin: Fixing documentation linking and typoTristan Van Berkom2018-12-061-5/+6
| | | | | | | | | | Fixes the configurable warning part of this to properly link to the project.conf documentation describing what configurable warnings actually are. Also fix the plugin raise the actual configurable warning which it advertizes that it raises; which is 'inconsistent-submodule', not 'inconsistent-submodules'.
* types.py: Moving CoreWarnings to the types.py moduleTristan Van Berkom2018-12-061-2/+1
| | | | | | This also ensures it is exposed via the main buildstream __init__.py file, fixes the imports of CoreWarnings, and adjusts the documentation links to point to the right place.
* git source plugin: Track git tags and save them to reproduce a minimum ↵Valentin David2018-12-051-13/+203
| | | | | | | | | | | | | | | shallow repository Instead of tag information being fetched which can change with time, they are tracked and saved in the projects.refs/.bst. Then we re-tag automatically the closest tag so that `git describe` works and is reproducible. This new feature is opt-in with the new `track-tags` configuration, and must be used to fix modules which are broken by our new policy of omitting the `.git/` repository when staging git sources. This fixes issue #487
* buildstream/plugins/sources/git.py: warn about checkout by defaultJavier Jardón2018-12-031-0/+5
| | | | See #783
* Add support for .netrc in remote/tar/zip pluginsValentin David2018-11-291-1/+79
| | | | Fixes #723.
* local.py: Implement _get_local_path()Jürg Billeter2018-11-271-0/+3
|
* sources/pip.py: Use move_atomic instead of manual os.renameBenjamin Schubert2018-11-191-7/+7
| | | | | This uses move_atomic insteand of the manual os.rename and manual error checking and throws a SourceError for consistency with other modules.
* Extract atomic move function to utils.pyBenjamin Schubert2018-11-191-19/+11
| | | | | | | | Moving atomically a file/directory can be tricky since different errors might be raised for the same underlying problem. Having a utility function to reduce this discrepancies will help in ensuring we have correct behavior
* Fix os.rename in git source element to correctly handle error codesBenjamin Schubert2018-11-191-15/+18
| | | | | | | | | | | | | According to the documentation (https://www.unix.com/man-page/POSIX/3posix/rename/), when the directory already is there, either EEXIST or ENOTEMPTY could be thrown. Previously only ENOTEMPTY was checked. Done: - Separated the move into its own function - Check for both errors - Create unit tests for it, covering most test cases
* plugins/sources/pip.py: also look for python version named "python"Benjamin Schubert2018-11-191-0/+1
| | | | | | | | In some virtual environments, we might not have the binary with the exact python version, which would lead the current pip plugin to fail to find a working pip version. This adds "python" at the start of the list of valid python versions
* plugins/sources/pip.py: Accomodate characters '-','.','_' for packagesctolentino82018-11-021-4/+19
|
* sources: Don't redundantly document directory configurationChandan Singh2018-10-2510-34/+26
| | | | | | | | | | | | Instead of redundantly documenting "directory" configuration option in each source plugin, refer to the built-in functionality documentation of the Source class. * source.py: Add anchor for built-in functionality. * sources/plugins/*.py: Remove directory config option, and add reference to Source's built-in functionality. Fixes #713.
* plugins/source/git.py: Rationalised unused return valuesDaniel Silverstone2018-10-251-2/+2
| | | | | | | The `assert_ref_in_track()` method's return value was never used so rationalise it to never return a value. Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
* _downloadablefilesource: handle ValueError-sAngelos Evripiotis2018-10-251-1/+3
|
* plugins/sources: refactor, rm needless ExitStack-sAngelos Evripiotis2018-10-232-6/+4
| | | | | Remove two uses of ExitStack that aren't necessary and make the code a little harder to follow.
* Spelling fixes: 'affect', not 'effect' as verbAngelos Evripiotis2018-10-191-1/+1
| | | | | Fix most instances of 'effect' being used as a verb instead of a noun. Skipped generated instance in doc/source/conf.py.
* Fix python warnings: Use collections.abc instead collectionsjjardon/warnings_collectionsJavier Jardón2018-10-181-1/+1
| | | | | | | | Since python 3.3, collections has been moved to collections.abc module. For backwards compatibility, they continue to be visible in this module through Python 3.7. Subsequently, they will be removed entirely. See https://docs.python.org/3/library/collections.html
* Fix spelling of it's and itsRichard Dale2018-10-163-4/+4
|
* git.py: Added timed activity around fetching, displaying the URL being fetched.Tristan Van Berkom2018-10-031-4/+12
| | | | This will take care of silencing the status messages while checking submodules.
* git.py: Make `ref` human readableJürg Billeter2018-09-271-2/+24
| | | | Use `git describe` to prefix `ref` with closest tag, if available.
* plugins/git.py: Warn if ref is not in given trackTom Pollard2018-09-031-4/+41
| | | | | | | | | | | Add a helper function assert_ref_in_track to git.py GitMirror() which is used when staging & initing the source workspace. It checks the element's ref exists in the track (branch/tag) if it has been specified, raising a warning if necessary. The warning makes use of the warning token 'REF_NOT_IN_TRACK' from the configurable CoreWarnings. If the element has been tracked with bst, it is assumed that the value of ref exists in the track as it was generated from it & as such is not asserted.
* source.py: Added `primary` argument to URL marking APIsTristan Van Berkom2018-09-021-4/+13
| | | | | | | | | | | | | The Source must now mention whether the marked or translated URL is "primary" or not. Even when a Source may have multiple URLs, the auxilliary URLs are derived from the primary one, not only is this true for git, but it is mandated by our tracking API which assumes there is a primary URL. This adjusts the `git` source and the test `fetch_source.py` source to behave properly and advertize it's primary URL properly. This is a part of #620
* plugins/source/git.py: Fix formatting of url in trackingTristan Van Berkom2018-08-301-1/+3
| | | | | This was displaying the aliased URL which is pretty useless, use the translated URL for the timed activity.
* Fix E741 warningsjjardon/pycodestyle_fixesJavier Jardón2018-08-282-5/+5
|
* Add Error to git and ostree sources configureWilliam Salmon2018-08-172-0/+13
| | | | | | | | | Raise a error at configure time if the track and ref properties are not present in the sources. This is to address https://gitlab.com/BuildStream/buildstream/issues/471 that documented unhelpful behaviour when tracking git sources. However the issue was also identified in ostree.
* Fix ostree repository mirroringValentin David2018-08-161-10/+20
| | | | | | | | | | | Ostree mirrors were not sharing the same local repository, so it was impossible the request refs from the right local repository when data was fetched from a mirror rather than upstream. Instead of having several repository with one remote each, we now have one repository with several remotes. This fixes #538.
* Add pip source pluginChandan Singh2018-08-151-0/+237
| | | | | | | | | | | `pip` source plugin can stage python packages that are either specified directly in the element definition or picked up from `requirements.txt` from previous sources. In order to support the latter use-case (which is also the primary motivation for this plugin), this plugin requires access to previous sources and hence is an example of a Source Transform source. Also, bump `BST_FORMAT_VERSION` as this patch adds a new core plugin.
* git.py: Add configurable warning for inconsistent-submoduleJosh Smith2018-08-151-1/+10
| | | | This follows the implementation of configurable warnings.
* buildstream/plugins/sources/git.py: Fix clone side effect in getting fetchersValentin David2018-08-141-1/+3
| | | | | | | We now delay refreshing submodules until we have a clone, that is after we have fetched the main fetcher. Fixes #537
* buildstream/plugins/sources/zip.py: Fix non-determism in staging.Valentin David2018-08-121-2/+30
| | | | | Staging could end-up with file with different rights depending on the umask. The extracted files need to get their access rights fixed.
* buildstream/plugins/sources/local.py: Make staging deterministic.Valentin David2018-08-121-1/+25
| | | | | | | | | | | | | | Instead of copying metadata on files staged by local, we manually set mode to 0755 or 0644 depending on whether user execution was enabled on source file. This makes file modes deterministic independently on the way source was distributed. Non-deterministic mode copying all metadata can still be enabled by disable 'deterministic' Boolean configuration on the plugin. Fixes #527.
* Add warning to git track if track and ref are not presentWilliam Salmon2018-08-071-0/+6
| | | | | This is to address https://gitlab.com/BuildStream/buildstream/issues/471 that documented unhelpfull behavour when tracking git sources.
* remote.py: Add support for marking downloaded files executableEd Baunton2018-08-071-2/+17
| | | | | | | | | | Add an optional flag to make files executable after having downloaded them. Instead of leaving the permissioning of downloaded file in remote.py up to the user's umask; expressly set permissions to 0644 or 0755 if executable. Bump format version to 13.
* ostree plugin: validate project pathsTiago Gomes2018-08-021-3/+5
|
* patch plugin: validate project pathsTiago Gomes2018-08-021-9/+4
|
* local plugin: validate project pathsTiago Gomes2018-08-021-7/+5
| | | | | | The autotools example had to be copied over inside of the junction example, as referring to a path outside of the project directory is now disallowed.
* git.py: Handle concurrent download completions properlyTristan Van Berkom2018-07-291-4/+15
| | | | | | | | | | | | | | | | | | Use os.rename() to rename the cloned temporary repository into place in the source cache, and issue a STATUS message when discarding a duplicate clone, in the case where the same repository is cloned twice in parallel. The problem with using shutil.move() is that it will create the source directory in a subdirectory of the destination when the destination exists, so it's behavior depends on whether the destination exists. This shutil.move() behavior has so far hidden the race condition where a duplicate repo is created in a subdirectory, as you need to have three concurrent downloads of the same repo in order to trigger the error. This fixes issue #503
* git.py: Improve mirror supportJonathan Maw2018-07-271-45/+49
| | | | | | | | | | | | When implementing fetching from mirrors, I encountered some problems with the git source: 1. The mirror URL was using translate_url()'s output, so if a different alias was used, then fetching from the mirror would go to a different directory, and be inaccessible. 2. After fixing that, fetching was unable to pull from a URL other than the one used at repository creation, meaning it wouldn't actually pull from the mirror.
* bzr.py: Improve mirror supportJonathan Maw2018-07-271-4/+7
| | | | | | | This fixes: * Bzr repositories pulling from the branch they were created with. * Bzr's _ensure_mirror() not actually checking that it successfully mirrored the ref.
* git.py: Update git plugin to fail temporarily on clone failuresJosh Smith2018-07-271-3/+7
| | | | | The git plugin will now make use of the fail_temporarily parameter to Plugin.call(), allowing failures to trigger a retry.
* _downloadablefilesource.py: Update network errors to be temporaryJosh Smith2018-07-271-2/+2
| | | | This follows the change in 67ecd97a05279a3b7570ad59f05bf0a5973ef04c.
View Lorry Controller Status