summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* artifactcache: add some logging to cleanup operationtiagogomes/cache-calculation-revampTiago Gomes2018-08-132-3/+9
|
* Move quota checking code into a separate methodTiago Gomes2018-08-132-56/+64
| | | | | | | | | | | Instead of running `utils._get_dir_size(self.artifactdir)` to get the size of the cache, it would be better to reuse ArtifactCache.get_cache_size(), otherwise we would have to fully calculate the size of the artifact cache twice at startup. As the ArtifactCache is not yet instantiated when Context.load() runs, move most of the quota checking code into a separate function which can be called after the ArtifactCache is instantiated.
* Revamp calculation of cache size codeTiago Gomes2018-08-1311-140/+51
| | | | | | | | | | | | Before this commit, the cache size was being calculated every time an artifact was committed to or pulled to the artifact cache. This is a very IO demanding operation that doesn't scale as the cache gets bigger. Now that the commit() and pull() methods of the ArtifactCache return the number of bytes added, we can rework the code to calculate the size of the cache only once at startup and to dynamically update it afterwards. The CacheSize job is no longer required and can be removed.
* element: use ArtifactCache.commit() return valueTiago Gomes2018-08-131-2/+2
| | | | | | The ArtifactCache.commit() return value is more accurate for what __artifact_size is being used for, as it takes the deduplication done by the CAS cache into account.
* artifactcache: return bytes required to cache artifactTiago Gomes2018-08-134-37/+63
| | | | | Track number of bytes required to commit or pull an artifact and return this data on pull() and commit(). This takes deduplication into account.
* cascache: be more accurate on amount of data removedTiago Gomes2018-08-131-1/+2
| | | | refpath contains only metadata, but every byte counts.
* cascache: fix oversightTiago Gomes2018-08-131-1/+1
|
* cascache: rename _fetch_directory to _fetch_treeTiago Gomes2018-08-131-3/+3
| | | | For symmetry with _create_tree.
* Merge branch 'valentindavid/ruamel-version' into 'master'Valentin David2018-08-131-1/+1
|\ | | | | | | | | | | | | Set version of ruaml.yaml to at least 0.15.41 but strictly less than 0.15.52. Closes #571 See merge request BuildStream/buildstream!646
| * Set version of ruaml.yaml to strictly less than 0.15.52.Valentin David2018-08-131-1/+1
|/ | | | | | | This allows to use version 0.15.51 rather than 0.15.0 which is required for Python 3. Fixes #571.
* Merge branch 'valentindavid/fix_included_junction_track' into 'master'Tristan Van Berkom2018-08-132-2/+37
|\ | | | | | | | | | | | | Fix tracking of junctions used in project.conf Closes #565 See merge request BuildStream/buildstream!619
| * Fix tracking of junctions used in project.conf.Valentin David2018-08-132-2/+37
|/ | | | | | | | | | | | | | Stream._load() now returns early without resolving build pipeline when only tracking. Resolving track pipelines does not require to fully load project configurations when when elements to track are only junctions. However build pipelines require to fully load project configurations. This might not be possible in the case a project configuration includes a file from a junction that yet needs to be tracked. Fixes #565.
* Merge branch 'chandan/remote-source-cachekey-test' into 'master'Tristan Van Berkom2018-08-136-1/+18
|\ | | | | | | | | remote source: Add cachekey test See merge request BuildStream/buildstream!636
| * remote source: Add cachekey testChandan Singh2018-08-136-1/+18
|/ | | | | Add cachekey tests for the recently added `remote` source plugin to ensure that future changes do not break API compatibility.
* Merge branch 'chandan/update-fedora-install-docs' into 'master'Javier Jardón2018-08-121-1/+2
|\ | | | | | | | | doc: Fix install instructions for Fedora See merge request BuildStream/buildstream!640
| * doc: Fix install instructions for FedoraChandan Singh2018-08-121-1/+2
|/ | | | | | | | | `python3-arpy` package is not available on Fedora so it has to be installed using `pip`. Update instructions accordingly. For reference, here is a pipeline that failed when trying to install `arpy` using current instructions: https://gitlab.com/BuildStream/buildstream-docker-images/-/jobs/88553245.
* Merge branch 'valentindavid/deterministic-source' into 'master'Tristan Van Berkom2018-08-1230-30/+246
|\ | | | | | | | | | | | | Deterministic staging Closes #543, #544, #555, and #527 See merge request BuildStream/buildstream!616
| * Bump BST_CORE_ARTIFACT_VERSION for deterministic source pluginsValentin David2018-08-1226-26/+26
| |
| * Add some integration tests for source plugin determinism.Valentin David2018-08-121-0/+155
| |
| * buildstream/plugins/sources/zip.py: Fix non-determism in staging.Valentin David2018-08-121-2/+30
| | | | | | | | | | Staging could end-up with file with different rights depending on the umask. The extracted files need to get their access rights fixed.
| * Use deterministic umask when staging sources.Valentin David2018-08-121-1/+10
| | | | | | | | | | | | This fix is applied to plugins bzr, git, patch. Fixes #543 #544 #555.
| * buildstream/plugins/sources/local.py: Make staging deterministic.Valentin David2018-08-121-1/+25
|/ | | | | | | | | | | | | | Instead of copying metadata on files staged by local, we manually set mode to 0755 or 0644 depending on whether user execution was enabled on source file. This makes file modes deterministic independently on the way source was distributed. Non-deterministic mode copying all metadata can still be enabled by disable 'deterministic' Boolean configuration on the plugin. Fixes #527.
* Merge branch 'jjardon/ci_fedora28' into 'master'Javier Jardón2018-08-123-12/+12
|\ | | | | | | | | Add job to run tests in fedora 28 See merge request BuildStream/buildstream!643
| * dev-requirements.txt: Remove obsolete commentsJavier Jardón2018-08-122-8/+3
| | | | | | | | | | This also fixes an error with pylint in setup.py: setup.py:226:19: R1718: Consider using a set comprehension (consider-using-set-comprehension)
| * .gitlab-ci.yml: Use the same version of the docker image for all jobsJavier Jardón2018-08-121-4/+4
| |
| * .gitlab-ci.yml: Run tests in fedora28 imageJavier Jardón2018-08-121-0/+5
|/
* Merge branch 'phil/add-ubuntu-ci-job' into 'master'Javier Jardón2018-08-121-0/+3
|\ | | | | | | | | .gitlab-ci-yml: Add ubuntu 18.04 test See merge request BuildStream/buildstream!523
| * .gitlab-ci-yml: Add ubuntu 18.04 testPhil Dawson2018-08-121-0/+3
|/
* Merge branch 'chandan/fix-dev-reqs-typo' into 'master'Chandan Singh2018-08-121-1/+1
|\ | | | | | | | | Fix typo in dev-requirements.txt See merge request BuildStream/buildstream!641
| * Fix typo in dev-requirements.txtChandan Singh2018-08-121-1/+1
|/
* Merge branch 'chandan/dev-requirements' into 'master'Javier Jardón2018-08-113-13/+26
|\ | | | | | | | | Move development reqirements to dev-requirements.txt See merge request BuildStream/buildstream!637
| * Move development reqirements to dev-requirements.txtChandan Singh2018-08-113-13/+26
|/ | | | | | | | | | | | | | | | | | In some cases, such as when working inside a virtual environment, it can be desirable to install all dependencies for running tests using `pip`. This is currently not possible since setuptools does not support installing these dependencies in a virtual environment (by design). (See https://stackoverflow.com/a/21003259.) To circumvent this issue, move such requirements to `dev-requirements.txt` file that can be used easily with `pip install -r`. This also enables tests to be run directly using `pytest`, which can be more convenient than `-addopts` approach when one needs to add multiple options. This will also be useful in creating better testuite images, and fix some of the issues noticed in https://gitlab.com/BuildStream/buildstream-docker-images/merge_requests/56.
* Merge branch 'valentindavid/498_bwrap_environment' into 'master'Valentin David2018-08-101-3/+7
|\ | | | | | | | | | | | | Set environment in bwrap command line instead of its environment Closes #498 See merge request BuildStream/buildstream!565
| * Set environment in bwrap command line instead of its environmentValentin David2018-08-101-3/+7
|/ | | | Fixes #498
* Merge branch 'relative_workspaces' into 'master'Phillip Smyth2018-08-106-54/+97
|\ | | | | | | | | | | | | Patch for issue #191 support relative workspaces Closes #191 See merge request BuildStream/buildstream!504
| * Implementing relative workspacesrelative_workspacesPhillip Smyth2018-08-106-54/+97
|/ | | | | | | This fixes #191 A note has been added to NEWS explaining backwards compatibility issues
* Merge branch 'jmac/pin-ruamel-version' into 'master'Tristan Van Berkom2018-08-091-1/+1
|\ | | | | | | | | Pin ruamel.yaml version to <= 0.15 See merge request BuildStream/buildstream!630
| * setup.py: Pin ruamel.yaml version to <= 0.15Jim MacArthur2018-08-091-1/+1
|/
* Merge branch 'jjardon/max-jobs' into 'master'Javier Jardón2018-08-091-2/+4
|\ | | | | | | | | buildstream/_project.py: Restrict max-jobs See merge request BuildStream/buildstream!620
| * buildstream/_project.py: Restrict max-jobsJavier Jardón2018-08-091-1/+4
| | | | | | | | | | | | even if the machine have mores cores available Patch taken from YBD: https://gitlab.com/baserock/ybd/blob/master/ybd/app.py#L227
| * buildstream/_project.py:Javier Jardón2018-08-091-2/+1
|/ | | | | | | | | Multiprocessing with n_jobs given by multiprocessing.cpu_count() is not optimal for systems where not all CPU can be used (in particular, Docker, CI etc). As suggested in the multiprocessing docs (http://docs.python.org/3/library/multiprocessing.html#multiprocessing.cpu_count) using len(os.sched_getaffinity(0)) might be better.
* Merge branch 'valentindavid/fuse-create-flags' into 'master'Valentin David2018-08-092-3/+7
|\ | | | | | | | | | | | | Keep original flags for create in SafeHardlinks. Closes #143 See merge request BuildStream/buildstream!624
| * Keep original flags for create in SafeHardlinks.Valentin David2018-08-092-3/+7
|/ | | | | | | | | | When open(2) is used with flags O_CREAT|O_RDWR, the file descriptor must be readable. Unfortunately O_RDWR was not passed which made read fail with EBADF and mmap to signal SIGBUS. This issue happened with man-db for example. Fixes #143.
* Merge branch 'tiagogomes/issue-520' into 'master'Tiago Gomes2018-08-093-8/+10
|\ | | | | | | | | Fix race condition when calculating disk usage See merge request BuildStream/buildstream!600
| * cascache: move tmp directory one level uptiagogomes/issue-520Tiago Gomes2018-08-093-8/+10
|/ | | | | | | | | | | | | The CAS uses a temp directory while manipulating the cache, temporary files can be added and removed while adding artifacts to the cache here. Since calculation of the cache size happens in parallel to artifact cache additions, this causes race conditions in the size calculation job, as we end up calling `stat` on temporary files which are being removed in parallel. Handle this by moving the temporary directory out of the way, and avoid considering the tmp directory when calculating the cache size
* Merge branch 'chandan/remove-dead-code' into 'master'Chandan Singh2018-08-081-10/+0
|\ | | | | | | | | .gitlab-ci.yml: Remove unused script "install.sh" See merge request BuildStream/buildstream!617
| * .gitlab-ci.yml: Remove unused script "install.sh"Chandan Singh2018-08-081-10/+0
|/
* Merge branch 'juerg/python3.7' into 'master'Jürg Billeter2018-08-072-4/+6
|\ | | | | | | | | Python 3.7 support See merge request BuildStream/buildstream!615
| * _artifactcache/cascache.py: Fix for PEP 479 / Python 3.7Jürg Billeter2018-08-071-2/+5
| | | | | | | | | | | | Do not rely on `StopIteration` bubbling up. https://www.python.org/dev/peps/pep-0479/
| * Revert "Restrict version of pylint"Jürg Billeter2018-08-071-2/+1
|/ | | | | | | | | This reverts commit 4f168b9b6a02216e2fae24d758ae6b778e545869. The latest version of pytest_pylint works fine with pylint 2, which means there is no longer a reason to restrict the pylint version. pylint 2 is required for Python 3.7.
View Lorry Controller Status