summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* Cachekey update helper: Document ensuring . in PYTHONPATHrichardmaw/subprocess-PWDRichard Maw2018-09-141-1/+1
| | | | | | | | | | The current directory isn't always in the python module search path, so we have to ensure it is for the script to work. Strictly speaking, the user may already have a modified PYTHONPATH at which point PYTHONPATH=".${PYTHONPATH+:$PYTHONPATH}" is necessary, but it's probably premature to overcomplicate the documentation like that before we discover it's a problem.
* Bump Artifact VersionRichard Maw2018-09-1429-29/+29
| | | | | Since we now set PWD in the environment of builds existing builds may behave differently so must cache differently now.
* sandbox: deduplicate default environment logicRichard Maw2018-09-143-33/+29
|
* Merge branch 'richardmaw/builddir-sockets' into 'master'richardmaw-codethink2018-09-145-0/+77
|\ | | | | | | | | Fix: While caching build artifact: "Cannot extract [path to socket file] into staging-area. Unsupported type." See merge request BuildStream/buildstream!783
| * tests: cover builds that make socketsRichard Maw2018-09-143-0/+63
| |
| * Handle sockets when copying trees into artifactsRichard Maw2018-09-142-0/+14
|/ | | | | We can't include a socket in a CAS tree, but it's mostly meaningless to do so since there can't possibly be a process serving it.
* Merge branch 'richardmaw/element-cache-state-simplify' into 'master'richardmaw-codethink2018-09-131-4/+1
|\ | | | | | | | | Simplify element state by removing `__cached` See merge request BuildStream/buildstream!784
| * element: Remove __cached fieldRichard Maw2018-09-131-4/+1
|/ | | | | This can get out of sync with other two cache states, and we can do without it.
* Merge branch 'chandan/pypi-badge' into 'master'Chandan Singh2018-09-131-0/+3
|\ | | | | | | | | README.rst: Add status badges for PyPI release and Python versions See merge request BuildStream/buildstream!719
| * README.rst: Add status badges for PyPI release and Python versionsChandan Singh2018-09-131-0/+3
|/ | | | | | The first badge will work fine right away while the second badge will show "not found" until a release is made after merging this branch: https://gitlab.com/BuildStream/buildstream/merge_requests/718.
* Merge branch 'jjardon/pip_dependency' into 'master'Javier Jardón2018-09-131-0/+1
|\ | | | | | | | | source/install_source.rst: pip plugin depends on host pip See merge request BuildStream/buildstream!791
| * source/install_source.rst: pip plugin depends on host pipJavier Jardón2018-09-131-0/+1
|/
* Merge branch 'juerg/cas-batch' into 'master'Jürg Billeter2018-09-101-0/+49
|\ | | | | | | | | | | | | _artifactcache/casserver.py: Implement BatchReadBlobs Closes #632 See merge request BuildStream/buildstream!785
| * _artifactcache/casserver.py: Implement BatchReadBlobsJürg Billeter2018-09-101-0/+25
| | | | | | | | Fixes #632.
| * _artifactcache/casserver.py: Implement Capabilities serviceJürg Billeter2018-09-101-0/+24
|/
* Merge branch 'tiagogomes/issue-287' into 'master'Tiago Gomes2018-09-108-27/+140
|\ | | | | | | | | Add validation of configuration variables See merge request BuildStream/buildstream!678
| * Add tests for validating configuration variablesTiago Gomes2018-09-103-0/+101
| |
| * element: validate configuration variablesTiago Gomes2018-09-102-1/+12
| | | | | | | | Ensure that protected variables are not being redefined by the user.
| * docs: document protected variablesTiago Gomes2018-09-102-15/+22
| | | | | | | | | | And remove then from the defaults as they are dynamically set by BuildStream.
| * tests: avoid setting max-jobsTiago Gomes2018-09-101-11/+5
|/ | | | Setting "max-jobs" won't be allowed anymore in a following commit.
* Merge branch 'richardmaw/subprocess-PWD' into 'master'richardmaw-codethink2018-09-103-18/+34
|\ | | | | | | | | Ensure PWD is set in process environment See merge request BuildStream/buildstream!782
| * subprocesses: Ensure PWD is set in process environmentRichard Maw2018-09-103-18/+34
|/ | | | | | | | | | | | | | | Naive getcwd implementations (such as in bash 4.4) can break when bind-mounts to different paths on the same filesystem are present, since the algorithm needs to know whether it's a mount-point to know whether it can trust the inode value from the readdir result or to use stat on the directory. Less naive implementations (such as in glibc) iterate again using stat in the case of not finding the directory because the inode in readdir was wrong, though a Linux-specific implementation could use name_to_handle_at. Letting the command know what directory it is in makes it unnecessary for it to call the faulty getcwd in the first place.
* Merge branch 'tristan/fix-cache-exclusivity' into 'master'Tristan Van Berkom2018-09-1012-146/+198
|\ | | | | | | | | _scheduler/queues: Mark build and pull queue as requiring shared access to the CACHE See merge request BuildStream/buildstream!775
| * _artifactcache: There shalt be only one cache sizeTristan Van Berkom2018-09-108-86/+84
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This does a lot of house cleaning, finally bringing cache cleanup logic to a level of comprehensibility. Changes in this commit include: o _artifactcache/artifactcache.py: _cache_size, _cache_quota and _cache_lower_threshold are now all private variables. get_approximate_cache_size() is now get_cache_size() Added get_quota_exceeded() for the purpose of safely checking if we have exceeded the quota. set_cache_size() now asserts that the passed size is not None, it is not acceptable to set a None size cache anymore. o _artifactcache/cascache.py: No longer set the ArtifactCache 'cache_size' variable violently in the commit() method. Also the calculate_cache_size() method now unconditionally calculates the cache size, that is what it's for. o _scheduler/jobs/cachesizejob.py & _scheduler/jobs/cleanupjob.py: Now check the success status. Don't try to set the cache size in the case that the job was terminated. o _scheduler/jobs/elementjob.py & _scheduler/queues/queue.py: No longer passing around the cache size from child tasks, this happens only explicitly, not implicitly for all tasks. o _scheduler/queues/buildqueue.py & _scheduler/scheduler.py: Use get_quota_exceeded() accessor This is a part of #623
| * element.py: Remove _get_artifact_cache() accessor.Tristan Van Berkom2018-09-104-16/+16
| | | | | | | | | | | | | | | | | | | | | | | | The artifact cache is anyway a singleton, the Element itself has an internal handle on the artifact cache because it is needed very frequently; but an artifact cache is not element specific and should not be looked up by surrounding code on a per element basis. Updated _scheduler/queues/queue.py, _scheduler/queues/buildqueue.py and _scheduler/jobs/elementjob.py to get the artifact cache directly from the Platform
| * _scheduler/jobs/cleanupjob.py: Use Platform.get_platform() public accessorTristan Van Berkom2018-09-101-3/+5
| | | | | | | | | | | | | | | | This was previously poking directly at the Platform._instance. Also, use the name 'artifacts' to hold the artifact cache to be consistent with other parts of the codebase, instead of calling it 'cache'.
| * _scheduler/jobs/cachesizejob.py: Use Platform.get_platform() public accessorTristan Van Berkom2018-09-101-3/+5
| | | | | | | | | | | | | | | | This was previously poking directly at the Platform._instance. Also, use the name 'artifacts' to hold the artifact cache to be consistent with other parts of the codebase, instead of calling it 'cache'.
| * element.py: Remove _get_artifact_size()Tristan Van Berkom2018-09-103-36/+22
| | | | | | | | | | | | | | | | There is no justification to hold onto this state here. Instead, just make `Element._assemble()` return the size of the artifact it cached, and localize handling of that return value in the BuildQueue implementation where the value is observed.
| * _artifactcache/artifactcache.py: Sealing away some the estimated sizeTristan Van Berkom2018-09-104-28/+41
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously, the API contract was to expose the estimated_size variable on the ArtifactCache instance for all to see, however it is only relevant to the ArtifactCache abstract class code. Subclasses were informed to update the estimated_size variable in their calculate_cache_size() implementation. To untangle this and hide away the estimated size, this commit does the following: o Introduces ArtifactCache.compute_cache_size() API for external callers o ArtifactCache.compute_cache_size() calls the abstract method for the CasCache subclass to implement o ArtifactCache.compute_cache_size() updates the private estimated_size variable o All direct callers to ArtifactCache.calculate_cache_size(), have been updated to use the ArtifactCache.compute_cache_size() method instead, which takes care of updating anything local to the ArtifactCache abstract class code (the estimated_size)
| * _artifactcache/artifactcache.py: Removing unused variable.Tristan Van Berkom2018-09-101-1/+0
| | | | | | | | | | | | | | | | | | The ArtifactCache._local variable used to exist in order to use a special hack to allow absolute paths to a remote artifact cache, this was all for the purpose of testing. This has all gone away with the introduction of CAS, leaving behind a stale variable.
| * _scheduler: API fixes; _check_cache_size_real() -> check_cache_size()Tristan Van Berkom2018-09-103-10/+34
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Here we have a very private looking _check_cache_size_real() function which no-one would ever want to call from outside of the _scheduler, especially given it's `_real()` prefix we should look for another outward facing API to use. However this is not private to the scheduler, and is intended to be called by the `Queue` implementations. o Renamed this to check_cache_size() o Moved it to the public API section of the Scheduler object o Added the missing API documenting comment o Also added the missing API documenting comment to the private `_run_cleanup()` callback which runs in response to completion of the cache size calculation job. o Also place the cleanup job logs into a cleanup subdirectory, for better symmetry with the cache_size jobs which now have their own subdirectory
| * _artifactcache: Making public methods publicTristan Van Berkom2018-09-104-32/+35
| | | | | | | | | | | | | | | | | | | | | | The artifact cache provides the following public methods for external callers, but was hiding them away as if they are private. o ArtifactCache.add_artifact_size() o ArtifactCache.set_cache_size() Mark these properly public
| * element.py: Adding missing API documenting commentTristan Van Berkom2018-09-101-0/+7
| |
| * _scheduler/scheduler.py: Make CacheSizeJob() not require the CACHE exclusivelyTristan Van Berkom2018-09-101-1/+0
| | | | | | | | | | | | | | | | | | | | | | This runs after every pull, and does not need the cache exclusively, only the cleanup job requires the cache exclusively. Without this, every time a cache_size job is queued, all pull and build jobs need to complete before cache_size job can run exclusively, which is not good. This is a part of #623
| * _scheduler/resources.py: Set CACHE max resources to 0Tristan Van Berkom2018-09-101-1/+1
| | | | | | | | | | | | | | | | | | | | | | This means there is no cap for shared resource requests. Together with the previous commit, this causes the cleanup job and the pull/build jobs to all require unlimited shared access to the CACHE resource; while only the cleanup job requires exclusive access to the resource. This is a part of #623
| * _scheduler/queues: Mark build and pull queue as requiring shared access to ↵Tristan Van Berkom2018-09-102-2/+2
| | | | | | | | | | | | | | | | | | | | | | the CACHE This is what the whole resource.py thing was created for, the cleanup job must have exclusive access to the cache, while the pull and build jobs which result in adding new artifacts, must only require shared access. This is a part of #623
| * tests/artifactcache/expiry.py: Fix test case expectations.Tristan Van Berkom2018-09-101-0/+19
|/
* Merge branch 'coldtom/autotools-libtool' into 'master'Valentin David2018-09-081-0/+36
|\ | | | | | | | | Upstream freedesktop-sdk autotools config for libtool .la files See merge request BuildStream/buildstream!683
| * Upstream libtool .la file removalThomas Coldrick2018-09-081-0/+36
|/ | | | | | In freedesktop-sdk we add a script to our project.conf to remove libtool .la files from autotools projects after install, this seems like a sensible default, so we're attempting to send it upstream.
* Merge branch 'jmac/remote_execution_client' into 'master'Qinusty2018-09-0725-100/+1249
|\ | | | | | | | | Remote execution client See merge request BuildStream/buildstream!626
| * tests/artifactcache: Add pull unit-testsjmac/remote_execution_clientMartin Blanchard2018-09-071-0/+320
| | | | | | | | https://gitlab.com/BuildStream/buildstream/issues/454
| * tests/artifactcache: Add push unit-testsMartin Blanchard2018-09-072-0/+330
| | | | | | | | https://gitlab.com/BuildStream/buildstream/issues/454
| * tests/artifactcache: Add a simple test projectMartin Blanchard2018-09-077-0/+48
| | | | | | | | https://gitlab.com/BuildStream/buildstream/issues/454
| * qmake.py: Mark this as a BST_VIRTUAL_DIRECTORY‧pluginMartin Blanchard2018-09-071-1/+2
| | | | | | | | https://gitlab.com/BuildStream/buildstream/issues/454
| * meson.py: Mark this as a BST_VIRTUAL_DIRECTORY‧pluginMartin Blanchard2018-09-071-1/+2
| | | | | | | | https://gitlab.com/BuildStream/buildstream/issues/454
| * make.py: Mark this as a BST_VIRTUAL_DIRECTORY‧pluginMartin Blanchard2018-09-071-1/+2
| | | | | | | | https://gitlab.com/BuildStream/buildstream/issues/454
| * cmake.py: Mark this as a BST_VIRTUAL_DIRECTORY pluginMartin Blanchard2018-09-071-1/+2
| | | | | | | | https://gitlab.com/BuildStream/buildstream/issues/454
| * autotools.py: Mark this as a BST_VIRTUAL_DIRECTORY pluginJim MacArthur2018-09-071-1/+2
| | | | | | | | https://gitlab.com/BuildStream/buildstream/issues/454
| * element.py: Switch to SandboxRemote if config option is setJim MacArthur2018-09-071-1/+27
| | | | | | | | https://gitlab.com/BuildStream/buildstream/issues/454
| * element.py: Get the updated virtual directory after runningJim MacArthur2018-09-071-0/+2
| | | | | | | | | | | | | | | | | | Executing run() on a sandbox can now replace the virtual directory, since remote execution returns a potentially different directory rather than an update to the existing one. Call get_virtual_directory() again after running to accout for this. https://gitlab.com/BuildStream/buildstream/issues/454
View Lorry Controller Status