| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
| |
This needs to be added along with the status messages added
to the artifact cache, and this detail diverges from master.
This is because we have not bothered to backport !1071 which
refactors the CASCache to be a delegate object of the ArtifactCache
istead of a derived class - backporting !1071 would allow us to
remove these message handlers because the CAS server and test
fixture only use the CASCache object directly, not the business
logic in the ArtifactCache.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
This needs to be added along with the status messages added
to the artifact cache, and this detail diverges from master.
This is because we have not bothered to backport !1071 which
refactors the CASCache to be a delegate object of the ArtifactCache
istead of a derived class - backporting !1071 would allow us to
remove these message handlers because the CAS server and test
fixture only use the CASCache object directly, not the business
logic in the ArtifactCache.
|
| |
|
|
|
|
| |
Updates the known cache size in the main process while the cleanup
process is ongoing, so that the status indicators update live
while the cleanup happens.
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
Added some useful status messages when:
* Calculating a new artifact cache usage size
* Starting a cleanup
* Finishing a cleanup
Also enhanced messaging about what was cleaned up so far when
aborting a cleanup.
|
| |
|
|
|
| |
This also adds some comments around the main status bar heading
rendering function.
|
| | |
|
| |
|
|
|
|
|
|
| |
A frontend facing API for obtaining usage statistics.
I would have put this on Stream instead, but the Context
seems to be the de facto place for looking up the artifact cache
in general so let's put it here.
|
| | |
|
| |
|
|
|
| |
A simple object which creates a snapshot of current
usage statistics for easy reporting in the frontend.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This is not an error related to loading data, like a parse error
in the quota specification is, but a problem raised by the artifact
cache - this allows us to assert more specific machine readable
errors in test cases (instead of checking the string in stderr, which
this patch also fixes).
This also removes a typo from the error message in the said error.
* tests/artifactcache/cache_size.py
Updated test case to expect the artifact error, which consequently
changes the test case to properly assert a machine readable error
instead of asserting text in the stderr (which is the real, secret
motivation behind this patch).
* tests/artifactcache/expiry.py: Reworked test_invalid_cache_quota()
Now expect the artifact error for the tests which check configurations
which create caches too large to fit on the disk.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This will benefit from a better UtilError being raised, and
and turns the artifact cache's local function into a one liner.
The loop which finds the first existing directory in the
given path has been removed, being meaningless due to the
call to os.makedirs() in ArtifactCache.__init__().
The local function was renamed to _get_cache_volume_size() and
no longer takes any arguments, which is more suitable for the
function as it serves as a testing override surface for
unittest.mock().
The following test cases which use the function to override
the ArtifactCache behavior have been updated to use the new
overridable function name:
tests/artifactcache/cache_size.py
tests/artifactcache/expiry.py
|
| |
|
|
| |
We can streamline this call to os.statvfs() in a few places.
|
| |
|
|
| |
The artifact cache emits messages, and we want to allow that in preflight.
|
| |
|
|
|
|
| |
Now that the platform is independent of the context, explicit
instantiation is no longer required. This avoids issues with platform
instances used across test cases with mismatching context.
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
The artifact cache is no longer platform-specific.
|
| | |
|
| | |
|
| |
|
|
|
|
| |
On systems without user namespace support, elements with unsupported
sandbox config (UID/GID) are now individually tainted, which disables
artifact push.
|
| |
|
|
| |
Unsupported sandbox config (UID/GID) is now reported by the element.
|
| | |
|
| | |
|
| |\
| |
| |
| |
| | |
Backport bzr source plugin race condition to 1.2
See merge request BuildStream/buildstream!1287
|
| | |
| |
| |
| |
| |
| |
| |
| | |
This causes multiple source instances to interact with the same
backing data store at the same time, increasing the likelyhood
of triggering issues around concurrent access.
This more reliably triggers issue #868
|
| | |
| |
| |
| |
| |
| |
| |
| |
| | |
With get_element_state(), you need to invoke BuildStream once
for every element state you want to observe in a pipeline.
The new get_element_states() reports a dictionary with
the element state hashed by element name and is better to use
if you have more than one element to observe the state of.
|
| | |
| |
| |
| |
| | |
Follow up of last commit which uses exclusive locking to
protect bzr operations instead.
|
| |/
|
|
| |
This patch by itself fixes #868
|
| |\
| |
| |
| |
| | |
Backport scheduler fixes to 1.2
See merge request BuildStream/buildstream!1286
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
It was saying "There is not enough space to build the given element.",
this makes me think the error is associated to a specific element, but
this does not make sense to show up in a cleanup task.
Instead say "There is not enough space to complete the build.", which
should be more clear that even after cleaning up there is not enough
space.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This branch makes the following changes:
* jobs/job.py: No longer stores any interested resource list
Jobs are ephemeral again, they only ever exist while they
are running.
* queues/queue.py: Revert to only handling lists of elements
Elements pass through the queues, Queue.harvest_jobs()
replaces Queue.pop_ready_jobs() and now the Queue stops
creating jobs as soon as there are not enough resources
for the job.
Also removed unused `prepare()` abstract method.
* queues/buildqueue.py: Adapt the part where we launch a job
This part needs to be reworked anyway, just touch it up for
now so that it doesnt break with the surrounding changes.
* jobs/{cachesize,cleanup}job.py: Expose uniform complete callback
Allows the scheduler to manage resource deallocation for these
two job completions as a custom thing, at the same phase
that the Queues take care of their own resource deallocation.
* resources.py: No longer has knowledge of the job
Since jobs are ephemeral, they are not a suitable place
to store the resource identifiers, these must be provided
by the callers wherever needed.
Now the main Resources object is owned by the Scheduler
but shared with Queues, each take care of managing the
resources of the jobs they create through the same
resource API.
* scheduler.py: Reverted to only creating jobs on demand
This changes the flow of the scheduler such that whenever
jobs complete, the queues are interrogated for as many
jobs which can run at the moment but not more; and this
completely removes the waiting list.
For the internal cache management jobs, we handle this
with a little state instead of having a waiting list
and only launch when the resources permit it.
By abolishing the scheduler waiting list and creating jobs
on demand, we fix the order of element processing and consequently
fix issue #712.
|
| | |
| |
| |
| | |
This is redundant now that we report it through the JobStatus.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This changes the deepest callback from when a Job completes to
propagate a JobStatus value instead of a simple boolean, and updates
all of the effected code paths which used to receive a boolean
to now handle the JobStatus values.
This further improves the situation for issue #753, as now we avoid
queueing cache size jobs for pull jobs which are skipped.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
When queuing the special cache management related cleanup and
cache size jobs, we now treat these jobs as special and do the
following:
* Avoid queueing a cleanup/cache_size job if one is already queued
We just drop redundantly queued jobs here.
* Ensure that jobs of this type only run one at a time
This could have been done with the Resources mechanics,
however as these special jobs have the same properties and
are basically owned by the Scheduler, it seemed more straight
forward to handle the behaviors of these special jobs together.
This fixes issue #753
|
| | |
| |
| |
| |
| |
| |
| |
| | |
This is not used anywhere outside of the Scheduler, currently
only the Scheduler itself is allowed to queue a job at this level.
If the highlevel business logic for automatic queueing of auxiliary
jobs moves to another location, we can make this public again.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
* utils.py:_kill_process_tree(): Ignore NoSuchProcess errors
These are caused because we issue SIGTERM, and if the process
has not exited after a timeout, we kill it.
* _scheduler/jobs/job.py: Stop handling NoSuchProcess errors here
redundantly, they are already ignored.
It seems that we were ignoring it after sleeping when terminating
tasks from the scheduler... but we were not ignoring it when performing
the same pattern in the `Plugin.call()` -> `utils._call()` path, so
we were still getting these exceptions at termination time from host
tool processes launched by source plugins.
|
| |/
|
|
| |
Remove unneeded cruft.
|
| |\
| |
| |
| |
| | |
.gitlab-ci.yml: Use same version for all image
See merge request BuildStream/buildstream!1282
|
| |/ |
|
| |\
| |
| |
| |
| | |
.gitlab-ci.yml: Add tests for Fedora 30
See merge request BuildStream/buildstream!1281
|
| |/ |
|
| |\
| |
| |
| |
| | |
[1.2] .gitlab-ci.yml: Remove deprecated fedora-27 and add current stable fedora-29
See merge request BuildStream/buildstream!1276
|
| | | |
|
