<feed xmlns='http://www.w3.org/2005/Atom'>
<title>delta/buildstream.git/tests/artifactcache/expiry.py, branch bschubert/loader</title>
<subtitle>gitlab.com: buildstream/buildstream.git
</subtitle>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/buildstream.git/'/>
<entry>
<title>_artifactcache.py: Don't require the quota to be available on disk.</title>
<updated>2019-01-25T18:35:36+00:00</updated>
<author>
<name>Tristan Van Berkom</name>
<email>tristan.vanberkom@codethink.co.uk</email>
</author>
<published>2019-01-24T21:02:34+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/buildstream.git/commit/?id=22b3a0c182916b8eacb666aa7bc1ea829cd92ca4'/>
<id>22b3a0c182916b8eacb666aa7bc1ea829cd92ca4</id>
<content type='text'>
Instead only rely on the headroom to be enough to protect against
out of space conditions. The headroom can become configurable as
a separate step is required.

The changes to achieve this are:

  * Rename ArtifactCache.has_quota_exceeded() to ArtifactCache.full().

  * ArtifactCache.full() now also reports True if the available
    space on the artifact cache volume is smaller than the headroom.

    This ensures jobs get triggered to cleanup the cache when
    reaching the end of the disk.

  * When loading the artifact quota, it is now only an error if
    the quota exceeds the overall disk space, not if it does not
    fit in the available space.

    It is still a warning if the quota does not fit in the
    available space on the artifact cache volume.

  * Updated scheduler.py and buildqueue.py for the API rename

  * tests: Updated the artifactcache/expiry.py test for its
    expectations in this regard.

    Added a new test to test an error when quota was specified to
    exceed total disk space, and adjusted the existing tests to
    expect a warning when the quota does not fit in the available
    space.

This fixes issue #733 and #869.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Instead only rely on the headroom to be enough to protect against
out of space conditions. The headroom can become configurable as
a separate step is required.

The changes to achieve this are:

  * Rename ArtifactCache.has_quota_exceeded() to ArtifactCache.full().

  * ArtifactCache.full() now also reports True if the available
    space on the artifact cache volume is smaller than the headroom.

    This ensures jobs get triggered to cleanup the cache when
    reaching the end of the disk.

  * When loading the artifact quota, it is now only an error if
    the quota exceeds the overall disk space, not if it does not
    fit in the available space.

    It is still a warning if the quota does not fit in the
    available space on the artifact cache volume.

  * Updated scheduler.py and buildqueue.py for the API rename

  * tests: Updated the artifactcache/expiry.py test for its
    expectations in this regard.

    Added a new test to test an error when quota was specified to
    exceed total disk space, and adjusted the existing tests to
    expect a warning when the quota does not fit in the available
    space.

This fixes issue #733 and #869.
</pre>
</div>
</content>
</entry>
<entry>
<title>_artifactcache.py: Refactored to use utils._get_volume_size()</title>
<updated>2019-01-24T18:02:08+00:00</updated>
<author>
<name>Tristan Van Berkom</name>
<email>tristan.vanberkom@codethink.co.uk</email>
</author>
<published>2019-01-20T17:52:16+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/buildstream.git/commit/?id=9fd9fbb99fe54fda7599c39c014727a157c9ee2a'/>
<id>9fd9fbb99fe54fda7599c39c014727a157c9ee2a</id>
<content type='text'>
This will benefit from a better UtilError being raised, and
and turns the artifact cache's local function into a one liner.

The loop which finds the first existing directory in the
given path has been removed, being meaningless due to the
call to os.makedirs() in ArtifactCache.__init__().

The local function was renamed to _get_cache_volume_size() and
no longer takes any arguments, which is more suitable for the
function as it serves as a testing override surface for
unittest.mock().

The following test cases which use the function to override
the ArtifactCache behavior have been updated to use the new
overridable function name:

   tests/artifactcache/cache_size.py
   tests/artifactcache/expiry.py
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
This will benefit from a better UtilError being raised, and
and turns the artifact cache's local function into a one liner.

The loop which finds the first existing directory in the
given path has been removed, being meaningless due to the
call to os.makedirs() in ArtifactCache.__init__().

The local function was renamed to _get_cache_volume_size() and
no longer takes any arguments, which is more suitable for the
function as it serves as a testing override surface for
unittest.mock().

The following test cases which use the function to override
the ArtifactCache behavior have been updated to use the new
overridable function name:

   tests/artifactcache/cache_size.py
   tests/artifactcache/expiry.py
</pre>
</div>
</content>
</entry>
<entry>
<title>tests/artifactcache/expiry.py: Test that expiry happens first</title>
<updated>2019-01-24T16:55:24+00:00</updated>
<author>
<name>Tristan Van Berkom</name>
<email>tristan.vanberkom@codethink.co.uk</email>
</author>
<published>2019-01-20T00:31:41+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/buildstream.git/commit/?id=fdb8ff650e4c1dd274463a7cb2ef5708bcd096b7'/>
<id>fdb8ff650e4c1dd274463a7cb2ef5708bcd096b7</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>tests/artifactcache/expiry.py: Refactored to use get_element_states()</title>
<updated>2019-01-24T05:01:55+00:00</updated>
<author>
<name>Tristan Van Berkom</name>
<email>tristan.vanberkom@codethink.co.uk</email>
</author>
<published>2019-01-22T16:22:40+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/buildstream.git/commit/?id=6ecc2b0ab076fab452df4e0f0b8091c1fe89ef3a'/>
<id>6ecc2b0ab076fab452df4e0f0b8091c1fe89ef3a</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>_artifactcache.py: Raise ArtifactError() when quota size exceeds disk space.</title>
<updated>2019-01-23T13:57:53+00:00</updated>
<author>
<name>Tristan Van Berkom</name>
<email>tristan.vanberkom@codethink.co.uk</email>
</author>
<published>2019-01-19T15:41:57+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/buildstream.git/commit/?id=9be316419923997932c2f176b592de4d18210604'/>
<id>9be316419923997932c2f176b592de4d18210604</id>
<content type='text'>
This is not an error related to loading data, like a parse error
in the quota specification is, but a problem raised by the artifact
cache - this allows us to assert more specific machine readable
errors in test cases (instead of checking the string in stderr, which
this patch also fixes).

This also removes a typo from the error message in the said error.

  * tests/artifactcache/cache_size.py

    Updated test case to expect the artifact error, which consequently
    changes the test case to properly assert a machine readable error
    instead of asserting text in the stderr (which is the real, secret
    motivation behind this patch).

  * tests/artifactcache/expiry.py: Reworked test_invalid_cache_quota()

    Now expect the artifact error for the tests which check configurations
    which create caches too large to fit on the disk.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
This is not an error related to loading data, like a parse error
in the quota specification is, but a problem raised by the artifact
cache - this allows us to assert more specific machine readable
errors in test cases (instead of checking the string in stderr, which
this patch also fixes).

This also removes a typo from the error message in the said error.

  * tests/artifactcache/cache_size.py

    Updated test case to expect the artifact error, which consequently
    changes the test case to properly assert a machine readable error
    instead of asserting text in the stderr (which is the real, secret
    motivation behind this patch).

  * tests/artifactcache/expiry.py: Reworked test_invalid_cache_quota()

    Now expect the artifact error for the tests which check configurations
    which create caches too large to fit on the disk.
</pre>
</div>
</content>
</entry>
<entry>
<title>Mark 'old' checkout command as obsolete</title>
<updated>2019-01-22T12:32:43+00:00</updated>
<author>
<name>James Ennis</name>
<email>james.ennis@codethink.com</email>
</author>
<published>2019-01-09T12:36:02+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/buildstream.git/commit/?id=fbd159390e97a26f85bb7e8d6284b14bb77083ff'/>
<id>fbd159390e97a26f85bb7e8d6284b14bb77083ff</id>
<content type='text'>
This commit marks 'bst checkout' as a 'hidden' command. If used,
the user will be prompted to use the new 'bst artifact checkout'
command.

All tests which used 'bst checkout' have been modified to use
the new artifact sub-command.

This partially solves #822.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
This commit marks 'bst checkout' as a 'hidden' command. If used,
the user will be prompted to use the new 'bst artifact checkout'
command.

All tests which used 'bst checkout' have been modified to use
the new artifact sub-command.

This partially solves #822.
</pre>
</div>
</content>
</entry>
<entry>
<title>tests/artifactcache/expiry.py: Test refs directory cleanup</title>
<updated>2019-01-18T19:36:26+00:00</updated>
<author>
<name>Tristan Van Berkom</name>
<email>tristan.vanberkom@codethink.co.uk</email>
</author>
<published>2019-01-18T17:42:13+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/buildstream.git/commit/?id=c536ab6a859f551078c0ea07a3a20fdd9851276a'/>
<id>c536ab6a859f551078c0ea07a3a20fdd9851276a</id>
<content type='text'>
Enhance the test which checks removal of the extract directories
with an additional check that the ref directories are cleaned up
when removing artifacts.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Enhance the test which checks removal of the extract directories
with an additional check that the ref directories are cleaned up
when removing artifacts.
</pre>
</div>
</content>
</entry>
<entry>
<title>_cas: Rename artifactcache folder and move that to a root module</title>
<updated>2019-01-16T11:55:07+00:00</updated>
<author>
<name>Raoul Hidalgo Charman</name>
<email>raoul.hidalgocharman@codethink.co.uk</email>
</author>
<published>2018-12-07T17:29:21+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/buildstream.git/commit/?id=2683f98ae2aa90e3f23c2750224fe8a7410f336c'/>
<id>2683f98ae2aa90e3f23c2750224fe8a7410f336c</id>
<content type='text'>
Other components will start to reply on cas modules, and not the artifact cache
modules so it should be organized to reflect this.

All relevant imports have been changed.

Part #802
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Other components will start to reply on cas modules, and not the artifact cache
modules so it should be organized to reflect this.

All relevant imports have been changed.

Part #802
</pre>
</div>
</content>
</entry>
<entry>
<title>Mock storage space checks for tests.</title>
<updated>2018-12-13T10:24:11+00:00</updated>
<author>
<name>Benjamin Schubert</name>
<email>bschubert15@bloomberg.net</email>
</author>
<published>2018-08-10T17:10:13+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/buildstream.git/commit/?id=d03bf31619f739cb85ece06eb3e94a314441e097'/>
<id>d03bf31619f739cb85ece06eb3e94a314441e097</id>
<content type='text'>
Fix #530

- Extract free space computation in a function for easier
mocking
- Mock space computation during cache quota tests
- Mock cache size during cachque quota tests
- Add two more tests when the configuration would require
to much storage space
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Fix #530

- Extract free space computation in a function for easier
mocking
- Mock space computation during cache quota tests
- Mock cache size during cachque quota tests
- Add two more tests when the configuration would require
to much storage space
</pre>
</div>
</content>
</entry>
<entry>
<title>tests/artifactcache/expiry.py: Added test_never_delete_required_track()</title>
<updated>2018-09-14T12:07:46+00:00</updated>
<author>
<name>Tristan Van Berkom</name>
<email>tristan.vanberkom@codethink.co.uk</email>
</author>
<published>2018-09-14T11:08:11+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/buildstream.git/commit/?id=20b797cbf015be982b67e36709173941fbc16871'/>
<id>20b797cbf015be982b67e36709173941fbc16871</id>
<content type='text'>
Same test as test_never_delete_required(), except that this test ensures
that we never delete required artifacts when their cache keys are
discovered dynamically during the build.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Same test as test_never_delete_required(), except that this test ensures
that we never delete required artifacts when their cache keys are
discovered dynamically during the build.
</pre>
</div>
</content>
</entry>
</feed>
