<feed xmlns='http://www.w3.org/2005/Atom'>
<title>delta/buildstream.git, branch sam/symlink-speedup</title>
<subtitle>gitlab.com: buildstream/buildstream.git
</subtitle>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/buildstream.git/'/>
<entry>
<title>utils.py: Wrap calls to os.path.realpath() in an LRU cache</title>
<updated>2018-01-19T15:18:38+00:00</updated>
<author>
<name>Sam Thursfield</name>
<email>sam.thursfield@codethink.co.uk</email>
</author>
<published>2018-01-19T13:56:52+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/buildstream.git/commit/?id=33c0c7ab2f6eaed0bb5cd6d7993e80199605837a'/>
<id>33c0c7ab2f6eaed0bb5cd6d7993e80199605837a</id>
<content type='text'>
The os.path.realpath() function is expensive and we call it many times,
to the point that os.path.realpath() calls make up around 40% of the
total time spent in Element.stage_artifact().

The cleanest way to fix this is with a `functools.lru_cache()` wrapper
that caches recently used values. None of the code in question can be
removed (as the tests added in the previous commit will demonstrate).

I tested this by running `bst shell base/base-system.bst true` in
the GNOME modulesets project.

  o Without this patch there are 240,019 calls os.path.realpath()

  o With this patch there are 10,379 calls to os.path.realpath()

  o If we increase the cache size to 128 items, there are 10,359 calls
    to os.path.realpath().

  o If we reduce the cache size to 32 items, there are 10,426 calls.

  o In all cases the number of *unique* calls is 10,327.

This fixes issue #174.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
The os.path.realpath() function is expensive and we call it many times,
to the point that os.path.realpath() calls make up around 40% of the
total time spent in Element.stage_artifact().

The cleanest way to fix this is with a `functools.lru_cache()` wrapper
that caches recently used values. None of the code in question can be
removed (as the tests added in the previous commit will demonstrate).

I tested this by running `bst shell base/base-system.bst true` in
the GNOME modulesets project.

  o Without this patch there are 240,019 calls os.path.realpath()

  o With this patch there are 10,379 calls to os.path.realpath()

  o If we increase the cache size to 128 items, there are 10,359 calls
    to os.path.realpath().

  o If we reduce the cache size to 32 items, there are 10,426 calls.

  o In all cases the number of *unique* calls is 10,327.

This fixes issue #174.
</pre>
</div>
</content>
</entry>
<entry>
<title>HACKING.rst: Added instructions to properly format commit messages.</title>
<updated>2018-01-16T11:03:38+00:00</updated>
<author>
<name>Tristan Van Berkom</name>
<email>tristan.vanberkom@codethink.co.uk</email>
</author>
<published>2018-01-16T11:03:38+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/buildstream.git/commit/?id=633ef73519c695ccb254e3d5ed17c419f8737c79'/>
<id>633ef73519c695ccb254e3d5ed17c419f8737c79</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>Shorten the warnings raised when remote cache initialization fails</title>
<updated>2018-01-16T07:24:26+00:00</updated>
<author>
<name>Sam Thursfield</name>
<email>sam.thursfield@codethink.co.uk</email>
</author>
<published>2018-01-15T15:25:35+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/buildstream.git/commit/?id=bc0f6cd53403bcf487b8fbc42905a33bac618e42'/>
<id>bc0f6cd53403bcf487b8fbc42905a33bac618e42</id>
<content type='text'>
Before:

    [--:--:--] WARNING Failed to fetch remote refs from ssh://ostree@ostree.baserock.org:22000/cache: BuildStream did not connect successfully to the shared cache ssh://ostree@ostree.baserock.org:22000/cache: SSH error: ssh: connect to host ostree.baserock.org port 22000: Connection refused

After:

    [--:--:--] WARNING Failed to fetch remote refs from ssh://ostree@ostree.baserock.org:22000/cache. ssh: connect to host ostree.baserock.org port 22000: Connection refused
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Before:

    [--:--:--] WARNING Failed to fetch remote refs from ssh://ostree@ostree.baserock.org:22000/cache: BuildStream did not connect successfully to the shared cache ssh://ostree@ostree.baserock.org:22000/cache: SSH error: ssh: connect to host ostree.baserock.org port 22000: Connection refused

After:

    [--:--:--] WARNING Failed to fetch remote refs from ssh://ostree@ostree.baserock.org:22000/cache. ssh: connect to host ostree.baserock.org port 22000: Connection refused
</pre>
</div>
</content>
</entry>
<entry>
<title>Delay import of pkg_resources</title>
<updated>2018-01-16T06:27:57+00:00</updated>
<author>
<name>Gökçen Nurlu</name>
<email>gnurlu1@bloomberg.net</email>
</author>
<published>2018-01-11T17:38:11+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/buildstream.git/commit/?id=fc48f087c918231bbad7f80206b1f85e7acf8d8a'/>
<id>fc48f087c918231bbad7f80206b1f85e7acf8d8a</id>
<content type='text'>
This does not introduce a performance gain alone, but importing pkg_resources
is costly at the moment and can be avoided until it is actually needed.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
This does not introduce a performance gain alone, but importing pkg_resources
is costly at the moment and can be avoided until it is actually needed.
</pre>
</div>
</content>
</entry>
<entry>
<title>Make artifact cache tests work in python3.4</title>
<updated>2018-01-12T17:50:53+00:00</updated>
<author>
<name>Jonathan Maw</name>
<email>jonathan.maw@codethink.co.uk</email>
</author>
<published>2018-01-12T17:43:32+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/buildstream.git/commit/?id=571406d44dd1aafa8f1518a14fc23e3f111145cd'/>
<id>571406d44dd1aafa8f1518a14fc23e3f111145cd</id>
<content type='text'>
A LocalPath was being passed into os.path.isabs, which failed when it
tried to call the path's startswith method.

Converting it to a string before using _yaml.dump seemed to solve the
problem.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
A LocalPath was being passed into os.path.isabs, which failed when it
tried to call the path's startswith method.

Converting it to a string before using _yaml.dump seemed to solve the
problem.
</pre>
</div>
</content>
</entry>
<entry>
<title>element.py: Reuse dict for cache key calculations</title>
<updated>2018-01-12T10:21:57+00:00</updated>
<author>
<name>Jürg Billeter</name>
<email>j@bitron.ch</email>
</author>
<published>2018-01-08T18:33:13+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/buildstream.git/commit/?id=880ece190f54ad29dbcd359f84b87c95dcf0536f'/>
<id>880ece190f54ad29dbcd359f84b87c95dcf0536f</id>
<content type='text'>
Do not recalculate the shared part of cache keys. In particular, do not
call get_unique_key() multiple times for a single element or source.

Fixes #167
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Do not recalculate the shared part of cache keys. In particular, do not
call get_unique_key() multiple times for a single element or source.

Fixes #167
</pre>
</div>
</content>
</entry>
<entry>
<title>Use explicit element state updates</title>
<updated>2018-01-12T10:21:57+00:00</updated>
<author>
<name>Jürg Billeter</name>
<email>j@bitron.ch</email>
</author>
<published>2017-12-21T16:10:56+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/buildstream.git/commit/?id=bc492fa8f3973367c3817c84064629f3975b22bd'/>
<id>bc492fa8f3973367c3817c84064629f3975b22bd</id>
<content type='text'>
This adds the _update_state() method to the Element class to keep track
of element state and avoid calculating the same cache key multiple
times. This also consolidates the different get_cache_key
methods into a single method that always returns the cache key
calculated by _update_state(), if available.

Fixes #149, #173
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
This adds the _update_state() method to the Element class to keep track
of element state and avoid calculating the same cache key multiple
times. This also consolidates the different get_cache_key
methods into a single method that always returns the cache key
calculated by _update_state(), if available.

Fixes #149, #173
</pre>
</div>
</content>
</entry>
<entry>
<title>buildelement.py: Remove secret undocumented command lists</title>
<updated>2018-01-12T08:33:59+00:00</updated>
<author>
<name>Tristan Van Berkom</name>
<email>tristan.vanberkom@codethink.co.uk</email>
</author>
<published>2018-01-12T07:57:57+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/buildstream.git/commit/?id=170a9d469a36337990a324d4be2b5c42306b1b13'/>
<id>170a9d469a36337990a324d4be2b5c42306b1b13</id>
<content type='text'>
Originally this was created with also `bootstrap-commands` and
`test-commands` but these were never documented or used.

For `bootstrap-commands`, these originated in baserock but are
basically a part of the `configure-commands` stage, prepending
commands to `configure-commands` is more suitable here since we
now have the prepend/append list directives.

For `test-commands`, these were never used and it's unclear at
this time if it's the correct place for it. It would be interesting
to implement `test-commands` as a separate operation which can run
in parallel with reverse dependency builds (no need to block the
build of a reverse dependency on failing tests, we can still fail
the build as a whole based on a failing test without blocking
builds).
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Originally this was created with also `bootstrap-commands` and
`test-commands` but these were never documented or used.

For `bootstrap-commands`, these originated in baserock but are
basically a part of the `configure-commands` stage, prepending
commands to `configure-commands` is more suitable here since we
now have the prepend/append list directives.

For `test-commands`, these were never used and it's unclear at
this time if it's the correct place for it. It would be interesting
to implement `test-commands` as a separate operation which can run
in parallel with reverse dependency builds (no need to block the
build of a reverse dependency on failing tests, we can still fail
the build as a whole based on a failing test without blocking
builds).
</pre>
</div>
</content>
</entry>
<entry>
<title>NEWS: Added news entry for support for multiple artifact caches.</title>
<updated>2018-01-12T08:05:23+00:00</updated>
<author>
<name>Tristan Van Berkom</name>
<email>tristan.vanberkom@codethink.co.uk</email>
</author>
<published>2018-01-12T08:05:23+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/buildstream.git/commit/?id=36026a72a9458c72df07e33e411d9300eb6f66f2'/>
<id>36026a72a9458c72df07e33e411d9300eb6f66f2</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>Update copyright years of files I have touched</title>
<updated>2018-01-11T18:18:14+00:00</updated>
<author>
<name>Sam Thursfield</name>
<email>sam.thursfield@codethink.co.uk</email>
</author>
<published>2018-01-10T17:25:04+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/buildstream.git/commit/?id=079f702f3695a7197cf5cd2461c55812fc511816'/>
<id>079f702f3695a7197cf5cd2461c55812fc511816</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
</feed>
