summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* stream.py: Avoid overworking in load_selection where possiblejennis/load_sometimes_overworksJames Ennis2019-06-042-6/+12
| | | | | | | load_selection marks elements in the pipeline as required. In doing this elements need to pass through update state at least another 2 times. For commands which do not invoke the scheduler, we should avoid doing this work.
* _stream.py: Avoid repeated update_state calls for showJames Ennis2019-06-043-7/+15
| | | | | | | This patch ensures that we only need to enter Element._update state() once in order to resolve the cache keys and cached state of the elements in the pipeline.
* Merge branch 'bschubert/optimize-node_get' into 'master'bst-marge-bot2019-06-0411-77/+71
|\ | | | | | | | | Optimize _yaml.node_get See merge request BuildStream/buildstream!1370
| * _yaml: Optimize node_get to not create dummy Nodes unnecessarily.bschubert/optimize-node_getBenjamin Schubert2019-06-031-8/+8
| | | | | | | | | | This reduces considerably the number of nodes created and thus speeds up the calls to node_get.
| * _yaml.pyx: Forbid expected_type=Mapping, and remove isinstance checkBenjamin Schubert2019-06-0311-69/+63
|/ | | | | | | | | | | Calls to `isinstance` can be particularily costly. Using type() is much faster. The only known case where the `isinstance` was useful is for dictionnaries where we would ask for a 'Mapping' instead. Disallowing 'Mapping' for expected_type considerably speeds up the calls to this functions. Also add into NEWS
* Merge branch 'bschubert/fix-cython-trace' into 'master'bst-marge-bot2019-06-031-1/+6
|\ | | | | | | | | setup.py: Be more restrictive with BST_CYTHON_TRACE values See merge request BuildStream/buildstream!1368
| * setup.py: Be more restrictive with BST_CYTHON_TRACE valuesbschubert/fix-cython-traceBenjamin Schubert2019-06-031-1/+6
|/ | | | | | "0" evaluates to 'True' in python, which incorrectly switched on the BST_CYTHON_TRACE. Forcing an int for this environment variable allows us to ensure we are correct.
* Merge branch 'bschubert/cython-disable-linetrace' into 'master'bst-marge-bot2019-05-311-4/+5
|\ | | | | | | | | Setup.py: Disable linetrace by default and only enable when using coverage See merge request BuildStream/buildstream!1367
| * Setup.py: Disable linetrace by default and only enable when using coverageBenjamin Schubert2019-05-311-4/+5
|/ | | | | | | | | It turns out that enabling 'linetrace', does have a runtime cost even if not enabled in distutils. Therefore disabling it by default. In order to run coverage, we therefore need to retranspile the .pyx files with ENABLE_CYTHON_TRACE set.
* Merge branch 'raoul/1024-artifact-docs' into 'master'bst-marge-bot2019-05-316-39/+118
|\ | | | | | | | | | | | | Update docs regarding artifact and source caches Closes #1024 See merge request BuildStream/buildstream!1362
| * docs: Update configuring cache serverraoul/1024-artifact-docsRaoul Hidalgo Charman2019-05-314-33/+40
| | | | | | | | | | | | | | | | Now that we have both artifact and source caches the documentation is updated to reflect that. Some sections headings/links etc. have been expanded and changed. Part of #1025
| * doc: Add architecture section on cachesRaoul Hidalgo Charman2019-05-313-0/+71
| | | | | | | | Part of #1024
| * docs: update to reflect artifact as a protoRaoul Hidalgo Charman2019-05-311-6/+7
|/ | | | Part of #1024
* Merge branch 'danielsilverstone-ct/iterative-loader-bits' into 'master'bst-marge-bot2019-05-302-27/+65
|\ | | | | | | | | Make more parts of the loader iterative See merge request BuildStream/buildstream!1365
| * _loader/loade{r,lement}.py: Do not re-collect already collected elementsdanielsilverstone-ct/iterative-loader-bitsDaniel Silverstone2019-05-302-4/+12
| | | | | | | | | | | | | | When multiple top level elements are specified we need to keep track of whether we have completed the iterative collection process. Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
| * loader.py: Make _collect_element() iterativeDaniel Silverstone2019-05-301-14/+41
| | | | | | | | | | | | | | To reduce stack usage during load, make the LoadElement to MetaElement conversion be iterative. Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
| * loader.py: Make _sort_dependencies() a static iterative methodDaniel Silverstone2019-05-301-10/+13
|/ | | | | | | | The _sort_dependencies() method does not need to be an instance method, nor does it need to be recursive. This fixes both of those things which can get us closer to being able to cythonize the loader. Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
* Merge branch 'danielsilverstone-ct/iterative-circdeps' into 'master'bst-marge-bot2019-05-301-34/+39
|\ | | | | | | | | Rewrite `Loader._check_circular_deps()` to be iterative See merge request BuildStream/buildstream!1364
| * loader.py: Rewrite _check_circular_deps() to be iterativeDaniel Silverstone2019-05-301-34/+38
| | | | | | | | | | | | | | | | In an effort to reduce the places where the stack might be a problem as we Cythonize things, rewrite the circular dependency checker to be iterative. Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
| * loader.py: Make _check_circular_deps() staticDaniel Silverstone2019-05-301-2/+3
|/ | | | | | | | This did not need to be an instance method, making it static might improve performance and definitely makes it clear that it's not actually bound to the loader instances. Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
* Merge branch 'bschubert/cython' into 'master'bst-marge-bot2019-05-3015-357/+725
|\ | | | | | | | | Introduce Cython for better performances See merge request BuildStream/buildstream!1350
| * _yaml: Replace strings by a C enum for Representer states.bschubert/cythonBenjamin Schubert2019-05-301-44/+57
| | | | | | | | | | This allows for a quicker comparison while keeping a good readability of the code
| * _yaml: rework SYNTHETIC_COUNTER to be a C functionBenjamin Schubert2019-05-291-12/+15
| | | | | | | | | | | | | | The `SYNTHETIC_COUNTER` is an iterator that is called a lot in _yaml, one for each synthetic node. Cython is not able to optimize `itertools.counter` well enough. Providing a custom C function allows to reduce the amount of python code called in this critical codepath
| * _variable: Import _yaml from C.Benjamin Schubert2019-05-294-26/+57
| | | | | | | | | | This requires the addition of a definition file (.pxd), to list symbols exported.
| * _yaml: provide c definitions of functions called internallyBenjamin Schubert2019-05-291-12/+11
| | | | | | | | | | | | | | | | | | Providing c definitions for functions allows us to not have to go back to the python interpreter when inside the module. We therefore gain more performance. One gotcha is that keyword only arguments are not valid in cpdef functions.
| * _yaml: introduce FileInfo extension classBenjamin Schubert2019-05-291-13/+38
| | | | | | | | | | | | | | | | We used to have a tuple to keep file information. This makes it hard to read, accessing attributes by index. With an extension class FileInfo, we get better readability, without sacrificing performance
| * _yaml: Internalize `ProvenanceInformation`Benjamin Schubert2019-05-291-16/+11
| |
| * _yaml: Cythonize `Representer`.Benjamin Schubert2019-05-291-44/+86
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | `Representer` is the main interface with the `ruamel` library to parse the yaml files. Rewriting it with Cython introduces significant performance gains. Since `Representer` is not a python class anymore, we can't call `getattr` on it, and therefore have to do a manual switch on the types of events. While this is harder to read, it is also much more performant. Finally, sotp using `yaml.parse`, but call the parser manually, in order to avoid going in and out of the python code. This part could be made even better in the future when `ruamel` becomes stable and if they expose cython definitions, as they are coded in Cython.
| * _yaml: Cythonize public api of _yamlBenjamin Schubert2019-05-291-49/+62
| | | | | | | | | | | | | | | | | | _yaml is a bottleneck in a normal BuildStream run. Typing the external API allows us to reduce this bottleneck. Since we type the input variables, we can also remove some asserts that are checking if the parameters are of the correct type as Cython will throw TypeError if called incorrectly.
| * _yaml: Cythonize all internal functionsBenjamin Schubert2019-05-291-17/+23
| | | | | | | | | | | | Internal functions are not used outside the module, and are therefore easy to refactor and only call from C, leading us to significant performance gains.
| * _yaml: Cythonize and internalize NodeBenjamin Schubert2019-05-294-159/+172
| | | | | | | | | | | | | | | | | | | | | | Node used to be a NamedTuple that we used to access by index for speed reasons. Moving to an extension class allows us to access attributes by name, making the code easier to read and less error prone. Moreover, we do gain speed and memory by doing this move. Also fix a few places where we would not have an entire `Node` object but were instead just returning a tuple, missing some entries.
| * _variables: Cythonize Variables.substBenjamin Schubert2019-05-291-5/+9
| |
| * _variables: Cythonize Variables._flattenBenjamin Schubert2019-05-291-2/+6
| |
| * _variables: make Variables extension classBenjamin Schubert2019-05-291-2/+5
| | | | | | | | | | | | Extension class are faster for access and take less memory than a normal Python class. Variables is self contained and easy to cythonize.
| * _variables: Cythonize _parse_expstrBenjamin Schubert2019-05-291-6/+9
| | | | | | | | Also type the `PARSE_CACHE` in order to speedup access to it.
| * _variables: Cythonize _expand_expstrBenjamin Schubert2019-05-291-4/+4
| |
| * _variables: Cythonize _internal_expandBenjamin Schubert2019-05-294-18/+39
| | | | | | | | | | | | | | Move _variables.py to be a Cython module. `_internal_expand` is a function that is called a lot in BuildStream. It is also entirely isolated and easy to cythonize.
| * Introduce Cython to the project and documentBenjamin Schubert2019-05-299-3/+174
| | | | | | | | | | | | | | | | | | | | | | | | | | Cython requires a plugin to allow coverage of cython files, which was updated in coveragerc. It also means we need to build the dependencies and install cython for coverage. Cython requires access to both source and compiled files when running coverage. We therefore need to install project in develop mode. Updated documentation to explain how to run tests without tox but with coverage
| * Introduce pyproject.tomlBenjamin Schubert2019-05-295-2/+24
| | | | | | | | | | | | | | | | | | Using pyproject.toml, defined in PEP518, allows us to have an isolated build environment, ensuring that Cython can be installed before calling setup.py in tox. This allows us to use cython helpers in the setup.py script. This is a prerequisite for introducing Cython in the codebase
| * _variables.py: Optimize storing and usage of the variables resolutionBenjamin Schubert2019-05-291-20/+20
|/ | | | | This removes the precomputation of the size, which makes it slightly faster.
* Merge branch 'bschubert/iterative-loader' into 'master'bst-marge-bot2019-05-291-35/+88
|\ | | | | | | | | Make the loading process iterative See merge request BuildStream/buildstream!1363
| * loader: Reduce complexity of the _load_file functionBenjamin Schubert2019-05-291-27/+22
| | | | | | | | | | Delegate properly everything concerning a sub-loader to the sub-loader Stop iterating twice the loop for every dependency
| * _loader/loader.py: Rewrite _load_file() semi-iterativelydanielsilverstone-ct/iterative-loaderDaniel Silverstone2019-05-291-35/+93
|/ | | | | | | | | | In order to reduce the cost of recursion in loading a project, this commit makes a semi-iterative treatment of the _load_file() pathway. It's only semi-iterative in the sense that whenever we cross a loader boundary (junctions) we recurse. This should mean that high-depth projects load more safely. Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
* Merge branch 'jjardon/pygobject3-devel' into 'master'Javier Jardón2019-05-291-0/+2
|\ | | | | | | | | | | | | .gitlab-ci.yml: ostree plugin depends on ostree/python3-gobject-base Closes #1035 See merge request BuildStream/buildstream!1361
| * .gitlab-ci.yml: ostree plugin depends on ostree/python3-gobject-baseJavier Jardón2019-05-291-0/+2
|/ | | | [ci skip]
* Merge branch 'jjardon/fix_overnigth_test' into 'master'Javier Jardón2019-05-271-2/+2
|\ | | | | | | | | .gitlab-ci.yml: Fix overnigth tests See merge request BuildStream/buildstream!1359
| * .gitlab-ci.yml: Fix overnigth testsJavier Jardón2019-05-271-2/+2
|/ | | | Use bst-plugins-experimental with ostree plugin included
* Merge branch 'bschubert/normalize-path-in-context-tests' into 'master'bst-marge-bot2019-05-241-1/+1
|\ | | | | | | | | Fix: tests/context.py: Normalize path of XDG_CACHE See merge request BuildStream/buildstream!1351
| * tests/context.py: Normalize path of XDG_CACHEBenjamin Schubert2019-05-241-1/+1
|/ | | | | | | | | When running tests without --basetemp ./tmp, the path to the XDG_CACHE will not be normalized, and we check that it is equal to the normalized version of it. This test only passes because it assumed "--basetemp ./tmp". Now, it would work whatever the basetemp.
* Merge branch 'raoul/1023-grpc-forking' into 'master'Jürg Billeter2019-05-241-49/+98
|\ | | | | | | | | | | | | Ensure grpc channels are in separate process Closes #1023 See merge request BuildStream/buildstream!1357
View Lorry Controller Status