summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* WIPbschubert/no-sanitizeBenjamin Schubert2019-06-273-11/+21
|
* _cachekey: Don't use 'node_sanitize', but define __json__ in 'Node'Benjamin Schubert2019-06-272-2/+5
| | | | | This allows us to dump the data in an ordered way without depending on 'node_sanitize'.
* Merge branch 'bschubert/node-api-nodel' into 'bschubert/new-node-api'Benjamin Schubert2019-06-278-28/+20
|\ | | | | | | | | _yaml: Remove 'node_del' and support `del mapping[key]` See merge request BuildStream/buildstream!1424
| * _yaml: Remove 'node_del' and support `del mapping[key]`Benjamin Schubert2019-06-278-28/+20
|/ | | | | - Also add a convenience method 'safe_del' catching the exception when we don't care if the value was there or not.
* Merge branch 'bschubert/node-copy' into 'bschubert/new-node-api'Benjamin Schubert2019-06-277-85/+51
|\ | | | | | | | | _yaml: Remove 'node_copy' and add 'Node.copy()' See merge request BuildStream/buildstream!1432
| * _yaml: Remove 'node_copy' and add 'Node.copy()'bschubert/node-copyBenjamin Schubert2019-06-277-85/+51
|/ | | | Also adaprt every part of the code calling it
* Merge branch 'bschubert/node-api-noget' into 'bschubert/new-node-api'Benjamin Schubert2019-06-2670-757/+703
|\ | | | | | | | | introduce a Node api, and remove node_get See merge request BuildStream/buildstream!1401
| * _yaml: Add a 'get_node' on Mapping, when return type can be of multiple typesbschubert/node-api-nogetBenjamin Schubert2019-06-253-12/+27
| | | | | | | | | | | | | | | | | | | | | | Sometimes, we might want to have a list or a string, in which case, we can now use 'get_node', which can return multiple types. This method doesn't contain a 'default' value, as building a 'Node' from it would be hard. We therefore just have a 'allow_none' flag that allows returning 'None' when the value is not found. - includes: use the new 'get_node' instead of trying multiple versions
| * _yaml: Never create base 'Node' directlyBenjamin Schubert2019-06-252-5/+8
| | | | | | | | We shouldn't have to create normal nodes ever. Let's ensure we don't
| * _yaml: Remove 'node_get' and migrate all remaining calls to new APIBenjamin Schubert2019-06-2519-167/+67
| |
| * doc/bst2html: Remove usage of 'node_get' and use new APIBenjamin Schubert2019-06-251-13/+11
| |
| * element: Remove `node_get_member` and all references to itBenjamin Schubert2019-06-254-47/+3
| | | | | | | | This can now be done more easily with the Node api
| * plugin: remove 'node_get_list_element'Benjamin Schubert2019-06-251-37/+1
| | | | | | | | | | | | | | The new yaml API is able of getting these access with the same error messages handling. Having this helper is therefore not useful anymore.
| * element: remove 'node_subst_list_element'Benjamin Schubert2019-06-252-43/+1
| | | | | | | | | | This method is unused and can be easily reimplemented with the new Node API by looping over nodes.
| * tests: remove 'node_get_yaml_provenance()' helper and replace with the new APIBenjamin Schubert2019-06-256-62/+19
| | | | | | | | | | | | This function is hard to make generic and, with the new API, the access is simplified. Therefore, removing this function and migrating all its usages
| * _yaml: Introduce 'get_sequence()' and 'sequence_at()'/'mapping_at()'Benjamin Schubert2019-06-2519-70/+123
| | | | | | | | | | | | | | | | | | - Adding 'get_sequence' on MappingNode to access sequences in a mapping - Adding 'sequence_at' on SequenceNode to access sequences in a sequence - Adding 'mapping_at' on SequenceNode to access mappings in a sequence Using "*_at" in sequences allows us to quickly understand if we are dealing with a sequence or a mapping.
| * _yaml: Remove use of expected_type=None in 'node_get()'Benjamin Schubert2019-06-253-13/+11
| | | | | | | | | | | | | | In a strongly typed API with Node, having a 'None' as expected type is hard to make nice. Moreover, this is rarely used in the codebase. Thus, adapting the call sites to not use 'None' as an expected type.
| * _yaml: Add 'as_int()' on ScalarNodeBenjamin Schubert2019-06-258-15/+33
| | | | | | | | | | | | - Add the 'as_int()' method on 'ScalarNode' to replace 'node_get(mapping, key, int)' - Adapt all call sites to use the new API
| * _yaml: Add 'as_bool()' and 'is_none()' to ScalarNodeBenjamin Schubert2019-06-2516-34/+58
| | | | | | | | | | | | | | | | | | | | | | - 'as_bool()' casts a ScalarNode into a boolean, understanding both 'True' and 'False' as truthy-falsy values, as per node_get(type=bool) behavior - 'is_none()' allwos checking whether the scalar node contains a 'None' value. Since 'None' cannot be used when working with booleans, we need to have a way of checking for 'None' when we actually need the information of whether the value is unset. - Adapt all call places to use the new API
| * _yaml: Add 'as_str()' on ScalarNode and 'get_scalar()' on MappingNodeBenjamin Schubert2019-06-2546-157/+189
| | | | | | | | | | | | | | | | | | | | - 'get_scalar()' allows retrieving a scalar node from a mapping. - 'as_str()' casts a ScalarNode into a string (thus removing the node information). Both together, those replace 'node_get(mapping, key, type=str)' but also allow retrieving the 'Node' itself, which will allow in the future lazier provenance computation.
| * _yaml: add 'get_mapping()' to MappingNodeBenjamin Schubert2019-06-2519-77/+101
| | | | | | | | | | | | | | | | This allows to get a mapping node from another 'MappingNode', replacing 'node_get(my_mapping, key, type=dict)' Also changes all places where 'node_get' was called like that by the new API.
| * _yaml: Introduce a 'Sequence' NodeBenjamin Schubert2019-06-251-10/+20
| | | | | | | | | | | | | | This allows us to represent sequences (lists) more efficiently. This is the third type that is needed in order to know what kind of datastructure we are handling in our graph.
| * _yaml: Introduce a 'Mapping' Node classBenjamin Schubert2019-06-253-23/+38
| | | | | | | | | | This allows us to specialize the base 'Node' class to store mappings (dicts) more efficiently.
| * _yaml: Introduce a Scalar Node to encapsulate str/int/boolsBenjamin Schubert2019-06-251-12/+33
|/ | | | | | | | This node specializes the 'Node' class to represent a scalar value. This allows us to let users access leaf Nodes in the graph without having to find them with `node_find_target`, and also will allow us lazier computation of node provenance in some cases.
* Merge branch 'bschubert/remove-useless-sanitize' into 'master'bst-marge-bot2019-06-2512-24/+23
|\ | | | | | | | | _yaml: Remove useless calls to '_yaml.node_sanitize' See merge request BuildStream/buildstream!1412
| * _yaml: Remove useless calls to '_yaml.node_sanitize'Benjamin Schubert2019-06-2512-24/+23
|/ | | | | | Calling '_yaml.dump' will itself call '_yaml.node_sanitize', therefore we can remove all calls to it in places where we directly after call dump.
* Merge branch 'juerg/fetch-subprojects' into 'master'bst-marge-bot2019-06-259-122/+98
|\ | | | | | | | | Improve subproject fetching See merge request BuildStream/buildstream!1414
| * _stream.py: Use scheduler to fetch subprojectsJürg Billeter2019-06-251-2/+8
| |
| * Handle subproject fetching in the Stream classJürg Billeter2019-06-254-8/+23
| |
| * app.py: Create stream before loading projectJürg Billeter2019-06-252-26/+39
| | | | | | | | | | | | This is in preparation for moving subproject fetching to the Stream class. Project loading may require subproject fetching as part of processing include directives.
| * Always fetch subprojects as neededJürg Billeter2019-06-258-90/+32
|/ | | | | | Treat junction element sources the same as sources of any other element and always fetch subprojects as needed. Do not ask the user to manually fetch subprojects.
* Merge branch 'jennis/notify_reverse_deps' into 'master'bst-marge-bot2019-06-242-24/+93
|\ | | | | | | | | Notify direct reverse dependencies when an Element becomes ready for runtime and cached See merge request BuildStream/buildstream!1406
| * element.py: Notify reverse deps when ready for runtime and cachedjennis/notify_reverse_depsJames Ennis2019-06-242-23/+86
| | | | | | | | | | | | | | | | | | | | | | | | An Element becomes ready for runtime and cached when: 1. It has a strong cache key 2. It is cached 3. Its runtime dependencies are ready for runtime and cached (this ensures the runtimes of runtimes are also cached). This patch introduces the method attempt_to_notify_reverse_dependencies which will notify all direct reverse dependencies of an Element once said Element becomes ready for runtime and cached.
| * element.py: cache key should be set in __update_cache_keysJames Ennis2019-06-241-1/+6
| |
| * element.py: Set can_query_cache_callback to None when calledJames Ennis2019-06-241-0/+1
|/ | | | | Each other callback callsite immediately sets the callback to None once invoked. We should be consistent.
* Merge branch 'raoul/1044-blobs-on-demand' into 'master'bst-marge-bot2019-06-248-27/+229
|\ | | | | | | | | | | | | CLI options for on demand blob fetching Closes #1044 See merge request BuildStream/buildstream!1400
| * tests: Add case for using pull and use buildtreesRaoul Hidalgo Charman2019-06-241-0/+29
| | | | | | | | Part of #1044
| * tests: Add shell test with pull optionRaoul Hidalgo Charman2019-06-241-0/+50
| | | | | | | | Part of #1044
| * _stream/cli: Add pull option to shellRaoul Hidalgo Charman2019-06-242-15/+30
| | | | | | | | | | | | This will fetch artifacts if they're not local or are incomplete. Part of #1044
| * tests: Add tests to buildcheckout for pullingRaoul Hidalgo Charman2019-06-241-1/+63
| | | | | | | | | | | | | | | | One test to check that artifacts are pulled and a checkout is successful when the --pull option is specified. Another to check that it attempts to pull but fails gracefully if the artifact isn't present. Part of #1044
| * cli.py/_stream.py: Add pull option to checkoutRaoul Hidalgo Charman2019-06-242-6/+24
| | | | | | | | | | | | | | This will set up a pull queue and pull relevant artifacts if not present or incomplete. Part of #1044
| * tests/partial: Add test to cover find_missing_blobsRaoul Hidalgo Charman2019-06-211-0/+25
| | | | | | | | Add test that ensures method is actually covered now and it works.
| * _artifactcache.py: fix find_missing_blobs methodRaoul Hidalgo Charman2019-06-212-5/+8
|/ | | | | | | This never will have worked because digests aren't hashable, and this method wasn't being covered in any tests. Also changed `remote_missing_blobs` to ensure the blobs list is an iterator and updated its docs.
* .gitlab-ci.yml: Use up-to-date bst-plugins-experimentalJavier Jardón2019-06-201-1/+1
| | | | Fixes #1046
* Merge branch 'tpollard/shellbuildtree' into 'master'bst-marge-bot2019-06-193-17/+29
|\ | | | | | | | | _frontend/cli.py: Tweak 'try' & 'always' bst shell buildtree handling See merge request BuildStream/buildstream!1392
| * _frontend/cli.py: Tweak 'try' & 'always' bst shell buildtree handlingTom Pollard2019-06-193-17/+29
|/ | | | | | | | | | | If the cached Artifact wasn't originally generated with a buildtree, then there's no need to attempt to find it in remotes by entering Stream(). In interactive mode we only present the user the option to attempt the pull with the above assumption on buildtree_exists, so apply the same to applicable non interactive choices. This also includes some tweaks in integration/shellbuildtrees.py to reflect changes and cover cases that were missing.
* Merge branch 'juerg/grpc' into 'master'Jürg Billeter2019-06-192-0/+6
|\ | | | | | | | | Guard against gRPC channels in the main process See merge request BuildStream/buildstream!1405
| * _sandboxremote.py: Guard against gRPC channels in the main processjuerg/grpcJürg Billeter2019-06-191-0/+3
| |
| * casremote.py: Guard against gRPC channels in the main processJürg Billeter2019-06-191-0/+3
|/
* Merge branch 'aevri/smallerjobs' into 'master'bst-marge-bot2019-06-191-6/+6
|\ | | | | | | | | jobs/job: send ChildJob the context, not scheduler See merge request BuildStream/buildstream!1407
View Lorry Controller Status