summaryrefslogtreecommitdiff
path: root/buildstream/_artifactcache/pushreceive.py
Commit message (Collapse)AuthorAgeFilesLines
* _artifactcache/pushreceive.py: Cleanup reported error when receiving ↵Tristan Van Berkom2018-06-071-3/+2
| | | | | | | oversized artifacts This user facing string was redundantly declared in two places, only the message when catching the error was ever printed.
* pushreceive.py: Abstract the buffer_James Ennis2018-06-071-9/+8
|
* pushreceive.py: Ensure huge artifacts are not pushedJames Ennis2018-06-071-5/+32
|
* pushreceive.py: Add comments to code and name variables more sensiblyJames Ennis2018-06-071-1/+13
|
* pushreceive.py: Remove LRP artifacts from cache, introduce clean_up_cache()James Ennis2018-06-071-3/+54
| | | | This fixes #136
* pushreceive.py: Ensure there is a repopathJames Ennis2018-06-071-0/+3
|
* _artifactcache/pushreceive.py: Add Click type for CLI argument 'repo'Chandan Singh2018-06-051-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The CLI for `bst-artifact-receive` expects a `repo` argument, which is supposed to be a directory, but Click currently expects it to be just any string. This results in stack traces like the one below when the argument provided is not a directory: $ ~/.local/bin/bst-artifact-receive --pull-url http://foo foobaz Traceback (most recent call last): File "/root/.local/bin/bst-artifact-receive", line 8, in <module> sys.exit(receive_main()) ... File "/src/buildstream/buildstream/_artifactcache/pushreceive.py", line 581, in __init__ self.repo.open(None) GLib.Error: g-io-error-quark: /src/buildstream/43fref: opendir(/src/buildstream/foobaz): No such file or directory (1) Add types for this argument such that it throws better error messages when it receives bad arguments. With the Click types added, it will instead fail with messages like these: $ ~/.local/bin/bst-artifact-receive --pull-url http://foo foobaz Usage: bst-artifact-receive [OPTIONS] REPO Error: Invalid value for "repo": Directory "foobaz" does not exist. $ ~/.local/bin/bst-artifact-receive --pull-url http://foo setup.py Usage: bst-artifact-receive [OPTIONS] REPO Error: Invalid value for "repo": Directory "setup.py" is a file. Fixes #409.
* _artifactcache/pushreceive.py: Modified incorrect error messagePhillip Smyth2018-05-011-1/+1
| | | | | | | changed encode_message push exception from 'Command must by GLib.Variant' to 'Command must be PushCommand'
* _artifactcache/pushreceive.py: Do not return True for skipped pushJürg Billeter2018-04-301-1/+1
|
* _artifactcache/pushreceive.py: Fix commit object check in send_info()Jürg Billeter2018-04-301-1/+1
| | | | Fixes #325.
* Add artifact cache receive profiling domainJim MacArthur2018-04-231-0/+14
| | | | | | _artifactcache/pushreceive.py: Wrap OSTreeReceiver.do_run in a profiling domain. _profile.py: Add 'ARTIFACT_RECEIVE' domain.
* _artifactcache/pushreceive.py: Fixed ProcessWithPipes to declare attributes ↵Tristan Van Berkom2018-04-201-0/+3
| | | | in constructor
* buildstream/_artifactcache/pushreceive.py: Fixing unused variablesTristan Van Berkom2018-04-191-5/+5
|
* pylint - dealt with whitespace issues and disabled pep8 warningsJames Ennis2018-03-141-1/+1
|
* pushreceive.py: byteorder functions are now more sensibly namedJames Ennis2018-03-141-10/+10
|
* pylint - dealt with redefined-outer-name and redefined-built in warningsJames Ennis2018-03-141-8/+8
|
* pylint - dealt with import warningsJames Ennis2018-03-141-4/+5
|
* pylint - dealt with len-as-condition warningJames Ennis2018-03-141-3/+3
|
* pylint - dealt with dangerous-default-value warningJames Ennis2018-03-141-2/+2
|
* Shorten the warnings raised when remote cache initialization failsSam Thursfield2018-01-161-2/+2
| | | | | | | | | | Before: [--:--:--] WARNING Failed to fetch remote refs from ssh://ostree@ostree.baserock.org:22000/cache: BuildStream did not connect successfully to the shared cache ssh://ostree@ostree.baserock.org:22000/cache: SSH error: ssh: connect to host ostree.baserock.org port 22000: Connection refused After: [--:--:--] WARNING Failed to fetch remote refs from ssh://ostree@ostree.baserock.org:22000/cache. ssh: connect to host ostree.baserock.org port 22000: Connection refused
* _artifactcache: Use pushreceive also for local artifact repositoriesJürg Billeter2018-01-111-13/+58
| | | | | This reduces the differences between local and remote artifact repositories, increasing code coverage of tests.
* _artifactcache/pushreceive.py: Flush but do not close stdoutJürg Billeter2018-01-111-1/+1
| | | | | Closing stdout will cause an exception when used with multiprocessing as that calls sys.stdout.flush() as well.
* Improve error messages for SSH protocol handshakeSam Thursfield2018-01-111-15/+21
| | | | | | | | | | | | | If you have an artifact remote cache in your config using the ssh:// protocol and some SSH issue prevents us from connecting, we now show the user the stderr output from the `ssh` program, e.g. [--:--:--] WARNING Failed to fetch remote refs from ssh://artifacts@172.17.0.2:22200/artifacts/: BuildStream did not connect successfully to the shared cache ssh://artifacts@172.17.0.2:22200/artifacts/: Permission denied (publickey,keyboard-interactive). Previously we would just show the error message raised by the protocol code, which was less useful, e.g.: [--:--:--] WARNING Failed to fetch remote refs from ssh://artifacts@172.17.0.2:22200/artifacts/: BuildStream did not connect successfully to the shared cache ssh://artifacts@172.17.0.2:22200/artifacts/: Expected reply, got none
* Remove unused importsGökçen Nurlu2017-12-071-2/+0
|
* _artifactcache.py: Fixed stack trace regression in pushing of artifacts.Tristan Van Berkom2017-11-241-2/+2
|
* Remove the push-port config optionSam Thursfield2017-11-221-8/+8
| | | | | | | | | | | | Ports can and should be specified by using proper ssh:// URL forms, e.g: ssh://artifacts@example.com:22200/artifacts The alternate form of artifacts@example.com:artifacts isn't a valid URL, and doesn't let you specify a different port. People are used to this form due to Git continuing to use it but we should encourage people to use proper URLs.
* Artifact push URLs must redirect to the corresponding pull URLSam Thursfield2017-11-221-11/+49
| | | | | | | | | | | | | | | | | This change is part of issue #112 ("Artifact configuration is confusing and fragile, need canonical push/pull urls.") It updates the bst-artifact-receive program to send a pull URL to clients who access it over SSH. This requires extra configuration in the artifact cache server, so that it knows the correct pull URL. Versions of BuildStream which contain this commit will not be able to communicate over SSH with artifact caches that have not been updated to the same version. Currently this is just used to double-check user configuration, but we can now simplify the user facing configuration for artifact caches completely.
* Switch old-style string formattings to new '.format()'Gökçen Nurlu2017-11-171-14/+14
|
* Refactor, remove some unused imports138-aborting-bst-push-command-causes-stack-traceAngelos Evripiotis2017-11-131-1/+0
| | | | | | | When reading element.py, I was surprised that it was importing 'inspect'. Remove that import and some other unused ones to improve readability.
* _artifactcache/pushreceive.py: Fix stream offset checkJürg Billeter2017-07-281-1/+1
| | | | | The offset difference includes the padded bytes of the last file, i.e., up to 511 bytes in addition to the 512 byte block.
* _artifactcache/pushreceive.py: Ensure stream is at the right positionJürg Billeter2017-07-281-0/+7
|
* _artifactcache/pushreceive.py: Verify that we have the commit objectsJürg Billeter2017-07-271-0/+6
|
* _artifactcache/pushreceive.py: Add handshake after sending objectsJürg Billeter2017-07-271-1/+13
| | | | | Ensure all objects have been sent before moving them into the repository and do not terminate pusher while receiver is still processing.
* _artifactcache/pushreceive.py: Raise PushException on connection failureJürg Billeter2017-07-261-0/+4
| | | | | | Unexpected connection termination should not be considered a bug. Fixes #51
* _artifactcache/pushreceive.py: Drop branch check in receiverJürg Billeter2017-07-261-18/+1
| | | | | | | | | | The pusher already checks this and the check in the receiver does not provide any additional guarantees as it is prone to race conditions. This prevents a push error in case two clients push an artifact with the same key around the same time. Fixes #52
* Check for write access to remote artifact cache early on in the pipelinesam/artifactcache-preflight-checkSam Thursfield2017-07-211-38/+93
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously, the first time you configured an artifact cache, you would get to the end of your first build and then BuildStream would exit because of some stupid mistake like you got the address slightly wrong or you forgot to add the host keys of the remote artifact cache to `~/.ssh/known_hosts`. To avoid surprises, if there's an artifacts push-url configured we now try to connect to it as a preflight check so that issues are raised early. On success, you will see something like this: [--:--:--][90904fe4][ main:gnu-toolchain/stage2.bst ] START Checking connectivity to remote artifact cache [00:00:00][90904fe4][ main:gnu-toolchain/stage2.bst ] SUCCESS Connectivity OK On failure, it looks like this: [--:--:--][90904fe4][ main:gnu-toolchain/stage2.bst ] START Checking connectivity to remote artifact cache [00:00:03][90904fe4][ main:gnu-toolchain/stage2.bst ] FAILURE BuildStream will be unable to push artifacts to the shared cache: ssh: connect to host ostree.baserock.org port 2220: Connection timed out As a bonus, for some reason this check causes SSH to ask about unknown host keys rather than just failing, so you may now see messages like this if the host keys are unknown rather than an error: The authenticity of host '[ostree.baserock.org]:22200 ([185.43.218.170]:22200)' can't be established. ECDSA key fingerprint is SHA256:mB+MNfYREOdRfp2FG6dceOlguE/Skd4QwnS0tvCPcnI. ECDSA key fingerprint is MD5:8f:fa:ab:90:19:31:f9:f7:f1:d4:e5:f0:a2:be:56:71. Are you sure you want to continue connecting (yes/no)?
* _artifactcache/pushreceive.py: Do not list all refsJürg Billeter2017-07-201-5/+20
| | | | | | | The full ref list can easily exceed the maximum message size. Limit list to refs being pushed. Fixes #47
* _artifactcache/pushreceive.py: Fix exception handlingJürg Billeter2017-07-171-1/+2
| | | | | | exc_info was uninitialized. Fixes: 2d680719 ("Support pushing multiple refs")
* _artifactcache/pushreceive.py: Support pushing multiple refsJürg Billeter2017-07-141-5/+16
|
* _artifactcache/pushreceive.py: Fix obvious typoSam Thursfield2017-07-131-2/+2
| | | | This was causing tracebacks when pushing.
* _artifactcache: Few changesTristan Van Berkom2017-07-111-8/+19
| | | | | | | o Allow specifying the push port o Additional timed messaging at push time o Return True if artifact was pushed, False if the artifact already existed remotely, raise ArtifactError only on error
* _artifactcache/pushreceive.py: Changing up the bst-artifact-receive optionsTristan Van Berkom2017-07-111-4/+6
| | | | Make the repo an argument, so that it bails out if not specified.
* bst-artifact-receive: Use click instead of argparseTristan Van Berkom2017-07-101-14/+12
| | | | This will help with man page generation.
* _artifactcache/pushreceive.py: Our local copy of ostree-pushTristan Van Berkom2017-07-101-0/+603
Added entry point for running `bst-artifact-receive`
View Lorry Controller Status