summaryrefslogtreecommitdiff
path: root/swiftclient
Commit message (Collapse)AuthorAgeFilesLines
* Do not reveal auth token in swiftclient log messages by defaultliberty-eolstable/libertyJoel Wright2016-03-082-7/+105
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently the swiftclient logs sensitive info in headers when logging HTTP requests. This patch hides sensitive info in headers such as 'X-Auth-Token' in a similar way to swift itself (we add a 'reveal_sensitive_prefix' configuration to the client). With this patch, tokens are truncated by removing the specified number of characters, after which '...' is appended to the logged token to indicate that it has been redacted. Also include client.parse_header_string() for safe unicode handling of header data. Backport based on commits: c3f06417049e17a8d45ee5926c5043cb6c8aa9ef 4d44dcf36086add13d3353915c014f095ab99c6d ce569f46517e10f2ce0d27e9ee0a922ad1d84e2f 46d817828082105a69d4da53fef2f2fbefc54809 aa0edd00966237163451fc44cda2c593a5215cbe Co-Authored-By: Tim Burke <tim.burke@gmail.com> Co-Authored-By: Alistair Coles <alistair.coles@hpe.com> Co-Authored-By: Li Cheng <shcli@cn.ibm.com> Co-Authored-By: Zack M. Davis <zdavis@swiftstack.com> Change-Id: I71fc5aad23bc076b06f75888c3ea507feffc7b48 Closes-bug: #1516692
* Fix the http request headers being overwritten in loggingMin Min Ren2016-02-251-1/+0
| | | | | | | | | Fix the http request headers in put_object being overwritten in logging Cherry-picked from commit 61880c6f980cb8e613bdf6cb48a9a61ce7488162 Change-Id: Id0d1e36561a61ed1ce30d93c801ec32f058a6fa4 Closes-bug: #1501292
* absolute expiry option for tempURL generationZack M. Davis2015-09-042-6/+23
| | | | | | | | | | | | | | | | | | | | The `tempurl` subcommand's second positional argument is called `seconds` and has heretofore interpreted as the number of seconds for which the tempURL should be valid, counting from the moment of running the command. This is indeed a common, if not the most common, use-case. But some users, occasionally, might want to generate a tempURL that expires at some particular ("absolute") time, rather than a particular amount of time relative to the moment of happening to run the command. (One might make an analogy to the way in which Swift's expiring object support supports an `X-Delete-At` header in addition to `X-Delete-After`—and it's the former that must be regarded as ontologically prior.) Thus, this commit adds an `--absolute` optional argument to the `tempurl` subcommand; if present, the `seconds` argument will be interpreted as a Unix timestamp of when the tempURL should be expire, rather than a duration for which the tempURL should be valid starting from "now". Change-Id: If9ded96f2799800958d5063127f3de812f50ef06
* Merge "Log and report trace on service operation fails"Jenkins2015-09-042-54/+201
|\
| * Log and report trace on service operation failsJoel Wright2015-08-312-54/+201
| | | | | | | | | | | | | | | | | | This patch adds exception logging to the swift service API. Each operation that results in failure of any operation will now log the exception as well as report a timestamp and full stack trace in the results returned by the service API calls. Change-Id: I7336b28354e7740ea7d048bdf355e3c1a1b4436c
* | Merge "Increase httplib._MAXHEADERS to 256."Jenkins2015-09-031-1/+4
|\ \
| * | Increase httplib._MAXHEADERS to 256.Charles Hsu2015-08-271-1/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | By default Swift increase the number of max metadata count to 90 and extra header count to 32. That mean we can put 90 metadata to Account/Container/Object by default, when user put 90 metadata to a Account, the Account header count is close or more than 100. The swift client unable to access Account and get an error likes, ('Connection aborted.', HTTPException('got more than 100 headers',)) So the default _MAXHEADERS(100) won't enough. Change-Id: I5ffc4eb5d3e1ebc3dbdd7dc69376919ae3e1c5a8
* | | Merge "Stop Connection class modifying os_options parameter"Jenkins2015-09-031-1/+1
|\ \ \ | |_|/ |/| |
| * | Stop Connection class modifying os_options parameterAlistair Coles2015-08-251-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When a caller passes an os_options dict to the Connection class constructor, the constructor may modify the os_options dict, which can surprise the caller if they re-use the os_options dict. Specifically the os_options tenant_name and object_storage_url may be modified, and the changed values would then leak through to a subsequent Connection constructed using the same os_options dict. This fix simply constructs a new dict from the supplied os_options. The patch also adds a test that covers this and also verifies that a preauth_url passed as a keyword arg to Connection() will take precedence over any object_storage_url in an os_options parameter. Closes-Bug: 1488070 Change-Id: Ic6b5cf3ac68c505de155619f2610be9529e15432
* | | Merge "Reduce memory usage for download/delete and add --no-shuffle option ↵Jenkins2015-08-282-48/+99
|\ \ \ | |/ / |/| | | | | to st_download"
| * | Reduce memory usage for download/delete and add --no-shuffle option to ↵Joel Wright2015-07-202-48/+99
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | st_download The current code builds a full object listing before performing either a multiple download or delete operation (and also shuffles this complete list in the case of a download). This patch removes the creation of the full object list and adds the ability to turn off shuffle for files when downloading. Also added is a limit on the number of list results that can be queued by a single call to service.list without consuming results (reduces memory overhead for large listings). Some tests added for service.py download and list. Change-Id: Ie737cbb7f8b1fa8a79bbb88914730b05aa7f2906
* | | Merge "Add minimal working service token support."Jenkins2015-08-251-19/+103
|\ \ \
| * | | Add minimal working service token support.Stuart McLaren2015-08-241-19/+103
| | |/ | |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add client changes to allow accessing alternative reseller_prefixes via a service token. ie client changes for this server side spec: https://review.openstack.org/#/c/105228 We assume that the service storage url has been passed in as a preauthurl. We rely on get_auth preserving this url. Change-Id: I1cfda178f0b6c8add46cfebd6bf38440caae2036
* | | flake8 ignores same hacks as swiftHiroshi Miura2015-08-181-4/+4
|/ / | | | | | | | | | | | | | | | | | | | | - blacklisted flake8 hacking - fix against E122 continuation line missing indentation or outdented Closes-bug: #1475516 Change-Id: I708d0a3466a1f85c84e478873e142821ce0774cb Signed-off-by: Hiroshi Miura <miurahr@nttdata.co.jp>
* | Merge "make Connection.get_auth set url and token attributes on self"Jenkins2015-08-061-7/+8
|\ \
| * | make Connection.get_auth set url and token attributes on selfZack M. Davis2015-07-241-7/+8
| |/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When a Connection is first __init__ialized (without providing a preauthurl or preauthtoken), the url and token attributes are None; they get set (to be reused on future requests) after one of the wrapper methods internally using _retry (head_account, get_container, put_object, and similar friends) is called. However, this had not been the case for get_auth, much to the momentary confusion and disappointment of programmers using swiftclient who expected to be able to get the token or storage URL off the Connection object after calling get_auth (perhaps in order to make an unusual kind of request that swiftclient doesn't already have a function for). This commit makes get_auth set the url and token attributes as one might expect. Change-Id: I0d9593eb5b072c8e3fa84f7d5a4c948c0bc6037a
* | Merge "swiftclient: add short options to help message"Jenkins2015-07-311-34/+43
|\ \ | |/ |/|
| * swiftclient: add short options to help messageHiroshi Miura2015-07-171-34/+43
| | | | | | | | | | | | | | | | | | | | | | | | - add usage strings for short option such as '-a' for '--all' This add all short options to usage text. - add missing --header usage help for download command - some cometic changes Closes-bug: #1475511 Change-Id: Ibfecac8764669540fa025787548133a50fa50b10 Signed-off-by: Hiroshi Miura <miurahr@nttdata.co.jp>
* | Merge "Python 3: Replacing unicode with six.text_type for py3 compatibility"Jenkins2015-07-161-1/+1
|\ \
| * | Python 3: Replacing unicode with six.text_type for py3 compatibilityjanonymous2015-06-281-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The "unicode" type was renamed to "str" in Python 3. Use six.text_type to make swiftclient compatible with Python 3. For more information about changes needed for py3 compatibility, see: https://wiki.openstack.org/wiki/Python3 Change-Id: Ic65607a69935652a1874340928f626fbcc35c014
* | | Merge "Always decode command-line arguments as UTF-8"2.5.0Jenkins2015-07-161-6/+9
|\ \ \ | |_|/ |/| |
| * | Always decode command-line arguments as UTF-8Tim Burke2015-07-071-6/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | There was always an implicit assumption that they were UTF-8 before, and by converting them to unicode we close another hole allowing raw bytes to appear in user-facing messages. Closes-Bug: #1431866 Change-Id: If2e41d9a592c3ad02818e9c6f0959fd4b73cd0e0
* | | Merge "Correct the help message of swift tempurl"Jenkins2015-07-111-4/+4
|\ \ \
| * | | Correct the help message of swift tempurlYangLei2015-06-301-4/+4
| | |/ | |/| | | | | | | | | | | | | | | | correct the help message of swift tempurl use <> instead of [] in Positional arguments. Change-Id: Ib60ce97cef03e0423082c497604525eba2300fa9
* | | Merge "Add some bash helpers for auth stuff"Jenkins2015-07-111-4/+50
|\ \ \
| * | | Add some bash helpers for auth stuffClay Gerrard2015-06-151-4/+50
| | | | | | | | | | | | | | | | Change-Id: If61ac9a050e7a115f37dbf4e74b904ac5dfd2052
* | | | Merge "Add ability to download objects to particular folder."Jenkins2015-07-112-3/+36
|\ \ \ \ | |_|_|/ |/| | |
| * | | Add ability to download objects to particular folder.Charles Hsu2015-07-082-3/+36
| |/ / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch adds "--output-dir" and "--remove-prefix" options to the "download" command and unit tests for it. Example: $ swift list example --prefix swift2.2 swift2.2/bin/swift-object-auditor swift2.2/bin/swift-object-expirer swift2.2/bin/swift-object-info swift2.2/bin/swift-object-replicator swift2.2/bin/swift-object-server swift2.2/bin/swift-object-updater When given "--output-dir <directory>", client downloads objects to <directory>. $ swift download example --prefix swift2.2 \ --output-dir new/swift/dir The folder structure: . └── new └── swift └── dir └── swift2.2 └── bin ├── swift-object-auditor ├── swift-object-expirer ├── swift-object-info ├── swift-object-replicator ├── swift-object-server └── swift-object-updater When given "--remove-prefix", client downloads objects without <prefix>. $ swift download example --prefix swift2.2 \ --remove-prefix \ --output-dir swift The folder structure: . └── swift └── bin ├── swift-object-auditor ├── swift-object-expirer ├── swift-object-info ├── swift-object-replicator ├── swift-object-server └── swift-object-updater Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com> Change-Id: I7463fe2941cc94f9a50a4756a97c2ccdf946294d Implements: blueprint swiftclient-download-pseudo-folder-to-specific-target
* | | Merge "SwiftClient object upload beginning with / or "./""Jenkins2015-07-071-4/+4
|\ \ \
| * | | SwiftClient object upload beginning with / or "./"Pradeep Kumar Singh2015-06-171-4/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently SwiftClient populate response dictionary before removing "./" or "/" at begining of object name. This patch fixes that by changing that order. Closes-bug: #1412425 Change-Id: I80222754caba5d42a468f4677ac539e46682dd31
* | | | Merge "Added check for negative segment-size"Jenkins2015-07-071-0/+3
|\ \ \ \
| * | | | Added check for negative segment-sizePradeep Kumar Singh2015-06-231-0/+3
| | |_|/ | |/| | | | | | | | | | | | | | | | | | Closes-Bug: #1453135 Change-Id: Ia9c2b27d998e6ac1889cc74c12e456c06ecd84d9
* | | | Merge "Remove all DLO segments on upload of replacement"Jenkins2015-07-021-12/+20
|\ \ \ \ | |/ / / |/| | |
| * | | Remove all DLO segments on upload of replacementTim Burke2015-03-231-12/+20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously, only the first container-listing's worth of segments was deleted, which would leave behind orphaned segments when the object was very large with small segments or the server's container_listing_limit was small. In addition, process DLO and SLO deletions on the segment thread pool, rather than the object thread pool. Change-Id: I1587375261a6237fa55a9cb96bda8dae918cc795 Related-Bug: #1418007
* | | | Fix docstring typo for SwiftService.uploadClint Byrum2015-06-181-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | The option from argparse in shell is 'header' and that is what is used in the code. Change-Id: I5c889192ef7c46c299dc0ec7cbc7c4d027dae6d5
* | | | Fix inconsistent usage of "Positional argument"Andreas Jaeger2015-06-081-1/+1
| |_|/ |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | All help texts uses "Positional argument" with the exception of tempurl. Update tempurl to use this as well so that the formatting tools work fine and can show this nicely on http://docs.openstack.org/cli-reference/content/swiftclient_commands.html#swiftclient_subcommand_tempurl like it's done for other options. Change-Id: Ib5502c23b236986bea5a4d4a63a46fca411a8494 Closes-Bug: #1463081
* | | Merge "Allow reading from object body on download"Jenkins2015-06-041-7/+32
|\ \ \
| * | | Allow reading from object body on downloadStuart McLaren2015-03-241-7/+32
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently, get_object returns a generator. This allows access to the object's data in multiples of 'resp_chunk_size'. This patch adds a read function to also allow accessing the object data using read(size). This allows, for example, the consumer of an object (where no byte range has been specified) to read up to certain boundaries while streaming to a new Large Object with segments of a specified size. Reading and chunking can be safely mixed. Related-Bug: 1367925 Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com> Change-Id: I1cfb67f45afc7015fd896f1a89bebae048871769
* | | | Make default get_auth timeout be NoneAlistair Coles2015-06-031-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Setting timeout to a default of False in get_auth() results in a requests timeout of 0.0 in keystoneclient, which causes a connection failure. This bug will cause func tests to not run when using keystoneauth. Added unit tests to verify correct default timeout is set in get_auth(). Drive-by: remove what seems like a stale TODO comment Change-Id: I17c781ce160a682b1768d315422ade0cdd2df198
* | | | Merge "Compare each chunk of large objects when downloading"Jenkins2015-06-011-12/+60
|\ \ \ \
| * | | | Compare each chunk of large objects when downloadingTim Burke2015-05-271-12/+60
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This allows the use of --skip-identical when downloading SLOs and DLOs, while still only requiring one request for normal objects. Change-Id: I237ec5a8cc4d1119b6d3f166db81a977a50a4e4e Related-Bug: #1379252
* | | | | Merge "Stop decoding object content"Jenkins2015-05-291-1/+2
|\ \ \ \ \ | |/ / / / |/| | | |
| * | | | Stop decoding object contentTim Burke2015-05-221-1/+2
| | |_|/ | |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously, we had urllib3 (via requests) automatically decode all responses with a Content-Encoding of deflate or gzip. This included object downloads, which would in turn cause etag or content-length mismatch errors. (See bug 1338464) This was apparently added in response to a third-party proxy sitting between the client and server which, having observed that the client would accept gzip-encoded content while the server sent an unencoded response, would perform the compression. (See bug 1282861) Now, we'll no longer let requests send any default headers, nor do any decoding. Change-Id: I6cc30a5c12e37de06d7322533a3c36ad15397cc8 Closes-Bug: 1338464 Related-Bug: 1282861
* | | | Merge "be more explicit in the --version CLI option"Jenkins2015-05-261-1/+1
|\ \ \ \
| * | | | be more explicit in the --version CLI optionJohn Dickinson2015-05-131-1/+1
| |/ / / | | | | | | | | | | | | Change-Id: I14f33c7ea90ab1fe58aa67f1ff70e527c88cd141
* | | | Remove simplejson dependencyTim Burke2015-05-203-16/+26
|/ / / | | | | | | | | | | | | | | | | | | In addition to removing an unnecessary dependency, this closes another hole that was allowing raw bytes to appear in user-facing messages. Change-Id: Ia0b76426a38e5a5c368c4c7e7ba2aef286758aca
* | | Merge "Use a socket timeout in get_auth"Jenkins2015-05-041-6/+13
|\ \ \
| * | | Use a socket timeout in get_authPete Zaitcev2015-04-261-6/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | A previous change added socket timeouts, but they weren't plumbed through to the get_auth code path. Make sure we're passing them along everywhere. Original-Author: Monty Taylor <mordred@inaugust.com> Change-Id: I398241898248e66d1f8c436c8ed2ec7a0e9387f6 Closes-bug: 1447847
* | | | Merge "Compare each chunk of large objects when uploading"Jenkins2015-04-302-36/+71
|\ \ \ \ | |/ / / |/| | |
| * | | Compare each chunk of large objects when uploadingTim Burke2015-03-232-36/+71
| |/ / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously, we compared the ETag from Swift against the MD5 of the entire large object. However, the ETag for large objects is generally the MD5 of the concatenation of the ETags for each segment, unless the object is a DLO whose segments span more than one page of a container listing. Rather than worry about ETags, just compare each chunk of the segmented file. This allows the use of --skip-identical when uploading SLOs and DLOs. Additionally, there are several test-related improvements: * The default arguments for OutputManager are now evaluated on construction, rather than on definition, so that TestOutputManager.test_instantiation will succeed when using nosetest as a test runner. (See also: bug 1251507) * An account_username option is now available in the functional tests config file for auth systems that do not follow the account:username format. * CaptureOutput no longer writes to the captured stream, and MockHttpTest now captures output. These were polluting test output unnecessarily. (See also: bug 1201376) Change-Id: Ic484e9a0c186c9283c4012c6a2fa77b96b8edf8a Closes-Bug: #1201376 Closes-Bug: #1379252 Related-Bug: #1251507
View Lorry Controller Status