summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* set loglevel as opt and add set_loglevel to urlopen and urlreadHEADmastermbussolotto2022-11-281-1/+18
|
* setup.py: Update urlJohn Vandenberg2022-09-021-1/+1
| | | http://urlgrabber.baseurl.org/ doesnt link to the current SCM
* Avoid using system proxy when proxy set to _none_Pablo Suárez Hernández2022-09-021-1/+1
|
* Fix the find_proxy bit different wayVictor Zhestkov2022-09-021-7/+7
|
* Fix wrong logic for find_proxy methodPablo Suárez Hernández2022-09-021-4/+6
|
* Drop six usagePablo Suárez Hernández2022-09-028-33/+20
|
* Convert dict_keys to list to not crash when delegate is enabledPablo Suárez Hernández2022-04-191-1/+1
|
* Use binary mode when reopening filesPablo Suárez Hernández2022-01-291-1/+1
|
* Respect ssl_verify_host set to FalseMarek Brychta2021-07-261-0/+2
|
* Fix for TextMeter as progress_optionJochen Breuer2020-04-021-2/+2
| | | | | | | | | Type errors would prevent TextMeter from being used as a progess_option. This commit fixes this with a type cast and an initialization of a variable as int. Fixes #24
* Decode bytes to a string for ftplib.parse150() (RhBug:1734527)Lukáš Hrázký2020-01-271-1/+9
| | | | | | | ftplib.parse150() expects a string, we need to decode the bytes before passing them to the function in Python 3. https://bugzilla.redhat.com/show_bug.cgi?id=1734527
* Enable copr builds and add packit configDominika Hodovska2019-11-222-0/+434
|
* Release 4.1.0urlgrabber-4-1-0Neal Gompa2019-10-082-1/+12
|
* setuptools: Update Development Status to "Production/Stable"Neal Gompa2019-10-081-1/+1
| | | | | | This module has been in use with YUM for well over a decade, and the API has not changed much in the last few years. At this point, most people would consider it production-grade.
* Revise setup.py to remove need for extra setup-time dependenciesNeal Gompa2019-10-062-72/+70
| | | | | | This rework removes the cyclic dependency on the urlgrabber code to install urlgrabber, while still supporting the propagation of setuptools properties into the module.
* Fix issue with URLGRABBER_DEBUG on urlgrabber scriptPablo Suárez Hernández2019-10-011-4/+4
|
* Fix issue when URLGRABBER_DEBUG is not an integer on Python3Pablo Suárez Hernández2019-09-301-2/+2
|
* Fix for usage of _levelNames from logging moduleJochen Breuer2019-09-062-2/+8
| | | | With Python3 the internal dict has been renamed to _levelToName.
* Support HTTP CONNECT with reget. BZ 1585596Michal Domonkos2019-08-241-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | Currently, we would reset the file upon seeing "200" in the response header. This is, however, easily fooled by the HTTP CONNECT method used to access an SSL server on behalf of a proxy (a common setting would be a company intranet behind a proxy server where an internal system is consuming Red Hat CDN repos with yum). The reason is that, in this protocol, there are two subsequent headers sent, the first of which is: "HTTP/1.1 200 Connection established". Therefore, we need to explicitly check for "200 OK". More details: https://tools.ietf.org/html/rfc7231#section-4.3.6 Kudos to Masahiro Matsuya for suggesting this patch! Note: As an alternative solution, it seems that setting the CURLOPT_SUPPRESS_CONNECT_HEADERS option on the curl handle would also do the trick (but that would require more scrutiny to ensure that nothing else breaks): https://curl.haxx.se/libcurl/c/CURLOPT_SUPPRESS_CONNECT_HEADERS.html
* test: handle unknown file content in test_retry_no_cacheMichal Domonkos2019-05-211-1/+6
|
* urlgrabber-ext-down: convert url into bytesMichal Domonkos2019-05-212-3/+3
| | | | | | | | | | | We need to convert the parsed url back into bytes before passing it to the PyCurlFileObject constructor (since _set_opts() expects self.scheme, constructed from the url, to be a bytes object). This caused the unit test "bypassing proxy cache on failure" to fail (together with a bug in the test itself which is also being fixed here). Closes #14.
* Revert "Simplify mirror conversion to utf8"Michal Domonkos2019-05-201-1/+6
| | | | | | | | | | | | | This reverts commit be8ee10e35319e80200d4ff384434d46fe7783d9. A list of dicts (as opposed to strings) is valid input as well; see the module-level doc string for details (section 2 under CUSTOMIZATION). In fact, the nested estimate() function in MirrorGroup.__init__() accounts for that, too. This fixes a traceback in YUM which does pass such a dict list. Closes #10.
* urlgrabber-ext-down: another python 3 compatPavel Raiskup2019-05-121-1/+8
| | | | | | | Expect that _readlines() returns array of bytes objects in Python 3 environments. Fixes rhbz #1707657 and #1688173
* Fix the confused license header in urlgrabber/__init__.py fileNeal Gompa2019-02-271-11/+13
| | | | | | | | | | urlgrabber was relicensed to LGPLv2+ a long time ago, and this file seemed to have only a partially updated license header for this. It references the GNU Library General Public License, so clearly the intent was the header was supposed to be fully updated for it, so let's just do that by switching to the header used in the other source files.
* Release 4.0.0urlgrabber-4-0-0Neal Gompa2019-02-252-2/+11
|
* Define setup_requires in setup.py and add six to install_requiresNeal Gompa2019-02-251-1/+2
|
* Raise an obvious error message when "urlgrabber-ext-down" is not installedMatthew Prahl2019-02-251-0/+5
|
* Add curl_obj option. BZ 1204825Michal Domonkos2019-02-251-1/+23
|
* Fix setup.py to use setuptools and have correct metadataNeal Gompa2019-02-251-3/+4
|
* makefile: detect modern Python 2 and 3 releasesNeal Gompa2019-02-251-7/+7
|
* Merge pull request #9 from keszybz/py3kNeal Gompa (ニール・ゴンパ)2019-02-2521-20880/+963
|\ | | | | Python3 compatibility
| * tests: do not exit with "success" on errorZbigniew Jędrzejewski-Szmek2019-02-241-2/+2
| | | | | | | | 2 is the standard code for "command line usage error".
| * Drop some unnecessary continuation backslashesZbigniew Jędrzejewski-Szmek2019-02-243-21/+19
| | | | | | | | When the expression is already in parentheses, the backslash has no effect.
| * Replace some type() with specific class namesZbigniew Jędrzejewski-Szmek2019-02-244-6/+6
| | | | | | | | | | We know what the types of basic types are, let's just put that directly in the code. It seems more idiomatic and slightly more efficient to do things this way.
| * Apply 'methodattrs' fixerZbigniew Jędrzejewski-Szmek2019-02-241-1/+1
| | | | | | | | Just a cleanup.
| * Apply 'asserts' fixerZbigniew Jędrzejewski-Szmek2019-02-243-86/+86
| | | | | | | | This just updates unittest method names.
| * test_grabber: enable the post testZbigniew Jędrzejewski-Szmek2019-02-241-1/+0
| |
| * test_grabber: define try..except block in test more narrowlyZbigniew Jędrzejewski-Szmek2019-02-241-19/+22
| | | | | | | | | | Bare except: is never nice. Also, let's put the try..except block around only the urlgrab() call.
| * py3: avoid "unbound variable" issueZbigniew Jędrzejewski-Szmek2019-02-241-4/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Under python3, the variable defined in "except ... as ..." is only valid until the end of the block. In this case it would undefine the variable that was defined above, not reassign it as under python2, leading to the following tb: Traceback (most recent call last): File "test/test_mirror.py", line 379, in test_retry_no_cache urlgrabber.grabber.parallel_wait() File "test/../urlgrabber/grabber.py", line 2374, in parallel_wait perform() File "test/../urlgrabber/grabber.py", line 2313, in perform if ug_err is None: UnboundLocalError: local variable 'ug_err' referenced before assignment
| * Do encoding/decoding in the subprocess callsZbigniew Jędrzejewski-Szmek2019-02-241-6/+11
| | | | | | | | | | s.endswith('x') works properly with bytes. s[-1] == 'x' doesn't, because s[-1] returns an integer. But .endswith() is clearer anyway, so that's OK.
| * Switch test URLs to a private serverZbigniew Jędrzejewski-Szmek2019-02-241-1/+2
| | | | | | | | | | The URLs under http://urlgrabber.baseurl.org/ are all broken. Let's switch until they are fixed.
| * Simplify quoter function declarationZbigniew Jędrzejewski-Szmek2019-02-241-7/+6
| | | | | | | | | | There isn't much point in defining a nested function which doesn't use its closure for anything. Also doing it just once is generally quicker.
| * Allow overriding the path to urlgrabber-ext-down with an env varZbigniew Jędrzejewski-Szmek2019-02-241-1/+3
| | | | | | | | | | | | | | | | | | Otherwise, the tests try to call the installed executable, which might be stale or not exist at all. I now run the tests with: URLGRABBER_EXT_DOWN=scripts/urlgrabber-ext-down PYTHONPATH=. python2 test/runtests.py URLGRABBER_EXT_DOWN=scripts/urlgrabber-ext-down PYTHONPATH=. python3 test/runtests.py
| * Replace number type enumeration by generic abc checkZbigniew Jędrzejewski-Szmek2019-02-241-1/+2
| | | | | | | | numbers existed already in 2.6.
| * Make the 'broker' mirror really brokenZbigniew Jędrzejewski-Szmek2019-02-245-20004/+1
| | | | | | | | | | | | | | | | | | I guess those files are not supposed to be there, because otherwise the test don't get the exception they expect. The test is updated to expect 404, not 403, because the file is simply missing. It would be possible to change the server configuration to return 403, but this doesn't seem particularly important for this test. So let's just ask for a file that doesn't exist and expect 404.
| * test_mirror: do not use a fixed port for the internal test serverZbigniew Jędrzejewski-Szmek2019-02-241-6/+14
| | | | | | | | | | | | | | | | | | | | | | The tests were fairly consistently failing with: error: [Errno 98] Address already in use Actually the port is a well-known port that could be used be some other program. So let's simplify things by opening a random port. It'd be nice to use the socket as a context manager, but unfortunately python2 does not support that.
| * Use a failing URL in two tests that are supposed to failZbigniew Jędrzejewski-Szmek2019-02-241-2/+2
| | | | | | | | | | | | IIUC, those tests called urlgrab or urlread with a valid address or a file name and expected it to fail. Let's call it with a URL that return 404 instead. Now the tests pass.
| * Add explicit encode/decode callsZbigniew Jędrzejewski-Szmek2019-02-245-20/+41
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is the patch that has the most potential for trouble in the whole series, because it affects python2 behaviour directly. io.BytesIO is the same as StringIO under python2, so this should have no effect on python2. Under python3 it is necessary to allow reading bytes from a byte data source. Under python2, encoding of an already encoding string is allowed, and actually works fine (is idempotent) for ASCII strings. So the effect of the .encode() calls under python2 should be limited. Under python3, they are of course necessary and can only be done once. So if there are errors here, they should show up when running under python3 pretty easily.
| * Preserve type in URLParser.quote()Zbigniew Jędrzejewski-Szmek2019-02-241-2/+4
| |
| * Add a wrapper function around urlunquote() that decodes automaticallyZbigniew Jędrzejewski-Szmek2019-02-241-4/+9
| |
View Lorry Controller Status