summaryrefslogtreecommitdiff
path: root/Lib/urllib/robotparser.py
Commit message (Expand)AuthorAgeFilesLines
* bpo-35922: Fix RobotFileParser when robots.txt has no relevant crawl delay or...RĂ©mi Lapeyre2019-06-161-2/+6
* bpo-21475: Support the Sitemap extension in robotparser (GH-6883)Christopher Beacham2018-05-161-0/+12
* bpo-32861: urllib.robotparser fix incomplete __str__ methods. (GH-5711)Michael Lazar2018-05-141-5/+12
* bpo-31325: Fix usage of namedtuple in RobotFileParser.parse() (#4529)Berker Peksag2017-11-231-5/+4
* Issue #25400: RobotFileParser now correctly returns default values for crawl_...Berker Peksag2016-09-181-2/+6
* Merge typo fixes from 3.5Martin Panter2016-05-081-1/+1
|\
| * Fix typos in comments, documentation and test method namesMartin Panter2016-05-081-1/+1
* | Issue #16099: RobotFileParser now supports Crawl-delay and Request-rateBerker Peksag2015-10-081-2/+37
|/
* Issue 21469: Minor code modernization (convert and/or expression to an if/el...Raymond Hettinger2014-05-121-1/+1
* Issue 21469: Mitigate risk of false positives with robotparser.Raymond Hettinger2014-05-121-2/+9
* #17403: urllib.parse.robotparser normalizes the urls before adding to ruleline.Senthil Kumaran2013-05-291-0/+1
* #4108: the first default entry (User-agent: *) wins.Georg Brandl2010-07-291-2/+4
* Fix Issue6325 - robotparse to honor urls with query strings.Senthil Kumaran2010-07-281-2/+4
* Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 ...Benjamin Peterson2008-07-311-0/+5
* Bug 3347: robotparser failed because it didn't convert bytes to string.Jeremy Hylton2008-07-181-2/+6
* Make a new urllib package .Jeremy Hylton2008-06-181-0/+191