Searched refs:robotparser (Results 1 – 18 of 18) sorted by relevance
| /external/python/cpython3/Lib/test/ |
| D | test_robotparser.py | 5 import urllib.robotparser
21 self.parser = urllib.robotparser.RobotFileParser()
117 urllib.robotparser.RequestRate
145 request_rate = urllib.robotparser.RequestRate(9, 30)
270 request_rate = urllib.robotparser.RequestRate(3, 15)
343 parser = urllib.robotparser.RobotFileParser()
359 cls.parser = urllib.robotparser.RobotFileParser(cls.robots_txt)
383 parser = urllib.robotparser.RobotFileParser(self.url('i-robot.txt'))
|
| /external/python/cpython3/Doc/library/ |
| D | urllib.robotparser.rst | 1 :mod:`!urllib.robotparser` --- Parser for robots.txt
4 .. module:: urllib.robotparser
10 **Source code:** :source:`Lib/urllib/robotparser.py`
92 >>> import urllib.robotparser
93 >>> rp = urllib.robotparser.RobotFileParser()
|
| D | urllib.rst | 15 * :mod:`urllib.robotparser` for parsing ``robots.txt`` files
|
| D | internet.rst | 28 urllib.robotparser.rst
|
| /external/python/six/documentation/ |
| D | index.rst | 700 | ``urllib.robotparser`` | :mod:`py2:robotparser` | :mod:`py3:urllib.robotpars…
702 | ``urllib_robotparser`` | :mod:`py2:robotparser` | :mod:`py3:urllib.robotpars…
|
| /external/python/cpython3/.github/ |
| D | CODEOWNERS | 187 **/*robotparser* @berkerpeksag
|
| /external/python/six/ |
| D | test_six.py | 185 assert item_name in dir(six.moves.urllib.robotparser)
186 getattr(six.moves.urllib.robotparser, item_name)
|
| D | six.py | 506 robotparser = _importer._get_module("moves.urllib_robotparser") variable in Module_six_moves_urllib
|
| /external/python/parse_type/tasks/_vendor/ |
| D | six.py | 477 robotparser = _importer._get_module("moves.urllib_robotparser") variable in Module_six_moves_urllib
|
| /external/python/cpython3/Misc/NEWS.d/ |
| D | 3.7.0b5.rst | 439 The urllib.robotparser's ``__str__`` representation now includes wildcard
|
| D | 3.6.6rc1.rst | 398 The urllib.robotparser's ``__str__`` representation now includes wildcard
|
| D | 3.6.4rc1.rst | 308 :meth:`RobotFileParser.parse() <urllib.robotparser.RobotFileParser.parse>`
|
| D | 3.7.0a3.rst | 388 :meth:`RobotFileParser.parse() <urllib.robotparser.RobotFileParser.parse>`
|
| D | 3.8.0a1.rst | 4771 <urllib.robotparser.RobotFileParser.site_maps>`. Patch by Lady Red, based on
5247 The urllib.robotparser's ``__str__`` representation now includes wildcard
|
| D | 3.5.0a1.rst | 3733 Reduced the risk of false positives in robotparser by checking to make sure
|
| /external/python/cpython3/Doc/whatsnew/ |
| D | 2.0.rst | 1112 * :mod:`robotparser <urllib.robotparser>`: Parse a :file:`robots.txt` file, which is used for writi…
|
| D | 3.6.rst | 1682 urllib.robotparser
1685 :class:`~urllib.robotparser.RobotFileParser` now supports the ``Crawl-delay`` and
|
| /external/python/cpython3/Misc/ |
| D | HISTORY | 1922 - Issue #21469: Reduced the risk of false positives in robotparser by
4919 - Issue #17403: urllib.parse.robotparser normalizes the urls before adding to
13170 - Issue #4108: In urllib.robotparser, if there are multiple ``User-agent: *``
16691 ``urllib``, ``urllib2``, ``urlparse``, and ``robotparser``. The old
16694 ``urllib.error``, and ``urllib.robotparser``. The
26780 robotparser - parse a robots.txt file, for writing web spiders.
|