Searched refs:RobotFileParser (Results 1 – 11 of 11) sorted by relevance
20 This module provides a single class, :class:`RobotFileParser`, which answers26 .. class:: RobotFileParser(url='')89 The following example demonstrates basic use of the :class:`RobotFileParser`93 >>> rp = urllib.robotparser.RobotFileParser()
21 self.parser = urllib.robotparser.RobotFileParser()339 parser = urllib.robotparser.RobotFileParser()354 cls.parser = urllib.robotparser.RobotFileParser(cls.robots_txt)378 parser = urllib.robotparser.RobotFileParser(self.url('i-robot.txt'))
22 class RobotFileParser: class
308 :meth:`RobotFileParser.parse() <urllib.robotparser.RobotFileParser.parse>`
491 RobotFileParser now correctly returns default values for crawl_delay and
388 :meth:`RobotFileParser.parse() <urllib.robotparser.RobotFileParser.parse>`
3036 Fix :meth:`RobotFileParser.crawl_delay` and3037 :meth:`RobotFileParser.request_rate` to return ``None`` rather than raise
4769 Added support for Site Maps to urllib's ``RobotFileParser`` as4770 :meth:`RobotFileParser.site_maps()4771 <urllib.robotparser.RobotFileParser.site_maps>`. Patch by Lady Red, based on
2725 RobotFileParser now supports Crawl-delay and Request-rate extensions. Patch
4484 RobotFileParser now correctly returns default values for crawl_delay and
1681 :class:`~urllib.robotparser.RobotFileParser` now supports the ``Crawl-delay`` and