Searched refs:RobotFileParser (Results 1 – 5 of 5) sorted by relevance
23 This module provides a single class, :class:`RobotFileParser`, which answers29 .. class:: RobotFileParser(url='')69 The following example demonstrates basic use of the RobotFileParser class. ::72 >>> rp = robotparser.RobotFileParser()
43 parser = robotparser.RobotFileParser()270 parser = robotparser.RobotFileParser()283 parser = robotparser.RobotFileParser(
19 class RobotFileParser: class
332 self.robots[root] = rp = robotparser.RobotFileParser()
11145 RobotFileParser no longer prompts interactively for a username and