robots_txt (definition) in projects: external

Searched defs:robots_txt (Results 1 – 2 of 2) sorted by relevance

/external/python/cpython3/Lib/test/
D	test_robotparser.py	13 robots_txt = '' variable in BaseRobotTest 47 robots_txt = """\ variable in UserAgentWildcardTest 58 robots_txt = """\ variable in CrawlDelayAndCustomAgentTest 75 robots_txt = """\ variable in SitemapTest 92 robots_txt = """\ variable in RejectAllRobotsTest 130 robots_txt = '' variable in EmptyFileTest 135 robots_txt = """\ variable in CrawlDelayAndRequestRateTest 157 robots_txt = """\ variable in InvalidRequestRateTest 174 robots_txt = """\ variable in InvalidCrawlDelayTest 186 robots_txt = """\ variable in AnotherInvalidRequestRateTest [all …]
/external/python/cpython2/Lib/test/
D	test_robotparser.py	14 robots_txt = '' variable in BaseRobotTest 42 robots_txt = """\ variable in UserAgentWildcardTest 53 robots_txt = """\ variable in RejectAllRobotsTest 66 robots_txt = """\ variable in UserAgentOrderingTest 84 robots_txt = """\ variable in GoogleURLOrderingTest 96 robots_txt = """\ variable in DisallowQueryStringTest 106 robots_txt = """\ variable in UseFirstUserAgentWildcardTest 119 robots_txt = """\ variable in EmptyQueryStringTest 129 robots_txt = """\ variable in DefaultEntryTest 140 robots_txt = """\ variable in StringFormattingTest [all …]