4xx404 / pyrobots Goto Github PK
View Code? Open in Web Editor NEWpyRobots will search a website for robots.txt, if found, it will download the robots.txt file. It will then iterate through each entry & attempt to download all disallowed files/directories listed in robots.txt.