meklu / rbt_prs Goto Github PK
View Code? Open in Web Editor NEWA robots.txt parser written in PHP, useful for checking whether or not your bot has access to a given URL. Should probably be cleaned up a little in the future™ due to being a little on the messier side.