Enum scripts used for information gathering.
Used for enum on the robots.txt page. Download spefific robots.txt Disallow directorys, files or pages or download every single robots.txt file that recieves a 200 OK response code. Also find out which pages may be giving you a 'bad' response code for further inspecting and enumeration.
Options
help - help
list - list all Disallow robots.txt directories that recieved a 200 OK response code (Enter corrosponding number with listed Directories to download indivdual directories,files or pages.)
bad - list all all Disallow robots.txt directories that received any other response code than a 200 OK response.
download all - Download all robots.txt directories that recieved a 200 OK response code.
Use regex to grab all subdomains on a given domain name.