andrascii / cpprobotparser Goto Github PK
View Code? Open in Web Editor NEWThis module represents the robots.txt parser written in C++, which answers questions about whether or not a particular user agent can fetch a URL on the Web site that published the robots.txt file. For more details on the structure of robots.txt files, see http://www.robotstxt.org/orig.html.
License: GNU General Public License v3.0