bedrockstreaming / roboxt Goto Github PK

As title, to let class behaviors be reliable, we should consider integrating unit tests.

According to the draft http://www.robotstxt.org/norobots-rfc.txt
User-agent names are case-insensitive.
Currently User-agent of 'Crawler' cannot read the record like:

User-agent: crawleR
Disallow: /no

Also if no matching record is found (i.e. no '*' record), the following error is occurred when you call isUrlAllowedByUserAgent().
Error: Call to a member function isUrlAllowed() on null in /path/to/vendor/m6web/roboxt/src/Roboxt/File.php:90

If there is no matching directive for the given user agent, it should fallback on "*" and do the same test.

As title, since the php-7.0 versions has been released for whiles, we should consider dropping php-5.x versions.

When using preg_match(), preg_quote($pattern, '@') is expected to be used to escape input.
Currently a warning like any of the following occur when a path contains some meta character:

PHP Warning: preg_match(): Compilation failed: missing ) at offset 13 in /path/to/vendor/m6web/roboxt/src/Roboxt/Directive/Directive.php on line 84
PHP Warning: preg_match(): Compilation failed: unmatched parentheses at offset 2 in /path/to/vendor/m6web/roboxt/src/Roboxt/Directive/Directive.php on line 84
PHP Warning: preg_match(): Unknown modifier 'c' in /path/to/vendor/m6web/roboxt/src/Roboxt/Directive/Directive.php on line 84

From the example:

$parser = new \Roboxt\Parser();
$file = $parser->parse("http://www.google.com/robots.txt");

Perhaps it would be more logical for the parse() method to accept the robots.txt content as string and add a new load() method, to load an url directly. Currently, you can't really use this library when you have the content already, or use a different http client (instead of file_get_contents()).

Or if you don't want to break BC, you could add a new method. But name of parse() doesn't imply it does any fetching of the url..