robots.txt matching is pretty buggy about crawly HOT 10 OPEN

@fstp Please let me know if I can help you in any way.

from crawly.

@serpent213 yeah, I forked the Gollum a few years ago due to the same reason. I had the plan to fix the parsing there but did not have enough time to do that. Unfortunately :(.

from crawly.

Hi,

I would like to pick this task and start improving the parser if that is okay.
@serpent213 could you resubmit your pull request to the forked repo from oltarasenko?
(https://github.com/elixir-crawly/gollum)

After it has been merged we can start to work on improving the parser in that fork since it seems like the original project is abandoned by now.

from crawly.

@fstp See elixir-crawly/gollum#1.

from crawly.

Thank you for the quick response! :)
@oltarasenko if you want you could merge the pull request from @serpent213 and then I can create my pull request based on that. Or is it better that I push directly to the pull request?

from crawly.

One of the commits introduces a failing test, IIRC. The Google-port has a lot of stuff tagged as skip to pass.

from crawly.

Ah okay I see, can I clone your pull request somehow and push the code directly there? Would be nice to run test driven development against the failing testcases

from crawly.

Maybe you can start your own branch and cherry-pick the commits?

Edit: Added you to my fork, @fstp, not sure that is of any help. Note that my fork is based on https://github.com/ravern/gollum, but it's rebased to https://github.com/elixir-crawly/gollum/.

from crawly.

Thanks! I will try to work directly on your fork, seems most convenient since we can just merge it all together then :)

from crawly.