Resolv issues with googlebot sometimes about legitbot HOT 7 CLOSED

Hi,

My guess is that is is an inconsistency between the forward and the reverse DNS records. Most likely Google dynamically and frequently updates the crawler's DNS records. So my theory is that sometimes the forward resolves correctly, but the reverse returns NXDOMAIN because it was not in the cache and hence the result is more up-to-date.

Does it make sense to you?

from legitbot.

Interesting. I have only a loose understanding of dns but I'd thought it'd propagate up to a master copy if it can't find a record?

Any tips how to solve it best in a rack-attack block? Or should it be fine to just rescue the exception?

from legitbot.

I'd thought it'd propagate up to a master copy if it can't find a record?

Yes. Local / closest DNS servers cache records. Consider your closest DNS server has a forward record in its cache and returns it. But it has to go further in the chain to get the result for a reverse query and gets NXDOMAIN in return, because it does not exist anymore.

from legitbot.

Does rescuing the exception seem like a decent plan, or does it open up other bots? Or should this be something rescued in the gem itself.

from legitbot.

Yes, rescuing the exception in a calling code might open the door for other bots. It should be rescued in the same way as for forward resolv:

from legitbot.

Published as 1.6.0

from legitbot.

Awesome, thanks!

from legitbot.