Twitter blocks any URL with ? via. robots.txt - Filter twitter URLs to remove "ref_src" about amber_wordpress HOT 2 OPEN

Alright, so in developing our own solution for this we concluded that the only sensible option would be to add two filters to the plugin itself, which we have submitted as PR #48

These are simple and harmless unless the filter is used, but allow enormous flexibility to developers implementing Amber.

They have separate filter names (amber_lookup_url and amber_enqueue_check_link) so that they can be handled separately if necessary, but should usually have the same filtering function/method attached, since for most cases, you'll want both places to do the same manipulations on the URLs.

In the Twitter case, we can use this filter to remove ?ref_url and any other URL query variables from the URL before it is checked (so the check has a chance to pass) as well as when the link is checked on display (so that it will match the database entry for that URL, which otherwise would be difference since the database would have the sans-ref_url version but the display checker would have the has-ref_url version).

This filter can also be used for all kinds of other things too! It could be used to do a very nuanced "blacklist" task that involved checking something more elaborate than Regex, it could be used to completely replace one domain with another etc.

Please, I beg of you, add this filter to the plugin.

If you like, you can also incorporate the Twitter filter, so that Twitter will work better for all your users. If not that's fine though, having access to the filter is much more important for us.

from amber_wordpress.

NOTE: looking at our error log, I'm seeing a lot of this, despite running the filter described above that avoids triggering twitter.com/robots.txt:

[28-Jun-2018 15:21:31 UTC] [...]/wp-content/plugins/amberlink/libraries/AmberChecker.php:AmberChecker::check:Blocked by robots.txt:https://t.co/wnn0C8bIVr 

Quick look at http://t.co/robots.txt explains it:

User-agent: twitterbot
Disallow:

User-agent: *
Disallow: /

So it seems any Twitter link in the t.co format will always die, which is a bummer.

Possibility to fix: Have a filter (like the one above) that does the full permalink expansion of t.co URLs before doing anything with them, so that the system only gets the working twitter.com version.

Haven't done it yet but documenting it here for myself and/or others.

from amber_wordpress.