Only put file downloads on cloudflare DDOS protection? about browscap-site HOT 15 CLOSED

We would need a redirect for all third-party apps too.

from browscap-site.

How do non-browser clients handle the redirects and ClourFlare's "redirecting you in 5 seconds" page?

from browscap-site.

They don't, that's what pretty much kills off the traffic. It only works when the traffic comes thru in a browser.

from browscap-site.

Have you thought about using cloudflare's caching to offload 99% of the download and checking bandwidth to them instead of you? You just have to expire their cache when you push a change.

from browscap-site.

We already use CloudFlare ;)

from browscap-site.

Using Cloudflare !== using Cloudflare caching

And from version number, and download links:
Cache-Control | no-cache

and main site:
Age | 0
Cache-Control | max-age=0, public

These are saying "don't cache me, at all"

Whats missing:
CF-Cache-Status: STATUS
where STATUS is hit/miss/expired/stale/ignored/revalidated/updating

That shows its not caching, so 100% of loads of the pages/downloads have to go off of your server, using your server's bandwidth, instead of using one of Cloudflare's biggest things, caching. Since you keep having bandwidth issues, you're missing a chance to offload massive amounts of it to Cloudflare.

You can learn more: https://support.cloudflare.com/hc/en-us/articles/200168266-What-do-the-various-CloudFlare-cache-responses-HIT-Expired-etc-mean-

from browscap-site.

Hmm odd; caching is enabled;

Maybe the DDoS protection interferes with it...

from browscap-site.

Guess I will have to look into the configuration more; but indeed that may help. It's definitely not letting 100% of traffic through, as before we enabled CloudFlare we were getting many TB more of data usage, so it is helping already, it could maybe just do more :)

from browscap-site.

I've noticed the cache-control: max-age=0, public header as well (I was experimenting with putting a CloudFront CDN in front of browscap.org for our build server so builds don't fail if you enable the DDOS protection. And I'm noticing this today, without DDOS protection present). This effectively tells CloudFront to not cache the file (due to the max-age=0 setting). Is this setting something that browscap.org (before Cloudflare) is sending? If so, may want to change that to something not 0.

Ah, perhaps here: https://github.com/browscap/browscap-site/blob/master/src/BrowscapSite/Controller/StreamController.php#L111

from browscap-site.

I'd say that @jaydiablo hit the nail on the head, that its those two lines (111, and 112) or something similar, which is causing it to give that info.

Looking at Symphony on BinaryFileResponse,

setMaxAge(integer $value)
Sets the number of seconds after which the response should no longer be considered fresh.

And 0 means its already stale.

expire()
Marks the response stale by setting the Age header to be equal to the maximum age of the response.
from Response

You probably want to set a positive on "setMaxAge", plus use setExpires(DateTime) instead of expire()

setExpires(DateTime $date = null)
Sets the Expires HTTP header with a DateTime instance.

from browscap-site.

That's a good shout; yes, I remember doing this now. The logic was that we would gather download counts and so on. I certainly think it's worth me spending some time playing with these settings to determine if it makes a different on the CloudFlare side.

from browscap-site.

Maybe the initial request should collect download stats and then redirect off to something that Cloudflare can cache on CDN more effectively?

from browscap-site.

Feedback / review on #37 is welcome 👍

from browscap-site.

Fun fact: the change in #37 did basically nothing. However, I've added a page rule for the specific download links that enables caching, and I'm seeing a drop in traffic hitting the server already. It'll definitely nerf the stats, but CloudFlare has an API, so we may be able to query that to find stats that way...

from browscap-site.

Interesting, I suppose the dual URL approach would work then (put the page rule on the URL that you redirect to after collecting stats and such), but may run the risk of breaking some people's implementations if they're not setup to follow redirects.

CloudFlare API may be the way to go.

from browscap-site.