nberlette / f1 Goto Github PK

I've noticed a couple instances of the oxblue.com photo stream being down recently. During these time periods, an outdated image in the response is coupled with inaccurate header metadata, making it seem as if its new content.

I need to add a deduplication step that checks the incoming bytes themselves and only proceeds with the scrape if they are new when compared to the most recent scrape. I've had to manually disable the workflow for now until the server returns to normal operation, in the meantime I'll work on a fix for this. I'll also remove all of the duplicates that have amassed during the last couple of outages.

Already have this mostly sketched out, it just isn't tested yet. And I need to figure out what to do with the files once generated since they're quite large (easily > 100mb each for multi-day timelapses).

I know nobody is going to read this but if you have any ideas, please chime in with them. Unless it's S3. If you're going to suggest S3, do not chime in. Thanks

Prior to the recent merge of PR #4, the scraping process was terribly inefficient and lazy. It did not perform anything beyond a rudimentary size comparison to determing if it had encountered new content, leading to several thousand duplicate images being collected over the last couple months.

While the new logic from the recent merge addresses the root issue, and prevents any duplicates from being collected now, it did not address those 3,500ish images that had accumulated prior to it.

This issue is a reference point for an incoming PR that will remedy that, removing all duplicates and thereby reducing the project size by a couple hundred MB.

Stay tuned

It seems that about 1 in every 5-10 runs is currently failing to complete, and that's after it has run for an average of 10x minutes longer than a typical successful run.

A normal workflow run without any issues, takes about 45-60 seconds to successfully complete from start to finish. In comparison, the average failed run is currently taking over 6 minutes and 30 seconds before it gives up and fails.

This has something to do with the "retries" logic of the scraper; I have a hunch that when the first attempt fails to retrieve new content, it is running into cached content while making its additional attempts.

I'm investigating the issue currently and will be pushing through a PR to fix it as soon as I can.