Near-constant download of uploaded stories content apparent in S3/AWS logs about covid-histories HOT 3 CLOSED

As far as I can tell, the Repair AV workflow was causing this data transfer. My understanding of this problem is as follows:

1. We had a lot of videos (24) in the queue from the admin-upload process, and the workflow doesn't commit the ids of completed videos if it is unexpectedly killed mid-way through.
2. We had some videos over 1gb in size, and it was while transcoding one of these that the workflow process was killed every time, presumably due to the file size.
3. Because the workflow was scheduled hourly, a new run would start (often even before the first run had failed), and begin working on the exact same videos until it failed by hitting the very large files.

For now, I have gone through the 24 un-transcoded videos in s3 and put the ids of those that are >= 1gb in size into a new txt file that the repair av script will reference and use to skip processing those videos. Ultimately, we'll need to compress and re-upload those particular videos.

from covid-histories.

@mukeshchugani10 One change to the code that would help address issues like this in the future would be a modifications of the workflow script such that each time a video is processed successfully, the updated list of video ids is committed to the repo. As far as I can tell, this commit only happens after all of the videos have been processed, which ultimately was what caused this particular issue.

We can set up a different ticket/workflow for handling very large (>1gb) video files, though, arguably, it would be better this size is never uploaded to s3 and we have some preprocessing, etc. Will figure that out with the next batch of uploads.

from covid-histories.