This is an autonomous image scraper developed using TypeScript, Deno, and GitHub Actions. It was purpose-built to document the historic Formula 1 track construction in Las Vegas, Nevada, slated to host the inaugural Heineken Silver Grand Prix on November 18th. The images will be stitched together into timelapse videos of the track's lifecycle, from start to finish 🏁.

Note: this video was created with images from 2023-08-15 - 2023-10-12

The first scrape happened on June 3rd, 2023. As of October 18th it has surpassed 18,500 commits, equivalent to over 1.2GB of image data. Photos are stored in the ./assets folder of this repository, and also persisted to a Deno KV database backed by FoundationDB.

The origin of the scraped images is an real-time photo feed, sourced directly from the official Formula 1 website.

⚠️ This project is for educational purposes and is not affiliated with Formula 1.

• Deno v1.37.2
• Rust-based JS runtime, sandboxed, with great TS/TSX support.
• Provides the tools for network and file system operations.
• TypeScript 5.2.2
• Superset of JavaScript featuring advanced static typechecking.
• Better type safety means more readable and maintainable code.
• GitHub Actions
• Provides free macOS virtual machines powering the scraper.
• Responsible for scheduled execution of the scraper workflow
• Temporarily stores the image artifacts
• Deno KV (currently in beta)
• Provides us with global data persistence and caching
• ffmpeg (timelapse feature is unstable)
• Leveraged to automatically generate timelapse videos

AI-generated F1 art created with SDXL 1.0 and the prompt "Formula 1 cars on the Las Vegas Strip"

The majority of the work happens in main.ts, despite it only being 3 lines of code. It is responsible for invoking the scraper located in src/scrape.ts, and is ran every 10 minutes by a GitHub Action defined by the workflow in main.yml.

Images are named after their capture time as a JPEG file in UTC. For example, an image captured at 2023-07-09T04:28:57 would be saved as ./assets/2023-07-09/04_28_57.jpg. The latest image is always saved as ./assets/latest.jpg for easy access.

1. GitHub Actions runs the scrape workflow every ~10 minutes, depending on traffic.
2. The runner checks out the repo, installs Deno, and prepares to scrape.
3. deno task scrape is executed, which runs the main.ts file.
4. main.ts imports scrape() from src/scrape.ts, which defines two inner functions, read and write.
   • Once it has checked that import.meta.main is set, the following steps are taken:
   1. 🔍 READ: read() is called with IMAGE_URL.
      • Internally, the Fetch API is used to download the image.
        If the request fails, it will be retried up to ATTEMPTS times, with a short pause between each successive attempt.
      • If all attempts are exhausted without success, the run will terminate.
      • Otherwise, a new instance of the Image class is returned.
   2. 💾 WRITE: write() is called, with the Image as its only argument. Before writing, however, it runs through some checks:
      1. The Image.hash is checked against the hash "table" in Deno KV.
         If an entry exists, the image is stale and won't make it any further. If Deno KV is unavailable, the image data is checked against latest.jpg via a timing-safe equality comparison, avoiding exposure to timing-based attacks.
         • If they are equal, the image has not updated at the origin. The process starts over at step 4 and repeats until a new image is found.
         • If the maximum number of ATTEMPTS is reached and no new image was found, the job terminates unsuccessfully.
      2. If we've made it this far, we have a fresh image and we need to store it.
         • Image.write() persists the image to Deno KV.

           The key is generated by the Image API, using the image timestamp.

         • The image timestamp is indexed with its unique SHA-256 hash in Deno KV.

           This prevents later scrapes from duplicating this image. It also means if you try to instantiate a new Image from an old hash, it will always return the original image and its original timestamp.

         • Image.writeFile() saves it to the local file system.

           The filename is generated by the Image API, using the image timestamp.

         • Image.writeFile() also saves it to ./assets/latest.jpg,
      3. The setOutput helper pipes the image metadata to the GitHub Actions runner, to be used in the commit step.
   3. The scrape is now complete and the runner proceeds to the final steps.
5. The photo is stored as a workflow artifact for 90 days.
6. 🤝 All changes are committed + pushed to the repository.
7. 🏁 The job finishes successfully and the runner is terminated. Hooray!

October 17th	October 15th	October 13th	October 11th
October 9th	October 7th	October 5th	October 3rd
October 2nd	September 28th	September 24th	September 20th
September 16th	September 12th	September 8th	September 4th

This project is not affiliated with Formula 1 and is for educational purposes only.