Scrape weather data from the web and store it into a local sqlite database. Work in progress! Right now there's just a single source (wunderground). Supply a URL and the script will collect the info and then store it to a local sqlite database.

Install the required packages. This could be different based on your existing environment. If something is missing, let me know! Choose the install option below that's right for your environment.

Conda Installs

1. You should create a new conda environment for running this app.
2. Run conda install --file installs.txt

Pip Installs

Run pip install -r installs.txt

The app stores the scraped data into a database. The database needs to exist before the app will use it.

Run the provided script to create the database. python3 prepare_database.py

To wipe it out and start over, use the -d argument.

python3 main.py

The path to the settings file is specified in .env. Open with a text editor if you need to relocate this file.

By default, the settings file is named settings.json and found in the local app directory. This file is used to specify the full path to the database and an array of the URLs to be collected.

To prevent spamming, the app will check each URL and the date against the database before scraping that same data again. If it has already been collected today, it won't happen. You can add additional URLs to the settings without worrying about duplicate requests.