This repository contains code for a webscraper for Lifetime Leaderboards | UMG Gaming Making use of the beatifulsoup4 and requests Postgres as a Database SQLAlchemy is used as a ORM to insert data into the db
Get Started with a Web Scraping Project
- Clone this Repo
git clone https://github.com/KenMwaura1/simple-web-scraper
- Change into into the simple-web-scraper folder
cd simple-web-scraper
- Create a virtualenv
python3 -m virtualenv env
- Activate virtualenv
source /bin/activate
OR use pipenv pipenv install
- Install the required dependecies
python3 -m pip install -r requirements.txt
Ensure Postgres is installed locally and running on port 5432
OR
run a Postgres container on Docker Easy PostgreSQL 10 and pgAdmin 4 Setup with Docker container on port 5432
The default db credentials are:
- host: localhost
- port: 5432
- user: test
- db: xp_db
- password: testpassword
Optionally can also set the db credentials as environment variables
export DATABASE_URL='postgres://test:testpassword@localhost:5432/xp_db'
or copy the included .env example file
cp .env.example .env
Add your credentials to the .env file.
1.Run the scraper
1a. To run the scraper and insert xp data into the db
python xp_scrape.py
1b. To run the scraper and insert earnings data into the db
python earnings_scrape.py
To run the tests run the following command at the root of the project.
pytest .