This is a django web application that can scrape Product Name, Price, Image Link from Etsy product links. The scraped data is then saved into a PostgreSQL database.
It has 3 pages: index, product, products.
Index contains an input area where the link for a specific Etsy product can be submitted.
After the submission, the page will display the status (whether it was successfully uploaded to database or not).
If successful, the page will display an ID number, which can be used to specifically display products by ID numbers in product page
Contains an input area for ID to retrieve related product information The required ID will be displayed on the index page when an Etsy product link is submitted.
Displays all products that are currently stored in the database.
The products that are sold out will have a price of -1.00
-
Preferrably, create a virtual environment
python -m venv venv venv/scripts/activate
If you receive a scripts is disabled on this system
error, enable scripts on your system:
-
Set-ExecutionPolicy Unrestricted -Scope Process
- This would allow running virtualenv in the current PowerShell session.
-
Set-ExecutionPolicy Unrestricted -Force
-
This would allow running scripts, thus virtualenv for the whole system. Which is a bit unsafe. To disable it:
-
Set-ExecutionPolicy Restricted -Force
-
-
Install dependencies
pip install -r -requirements.txt
-
Run the web application
python EtsyScraper/manage.py runserver
You can access the database with pgAdmin 4
- Or, through your preferred terminal by typing
psql postgres postgres
- If prompted with
Password for user postgres:
just typepostgres
.
There is also Dockerfile
and docker-compose.yml
files available for Docker builds.
NOTE: If using Docker, make sure to change database settings in EtsyScraper/settings.py
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'NAME': 'postgres',
'USER': 'postgres',
'PASSWORD': 'postgres',
#####
# USE CASE OPTIONS
# For Docker:
'HOST': 'db', # [MAKE SURE THAT THIS IS THE HOST WHEN USING DOCKER]
# For local use:
# 'HOST': '127.0.0.1',
#####
'PORT': 5432,
}
}
-
To build without
docker-compose.yml
theDockerfile
must be edited to comment out/Docker-compose/
parts and enable/Use without docker-compose/
partsdocker build --tag <ProjectName>
docker run <ProjectName>
-
To build with
docker-compose
, run these commands in shell:docker-compose build
- builds an imagedocker-compose run --rm app
- Run this ONLY FOR THE FIRST TIME so that PostgreSQL database files can be created. After this, only repeat step1
and3
after making changes to the project.docker-compose up
- runs the image in a container