Scrapes the basic info (mainly business hours) of all Unisport gyms.
Why you ask? Because as of now you can't view them all on one page and they are two clicks away from the frontpage. Who designed such moronic system, I don't know, but to me it's annoying since I have no interest in other stuff than seeing how long its open today.
Requires Python >=3.6 with preferably virtualenv. I use virtualenv-wrapper
, some have said pyenv
is pretty good too.
- Activate your virtualenv eg
workon varjos
- Install dependencies:
pip install -r requirements.txt
(Scrapy and Twisted) - Load the dev commands:
. cmds.sh
- Run the spider:
crawl
You should get ./frontend/unisport_gyms.json
file with the data scraped.
Use shell
to open interactive Scrapy shell to test CSS selectors without having to run the spider.
Requires Node.js >=10.
- Run:
node server.js
- The server should run at http://localhost:4040/
Reload the page after making changes to the files inside frontend
-folder.
Requires AWS account and one S3 bucket.
- Set the bucket's permissions to allow public bucket access and enable static website hosting
- Configure your local AWS user with access to that bucket
- You should replace the bucket name in my
cmds.sh
script with your own - Then deploy the code with
AWS_PROFILE=varjosport.net-ci deploy_front
where AWS_PROFILE is your local AWS profile.
Go to the bucket's website URL to see the app running eg http://varjosport.net.s3-website.eu-north-1.amazonaws.com
If you want to deploy it to your own domain incase this one dies out for some reason, you have to configure Route 53, Certificate Manager, and CloudFront too. Pretty basic configuration, so I didn't bother writing it down as a template.
Requires Node.js >=10. Docker if you want to deploy it. AWS account with one local AWS user with admin privileges (I'm lazy) and one S3 bucket (the same bucket you use to host the frontend).
- Install dependencies:
npm i
- Run
npm run invoke
to execute the lambda. It will most probably fail because I hard-coded the profile and the bucket. Change them to your own - Similar to the previous command,
npm run deploy
will deploy the lambda but I've hard-coded the parameters
Once deployed, instead of waiting 24 hours for the lambda to run, you can trigger it manually by going to your AWS console's Lambda page for this function and creating & sending a test event.