python_web_scraping
Python web scraping project for winter break 2017
Milestone 1:
Requests, Beautiful Soup and Pagination: Writing a simple script that that extracts quotes from http://quotes.toscrape.com/ and follows pagination links.
Watch this excellent tutorial that achieves the exact same functionality using a python library called scrapy. However for milestone 1, we will be using requests and BeautifulSoup4 libraries only. Brief tutorials for both are attached here:
Milestone 1 extension:
Some websites block requests with bad user agents. Check out http://mangafreak.com/robots.txt. All well made websites have a robots.txt that define request priviledges. Figure out if http://mangafreak.com is allowing your python scraper to access content and then fool the website to download one image (Hint: change headers). Use urllib
or urllib2
python library for this script. Here are some resources to begin with:
Milestone 2:
Use Scrapy library to download chapter-wise images from http://mangafreak.com. Use ImagePipelines and tweak settings to find best configurations to download images.
Milestone 2 extension:
Enable download speed and time statistics logging. Enable proxy settings. Host on scrapinghub cloud.
Contributing
- Fork this repo
- Make a new folder
python_web_scraping/yourname/
- make a pull request and keep adding commits to it.
- pull requests will be merged after every milestone if they function properly.