Crawl IMDB user's review
All runnable scripts are in: scripts link
Running steps:
- crawl_movie_list.sh : execute module movies_spider, crawl a list of movie(& its information) from the link
http://www.imdb.com/search/title?year={year},{year}&title_type=feature&sort=num_votes,desc
- crawl_reviews_list.py : read crawled data (from 1.) that is stored in
scrapyIMDB/data/movie_list.csv
and executecrawl_reviews.sh
- crawl_reviews.sh : execute module reviews_spider, crawl a list of reviews associated with the given movie