Light

andorfermichael / tripadvisor-scraper Goto Github PK

View Code? Open in Web Editor NEW

35.0 3.0 14.0 49 KB

Scrape the hotel reviews of a whole city on TripAdvisor

Python 100.00%

tourism tripadvisor scraper scraping python

tripadvisor-scraper's Introduction

TripAdvisor Scrapper

Scrape the hotel reviews of a whole city on TripAdvisor.

Requirements

python 3.5

Installation & Setup

Download and install required libs and data:

pip install bs4

Usage Scrapper

Store all reviews of New York City:

python tripadvisor-scrapper.py 60763 New_York_City_New_York

Store all reviews of Paris:

python tripadvisor-scrapper.py 187147 Paris_Ile_de_France

Store all reviews of Vienna:

python tripadvisor-scrapper.py 190454 Vienna

The scrapper requires the city location id and the city name as commandline arguments. Both can be retrieved from the url, for example, https://www.tripadvisor.com/Hotels-g60763-New_York_City_New_York-Hotels.html The city location id is the number after the g. The city name is the string from the dash after the city location id to the dash before Hotels.

Store all reviews of Vienna and additionally store the review urls list as pickle for rescraping later:

python tripadvisor-scrapper.py 190454 vienna --pickle store

A pickle is stored in data/timestamp-cityname

Store all reviews of Vienna using a review urls list loaded from pickle/20160601-1522-vienna.pickle:

python tripadvisor-scrapper.py 190454 Vienna --pickle load --filename 20160601-1522-vienna.pickle

A pickle to load has to be placed in the pickle directory at the same directory level as the tripadvisor-scrapper.py

Usage Totalizer

Put all reviews and hotel information of a city together:

python tripadvisor-totalizer.py /Users/admin/tripadvisor-scrapper/data/20160716-202314-vienna

Author

Michael Andorfer

tripadvisor-scraper's People

Contributors

Stargazers

Watchers

Forkers

mangelaccio hoidu benbbe steph-ga daniellatti hako411 wskwon ateefa prayoginotonegoro hustlijian lzlloveys erin-peace forkclone

tripadvisor-scraper's Issues

Script gets all review_urls and then freezes

It looks like the log works until it gets to the end of the review URL links, then it freezes up. I am wondering if others might have this problem as well. Thanks.

Script get all hotels on a page

Hello;

i have this url example https://www.tripadvisor.com/Hotels-g734462-Tlemcen_Tlemcen_Province-Hotels.html

In that province i found 5 hotels only, all the ones after are hotels in neighbour towns and province.

Script should be fixed to take only the current town/province hotels.

Regards

Getting an Indexerror on all scrapes

Getting this error for all of my scrapes.

Traceback (most recent call last):
  File "tripadvisor-scrapper.py", line 716, in <module>
    city_hotel_urls = parse_hotel_urls_of_city(BASE_URL, city_pagination_urls, headers)
  File "tripadvisor-scrapper.py", line 113, in parse_hotel_urls_of_city
    hotel_urls.append(base_url + soup.find_all('a', attrs={'class': 'property_title '})[j]['href'][1:])
IndexError: list index out of range

There are no csv file in Data folder or the process keep running forever

There is no csv file in the Data folder or the process keep running forever. Do you have any ideas about what is happening.

error on scrapping

Followed the step, but get an error like this:
: error: the following arguments are required: 187147, Paris_Ile_de_France

any idea why does this happen?

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.