geshan / domain-scraper Goto Github PK

A very simple domain.com.au rental listing scraper to compare listings as there is no compare option in domain. So this script pulls in X number of listing that helps you compare domain's rental listing as coming to a new city it is a very handy thing to have. This will be very helpful to people trying to move and even more helpful to people who just landed in Australian cities.

Very irresponsible and hacky software ahead that works and does the job :). It might break randomly.

It is only using Axios, Cheerio and Json2Csv for now.

Each domain rental listing has a unique Id like https://www.domain.com.au/12500140 redirects to

https://www.domain.com.au/27a-arcadia-road-chester-hill-nsw-2162-12500140 but the id 12500140 stays the same. So this script uses this unique id to visit that page and scrape 10 fields out of it then save it as a csv which is a lot easy to compare than visiting 10-20 pages. For an example I took 5 listing/IDs randomly. You can do it following the steps below:

• Add all your liked listing on domain.com.au in your shortlist. Yes you need to register and login.
• Then run this super small script on each of your shortlist page to get the IDs you want to compare. (only tested on chrome)

• Copy/replace the IDs to line no. 22-27, basically fill up the listingIds array correctly.
• Then run the script as shown below to get the csv of your listing Ids.

node index.js gives out domain-rentals.csv in the same directory with some fields for comparision like rent, no. of bedrooms, no. of bathrooms, no. of parking, available from etc. Hopefully you have a recent verison of node installed locally or try docker.

Just do docker-compose up. On each save it will run the scirpt as it is running with nodemon.

Would have been good to have them, it is just a useful hack.

It will not run for buying listings and it will have wierd characters here and there. Don't like this, send a PR! ;)

Yes of course, there are lots of things to improve. Any contribution/PR is really welcome.

• add docker and docker-compose
• add tests may be :D