diegoevangelisti / countdown-nwm-scraping Goto Github PK

This app performs web scraping to collect information related with products from two big New Zealand supermarkets. Uses cheerio.js to perform DOM manipulation on the server side and get data about their categories, product names, prices, offers, etc. The database in NoSQL (MongoDB), the main server side programming language is Node.js. Also a little bit of EJS (template enginee) and the classic (bad still alive) HTML+CSS+JavaScript+jQuery approach for the Admin Panel (FrontEnd). Last, but not least, the RESTful API that feed the main FrontEnd (a beatiful iOS app that is not hosted here) is deplayed in Heroku. Enjoy!

Download their latest versions according to your OS version:

• https://nodejs.org/en/download/
• https://www.mongodb.com/download-center/community
• https://robomongo.org/download

Then open the exe files, install the applications, and follow the instructions

In the terminal run:

git clone https://github.com/diegoevangelisti/countdown-nwm-scraping

Run the following statement:

cd countdown-nwm-scraping

npm init

npm install --save

Open another terminal and go to the folder where Mongodb server is installed - in windows: C:\Program Files\MongoDB\Server\4.0\bin

Then run: ./mongod

After that the server should be properly running and we can connect our app to mongodb

Open the Robo3T application nd create a new database named SSA and then connect to it

In the terminal type:

node index.js

Open the browser and go to: http://localhost:5000/

Then enter the following credentials:

username: backend password: backend

After login the app won't show data in Countdown and NWM because there is no data in the database

Go to Options -> Get Data (Countown and New World)

That data can be visualized in the back office going to: Options -> COUNTDOWN DATA and NEW WORLD DATA Can be checked in Robo3T as well.

Now the application is configured to perform web scraping in Countdown and New World shops. Setup: Takes all the categories, and scrapes 5 pages of 24 and 20 products from each supermarket respectively.