Project is a crawler for given url setup in src/index.ts file. Project is using node 8.x.x and Typescript and write results into standard output.

1. Install yarn and node on you local machine

You can install Yarn through the Homebrew package manager. This will also install Node.js if it is not already installed.

brew install yarn

You can install Yarn through MacPorts. This will also install Node.js if it is not already installed.

sudo port install yarn

For other system installation follow instruction on Yarn official page

Clone the repo and install Node dependencies

git clone https://github.com/marcelijanowski/web-crawler.git
yarn install

To run project on local machine run command:

yarn start

└── src                         # app source code
    └── index.ts                # app entry module with routing config

Test are written in mocha and chai. To run type in command line :

• Run yarn jest

To test is written code is following standard we use tslint and rules are in tslint.json file To run linter :

• Run yarn lint

Solution is slow and taking around 50 seconds for https://wiprodigital.com/ but suggestion at bottom . Sometime even not getting though all url as script is block by server and then we catching error and returing process results (not full unfortunatly).

1. Solution taking into consideration that many supages have same link so we need to keep track of already visited page This is done in line 19 & 20 inside index.ts file

const domainInsternalLinks = new Set(internalLinks(links, hostname));
const difference = differenceBetweenListOfUrls(domainInsternalLinks, allLinks);

2. For some domain like wipro script will hang for some time as cause page is trying script as spammer and blocking calls giving first 429 and then blocking connection completly. One solution will be doing call in random sequence of time but this will slow script.

3. Also at moment solution not taking into consideration links that are relative but this might be sorted with adding addional RegExp

4. This isn't like idea solution which need more work as using recursive function without looking on limitation of webpage open request. Ideal solution would be creating a queing system which will process link in concurent connections and react on 429 error code and retry to get data.

5. If we talking about cloud solution would be nice to use AWS Lambda or *GCP functions to process link by link and save it into S3 bucket or GCP Storage

6. Also would be good to Dockerize solution and run e2e on docker - unfortunatly lack of time for now.