This application involves building a simple web crawler which fetches all public links from a given website
- DFS - depth first search
- Set - remove duplicate
- check python3.6 is setup in your computer.
python --version
or
python3 --version
- Clone project
git clone https://github.com/XunPeng715/global-legaltech-corp.git
cd global-legaltech-corp
- Run application
- if you want to fetch all links from website
python spider.py https://www.tutorialspoint.com/index.htm
- if you want to go through up to 20 webpages
python spider.py https://www.tutorialspoint.com/index.htm 20