tr1ppyb0y / web-crawler Goto Github PK
View Code? Open in Web Editor NEWThis repository is set of files, which takes URL as input and ask for if you want to scrap website from base URL or from the current path you provided. It uses hash function to store links in database, since it's for indexing, i've used weak hash function.