chipbuster / link-juicer Goto Github PK

View Code? Open in Web Editor NEW

This project forked from stevemclaugh/linkjuicer

0.0 0.0 1.0 28 KB

A script for squeezing as many links as possible from dynamic web pages

License: MIT License

Python 100.00%

link-juicer's People

Contributors

Watchers

Forkers

chioranionutcatalin

link-juicer's Issues

Two competing processes will clobber the log and output files

If you run two instances of the link juicer concurrently with the same log/output file, they will clobber each other and leave you with garbage.

This needs to be fixed somehow. Simplest solution is a lock so that only one juicer can be running at a time. More advanced solutions would only prohibit access to the same file (so that multiple juicers can run at once as long as they have independent output files)

Script needs to accept arguments for command-line driving

Write yerself an argparse

Chrome scraping broken

Chrome scraping misses several fairly obvious links on the test site. Why?

Performance issues with Phantom and Gecko

Time to scrape default URL:

Browser	Time to scrape
Firefox	34:53
PhantomJS	42:xx
Chrome	16:35

(I closed the window by accident before getting the seconds reading for PhantomJS)

Scraping appears to be a pretty CPU-intensive activity. Is this a problem with the driver, a problem with the browser, or just a side effect of what we're doing?

Recommend Projects