This is a simple crawler using Rust. It fetches pages and extracts any http(s) links found in a href
elements.
Rust and Cargo
To run the crawler provide the URL to crawl
cargo run -- --url=https://example.com
It also accepts a optional timeout (default 1000 ms) to wait for any new pages
cargo run -- --url=https://example.com --timeout=800
To execute the tests
cargo test
If you want to see STDOUT during the test run
cargo test -- --nocapture