The spider project ported to Python.

1. pip install spider_rs

import asyncio

from spider_rs import Website

async def main():
    website = Website("https://choosealicense.com")
    website.crawl()
    print(website.get_links())

asyncio.run(main())

View the examples to learn more.

Install maturin pipx install maturin and python.

1. maturin develop

View the benchmarks to see a breakdown between libs and platforms.

Test url: https://espn.com

The benches above were ran on a mac m1, spider on linux arm machines performs about 2-10x faster.

Please submit a Github issue for any issues found.