atomicptr / crab Goto Github PK

A few months ago the snap part of the pipeline broke and I haven't been able to get it working again since then so I decided to remove snap support for now (might still push updates manually we'll see...).

Since I don't use Ubuntu nor Snap my interest in this is also kinda low anyway. If someone wants to help fix this I'm open to it.

When running crab crawl:sitemap https://www.domain.com/sitemap.xml --header user-agent=some-whitelisted-user-agent the request for the sitemap doesn't use the user agent header.

$ crab crawl:sitemap https://domain.com/sitemap.xml --output output.log
# NO prints here...

It might probably make sense to have a different option for persisting the logs as a valid json file...

$ crab crawl:sitemap https://domain.com/sitemap.xml --output-json output.json
$ cat output.json
[
    {"status": 200, "url": "https://domain.com/page-1", ...},
    ...
]

It should be possible to filter crawl logs, depending on what I'm doing I might just want to look for non 200 pages

An idea on how this could look like:

$ crawl crawl:sitemap https://domain.com/sitemap.xml --filter-status=!200
# Only lists results that are not 200...

Preferably as some sort of integration with the vegeta load testing tool.