Produces a site map that contains the publicly reachable links and assets for all pages within the specified domain.

A URL is considered to be in the specified domain if the protocol and host match exactly. For example, https://foo.com and https://foo.com/docs are considered to be in the same domain, however http://foo.com and https://bar.com are not.

At each step of the web crawl, we retrieve the HTML content for the page and parse it for all links and assets. These individual page maps are compiled together to create the final site map. Note that while all links and assets are included in a page map, only links that belong to the specified domain are crawled and thus produce their own page map.

Two methods are provided to create a site map for a particular domain, which are detailed below.

A CLI is provided to create a site map easily from the command line. Assuming this package has been installed via go install

cli --site INITIAL_URL --workers NUM_WORKERS --file OUTPUT_FILE

As an example, to produce a site map for foo.com using 100 workers and save it to sitemap.json

cli --site https://foo.com --workers 100 --file sitemap.json

A REST API has also been provided. Assuming this package has been installed via go install

api --port PORT

As an example, to run the server on port 8000

api --port 8000

With the server running, site maps are created via a simple GET request

GET http://localhost:8000/sitemap?site=INITIAL_URL&workers=NUM_WORKERS

As an example, to produce a site map for foo.com using 100 workers

GET http://localhost:8000/sitemap?site=https://foo.com&workers=100

When running the API server, additionally specify the path to the static gui directory of this repository. For example

api --port 8000 --static sitemapper/gui

With the API server running, visit http://localhost:8000 in your preferred browser. After filling out the form to specify the initial URL and number of workers, the site map will be displayed as an interactive graph.

Dark blue nodes represent web pages that belong to the domain, while gray nodes represent pages outside the domain. Light blue nodes are assets, such as images and javascript libraries. Hovering over a node displays the URL for that node.