A passive reconnaissance tool for known URLs discovery - it gathers a list of URLs passively using various online sources.

• Features
• Installation
  • From Binary
  • From source
  • From github
• Post Installation
• Usage
  • Examples
    • Basic
    • Regex filter URLs
    • Include Subdomains' URLs
• Contribution

• Collect known URLs:
  • Fetches from AlienVault's OTX, Common Crawl, URLScan, Github, Intelligence X and the Wayback Machine.
  • Fetches disallowed paths from robots.txt found on your target domain and snapshotted by the Wayback Machine.
• Reduce noise:
  • Regex filter URLs.
  • Removes duplicate pages in the sense of URL patterns that are probably repetitive and points to the same web template.
• Output to stdout for piping or save to file.

You can download the pre-built binary for your platform from this repository's releases page, extract, then move it to your $PATHand you're ready to go.

sigurlfind3r requires go1.17+ to install successfully. Run the following command to get the repo

go install -v github.com/signedsecurity/sigurlfind3r/cmd/sigurlfind3r@latest

git clone https://github.com/signedsecurity/sigurlfind3r.git && \
cd sigurlfind3r/cmd/sigurlfind3r/ && \
go build; mv sigurlfind3r /usr/local/bin/ && \
sigurlfind3r -h

sigurlfind3r will work after installation. However, to configure sigurlfind3r to work with certain services - currently github - you will need to have setup API keys. The API keys are stored in the $HOME/.config/sigurlfind3r/conf.yaml file - created upon first run - and uses the YAML format. Multiple API keys can be specified for each of these services.

Example:

version: 1.8.0
sources:
    - commoncrawl
    - github
    - intelx
    - otx
    - urlscan
    - wayback
    - waybackrobots
keys:
    github:
        - d23a554bbc1aabb208c9acfbd2dd41ce7fc9db39
        - asdsd54bbc1aabb208c9acfbd2dd41ce7fc9db39
    intelx:
        - 2.intelx.io:00000000-0000-0000-0000-000000000000

DiSCLAIMER: fetching urls from github is a bit slow.

sigurlfind3r -h

This will display help for the tool.

     _                  _  __ _           _ _____
 ___(_) __ _ _   _ _ __| |/ _(_)_ __   __| |___ / _ __
/ __| |/ _` | | | | '__| | |_| | '_ \ / _` | |_ \| '__|
\__ \ | (_| | |_| | |  | |  _| | | | | (_| |___) | |
|___/_|\__, |\__,_|_|  |_|_| |_|_| |_|\__,_|____/|_| 1.8.0
       |___/

USAGE:
  sigurlfind3r [OPTIONS]

OPTIONS:
   -d, --domain            domain to fetch urls for
  -eS, --exclude-sources   comma(,) separated list of sources to exclude
   -f, --filter            URL filtering regex
  -iS, --include-subs      include subdomains' urls
  -lS, --list-sources      list all the available sources
  -nC, --no-color          no color mode
   -s  --silent            silent mode: output urls only
  -uS, --use-sources       comma(,) separated list of sources to use
   -o, --output            output file

sigurlfind3r -d tesla.com

sigurlfind3r -d tesla.com -f ".(jpg|jpeg|gif|png|ico|css|eot|tif|tiff|ttf|woff|woff2)"

sigurlfind3r -d tesla.com -iS

Issues and Pull Requests are welcome!