jonluca / anubis Goto Github PK
View Code? Open in Web Editor NEWSubdomain enumeration and information gathering tool
Home Page: https://jonlu.ca/anubis/
License: MIT License
Subdomain enumeration and information gathering tool
Home Page: https://jonlu.ca/anubis/
License: MIT License
Make sure you include the following:
Also, please assign @jonluca to all issues at first.
Please replace this line with a brief summary of your issue.
Please note by far the quickest way to get a new feature is to make a Pull Request.
Otherwise, open an issue and tag it with "feature-request"
==============================================================
I'm trying to install into docker alpine image.
I tried python versions 3.6.12, 3.7.9, 3.8.5
installation command tried python -m pip install --no-cache-dir anubis-netsec
also tried to build from source with pip3 install .
ERROR: Could not find a version that satisfies the requirement sslyze==3.0.7 (from anubis-netsec) (from versions: 0.13.0, 0.13.1, 0.13.2, 0.13.3, 0.13.4, 0.13.5, ERROR: No matching distribution found for sslyze==3.0.7 (from anubis-netsec)
Make sure you include the following:
The default scan for anubis includes searching sublist3r
but gives an error: Exception when searching sublist3r
. Looking at the code:
results = requests.get('https://api.sublist3r.com/search.php', headers=headers, params=params)
It's calling to api.sublist3r.com
, but the sublist3r.com
domain name is now owned by some random domain parking company (pinging it resolves to lb01.parklogic.com
).
I would suggest removing sublist3r as a resolution tool in Anubis.
P.S. Fantastic tool! Thanks so much!
In readme.md it reads:
API Keys
If you wish to use the --aditional-info flag, you'll need to set up an API key for https://shodan.io.
Copy API_SAMPLE.py to APY.py,
You might want to change the last line to:
Copy API_SAMPLE.py to API.py,
mind the 'I' instead of the 'Y'.
Companion issue to jonluca/Anubis-DB#1
@jonluca
I would like to suggest reading domains from file to make it easier to process multiple domains at once.
Sadly i cant code to do it on my own to suggest it via PR.
I've never gotten this to install properly. Has there been any thought into deploying this as a Snap or Flatpak? I would hope a universal installer would eliminate the particulars of dependency problems and steps involved to set up in a mixed python environment. I'm one of those casual Linux users that gets lost once steps get too far in the weeds with pip installing things
Hello
Currently, there is no way to get only subdomains and not logo and debug messages.
So add --silent or just pipe debug to stderr, so you can do ’anubis -t substack.com 2>/dev/null’
When cloning the repo on a small bandwith connection I wondered about the relative transfer size vs. the working tree size. When preparing the snap in #22 you must've accidentally added to and then later removed the package blob from the repository. I think there is an argument for a quick clone, even without --depth 1
. Removing the snap and a python binary reduces the size from 19M to 3M, and there are other artifacts too.
I can recommend the oneliner at stackoverflow.com#42544963 to list the biggest blobs and than the java util bfg
to remove by size or path. It's an easier frontend than git filter-branch
.
Disadvantage of the removal is the rewrite of commit hashes since the introduction of each blob. bfg
will output a graph for this. People having forked from one of these commits would need to hard reset their forks and rebase. In my opinion this is not an issue with a smaller codebase where most people will prepare a PR from the latest origin/HEAD.
git rev-list --objects --all \
| git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' \
| sed -n 's/^blob //p' \
| sort --numeric-sort --key=2 \
| cut -c 1-12,41- \
| $(command -v gnumfmt || echo numfmt) --field=2 --to=iec-i --suffix=B --padding=7 --round=nearest \
| grep -vF --file=<(git ls-tree -r HEAD | awk '{print $3}') \
| tac | head -n20
827f720a141a 15MiB anubis_0.8.3+git1.cbabacf_amd64.snap
5d2ef13a6db4 4,3MiB prime/usr/bin/python3.5
cbbc03be80b6 1,4MiB anubis/common_subdomains.txt
6fc2f1710815 117KiB prime/usr/share/doc/python3.5/NEWS.gz
java -jar bfg.jar --delete-files 'anubis_0.8.3+git1.cbabacf_amd64.snap'
java -jar bfg.jar --delete-files 'python3.5'
Earliest Latest
| |
...................................................Dmmmmmmmm
D = dirty commits (file tree fixed)
m = modified commits (commit message or parents changed)
. = clean commits (no changes to file tree)
Down at 3344K.
optional:
java -jar bfg.jar --delete-files 'common_subdomains.txt'
java -jar bfg.jar --delete-folders 'prime'
java -jar bfg.jar --delete-folders 'stage'
java -jar bfg.jar --delete-files 'temp.txt'
after those deletes:
git reflog expire --expire=now --all && git gc --prune=now --aggressive
du -cs .
With all deletes, the repo is down at 540K.
Installing collected packages: anubis-netsec
Successfully installed anubis-netsec-0.2.0
~# anubis
Traceback (most recent call last):
File "/usr/local/bin/anubis", line 11, in
sys.exit(main())
File "/usr/local/lib/python3.5/dist-packages/anubis/cli.py", line 78, in main
import anubis.commands
File "/usr/local/lib/python3.5/dist-packages/anubis/commands/init.py", line 1, in
from .target import *
File "/usr/local/lib/python3.5/dist-packages/anubis/commands/target.py", line 26, in
from anubis.utils.ColorPrint import ColorPrint
ImportError: No module named 'anubis.utils'
@jonluca
not quite sure how to do a stack trace
pip3 install anubis-netsec
... downloads all the files...
Successfully built censys SSLyze docopt python-nmap shodan tls-parser click-plugins pycparser
Installing collected packages: certifi, urllib3, chardet, idna, requests, netaddr, censys, dnspython, setuptools, nassl, asn1crypto, six, pycparser, cffi, cryptography, tls-parser, SSLyze, docopt, python-nmap, click, click-plugins, colorama, XlsxWriter, shodan, anubis-netsec
Successfully installed SSLyze-1.3.4 XlsxWriter-1.0.2 anubis-netsec-0.7.6 asn1crypto-0.24.0 censys-0.0.8 certifi-2018.1.18 cffi-1.11.4 chardet-3.0.4 click-6.7 click-plugins-1.0.3 colorama-0.3.9 cryptography-2.1.4 dnspython-1.15.0 docopt-0.6.2 idna-2.6 nassl-1.0.3 netaddr-0.7.19 pycparser-2.18 python-nmap-0.6.1 requests-2.18.4 setuptools-38.5.1 shodan-1.7.7 six-1.11.0 tls-parser-1.2.1 urllib3-1.22
Segmentation fault (core dumped)
kern.log
Feb 24 17:02:08 kernel: [ 311.672024] random: crng init done
Feb 24 17:02:14 kernel: [ 317.955766] pip3[1638]: segfault at a9 ip 0000000000442f24 sp 00007ffda779e648 error 4 in python3.6[400000+3bf000]
Feb 24 17:05:22 kernel: [ 505.137210] pip3[1932]: segfault at 22e728 ip 00000000004dc698 sp 00007ffca67b2a00 error 4 in python3.6[400000+3bf000]
Feb 24 17:17:56 kernel: [ 1259.481690] pip3[11814]: segfault at 22e728 ip 00000000004dc698 sp 00007ffe9eed6350 error 4 in python3.6[400000+3bf000]
syslog
Feb 24 17:17:56 kernel: [ 1259.481690] pip3[11814]: segfault at 22e728 ip 00000000004dc698 sp 00007ffe9eed6350 error 4 in python3.6[400000+3bf000]
Some subdomains in the result contain *.
:
vpn.awscorp.uberinternal.com
corpmdm.uberinternal.com
*.uberinternal.com
vpn-china-szx1.uberinternal.com
--- snippet ---
jira-cdn-staging.uberinternal.com
*.tm1ext.awscorp.uberinternal.com
uberhubtest.uberinternal.com
gw08-usw1.awscorp.uberinternal.com
--- snippet ---
lsfosq_us1.uberinternal.com
*.it-tools.uberinternal.com
gw08-sae1.awscorp.uberinternal.com
There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.
Error type: undefined. Note: this is a nested preset so please contact the preset author if you are unable to fix it yourself.
For a sneak peek, check out the recursive_search_on_subdomain
branch
I've done a substantial reorganization of the code such that all the functions are now modules in anubis.scanners. This'll make it trivial to add new ones, and will prevent bloat in target.py.
The feature will be merged into master in the next few days.
@jonluca
Anubis version: none (cloned repo today)
Python version: 'Python 3.6.5rc1'
Machine Details:
00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v6/7th Gen Core Processor Host Bridge/DRAM Registers (rev 02)
00:02.0 VGA compatible controller: Intel Corporation HD Graphics 620 (rev 02)
00:04.0 Signal processing controller: Intel Corporation Skylake Processor Thermal Subsystem (rev 02)
00:14.0 USB controller: Intel Corporation Sunrise Point-LP USB 3.0 xHCI Controller (rev 21)
00:14.2 Signal processing controller: Intel Corporation Sunrise Point-LP Thermal subsystem (rev 21)
00:15.0 Signal processing controller: Intel Corporation Sunrise Point-LP Serial IO I2C Controller #0 (rev 21)
00:15.1 Signal processing controller: Intel Corporation Sunrise Point-LP Serial IO I2C Controller #1 (rev 21)
00:16.0 Communication controller: Intel Corporation Sunrise Point-LP CSME HECI #1 (rev 21)
00:17.0 SATA controller: Intel Corporation Sunrise Point-LP SATA Controller [AHCI mode] (rev 21)
00:1c.0 PCI bridge: Intel Corporation Sunrise Point-LP PCI Express Root Port (rev f1)
00:1c.4 PCI bridge: Intel Corporation Sunrise Point-LP PCI Express Root Port #5 (rev f1)
00:1c.5 PCI bridge: Intel Corporation Sunrise Point-LP PCI Express Root Port #6 (rev f1)
00:1f.0 ISA bridge: Intel Corporation Sunrise Point-LP LPC Controller (rev 21)
00:1f.2 Memory controller: Intel Corporation Sunrise Point-LP PMC (rev 21)
00:1f.3 Audio device: Intel Corporation Sunrise Point-LP HD Audio (rev 21)
00:1f.4 SMBus: Intel Corporation Sunrise Point-LP SMBus (rev 21)
01:00.0 3D controller: NVIDIA Corporation GM108M [GeForce 940MX] (rev a2)
02:00.0 Network controller: Qualcomm Atheros QCA6174 802.11ac Wireless Network Adapter (rev 32)
03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)
O.S.: Linux parrot 4.16.0-parrot5-amd64 #1 SMP Parrot 4.16.5-1parrot5 (2018-05-03) x86_64 GNU/Linux
Full Stack Trace:
https://pastebin.com/gzdyuaVn
Cryptography requirements:
sudo apt-get install build-essential libssl-dev libffi-dev python3-dev
build-essential is already the newest version (12.4).
libffi-dev is already the newest version (3.2.1-8).
libssl-dev is already the newest version (1.1.0h-2).
python3-dev is already the newest version (3.6.4-1).
Not able to get Shodan result because of the installation error.
Command "/usr/bin/python3.4 -u -c "import setuptools, tokenize;file='/tmp/pip-build-t8955aq8/cryptography/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-i5krkv2s-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-t8955aq8/cryptography/
readme.md reads:
pip install -r requirements
While the filename is requirements.txt
Suggestion: since Anubis is running using Python 3.6 and I'm running both Python2 and Python3, I had to specify pip3
in order to get the requirements installed. (macos 10.13.2)
┌──(kali㉿Kali)-[~/Anubis]
└─$ sudo apt-get install python3-pip python-dev libssl-dev libffi-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
Package python-dev is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
However the following packages replace it:
python2-dev python2 python-dev-is-python3
E: Package 'python-dev' has no installation candidate
I want to know the name of a man using fake names
This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.
These updates have all been created already. Click a checkbox below to force a retry/rebase of any.
Dockerfile
python 3.10-slim
requirements.txt
python_nmap ==0.7.1
shodan ==1.26.0
docopt ==0.6.2
requests ==2.27.1
censys ==2.1.1
dnspython ==2.1.0
Please note by far the quickest way to get a new feature is to make a Pull Request.
Otherwise, open an issue and tag it with "feature-request"
We are suggesting you add Spyse as one of the data sources for: DNS, subdomain enumeration, retrieving web information, hosts, certificates, and ports.
Potential is immense. Contact us if you need any help or a free subscription to make it free for users.
Alternatives could be shodan or censys.
I'm offering Spyse because it has different scanning and analyzing systems that provide a more clear and sometimes larger amounts of data.
Notice: I work for Spyse.
A future project might be to recreate a scan on all the IPs of the subdomains, or pass them off to nessus.
WARNING: The script anubis is installed in '/home/kali/.local/bin' which is not on PATH.
this is showing ............please tell me what to do next......
Evidence:
" git clone [email protected]:jonluca/Anubis.git
Cloning into 'Anubis'...
[email protected]: Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists."
Anubis (0.1.0, /opt/information-gathering/subdomains/Anubis)
asn1crypto (0.23.0)
attrs (17.3.0)
backports-abc (0.5)
censys (0.0.8)
certifi (2017.11.5)
cffi (1.11.2)
chardet (3.0.4)
click (6.7)
click-plugins (1.0.3)
colorama (0.3.9)
configparser (3.5.0)
coverage (4.4.2)
cryptography (1.9)
Django (1.11.7)
dnspython (1.15.0)
docopt (0.6.2)
enum34 (1.1.6)
funcsigs (1.0.2)
gevent (1.2.2)
greenlet (0.4.12)
idna (2.6)
ipaddress (1.0.18)
lxml (4.1.1)
nassl (1.0.1)
netaddr (0.7.19)
parse (1.8.2)
pdfkit (0.6.1)
pip (9.0.1)
pluggy (0.6.0)
py (1.5.2)
pyasn1 (0.4.2)
pycparser (2.18)
pyOpenSSL (17.5.0)
pytest (3.3.1)
pytest-cov (2.5.1)
python-nmap (0.6.1)
pytz (2017.3)
requests (2.10.0)
rsa (3.4.2)
setuptools (36.5.0)
shodan (1.7.7)
singledispatch (3.4.0.3)
six (1.11.0)
SSLyze (1.2.0)
tls-parser (1.1.1)
tornado (4.5.2)
typing (3.6.2)
urllib3 (1.22)
wheel (0.30.0)
XlsxWriter (1.0.2)
Make sure you include the following:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/requests/packages/urllib3/connection.py", line 142, in _new_conn
(self.host, self.port), self.timeout, **extra_kw)
File "/usr/local/lib/python3.6/site-packages/requests/packages/urllib3/util/connection.py", line 91, in create_connection
raise err
File "/usr/local/lib/python3.6/site-packages/requests/packages/urllib3/util/connection.py", line 81, in create_connection
sock.connect(sa)
TimeoutError: [Errno 60] Operation timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 578, in urlopen
chunked=chunked)
File "/usr/local/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 351, in _make_request
self._validate_conn(conn)
File "/usr/local/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 814, in _validate_conn
conn.connect()
File "/usr/local/lib/python3.6/site-packages/requests/packages/urllib3/connection.py", line 254, in connect
conn = self._new_conn()
File "/usr/local/lib/python3.6/site-packages/requests/packages/urllib3/connection.py", line 151, in _new_conn
self, "Failed to establish a new connection: %s" % e)
requests.packages.urllib3.exceptions.NewConnectionError: <requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x10f246828>: Failed to establish a new connection: [Errno 60] Operation timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 403, in send
timeout=timeout
File "/usr/local/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 623, in urlopen
_stacktrace=sys.exc_info()[2])
File "/usr/local/lib/python3.6/site-packages/requests/packages/urllib3/util/retry.py", line 281, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
requests.packages.urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='www.pkey.in', port=443): Max retries exceeded with url: /tools-i/search-subdomains (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x10f246828>: Failed to establish a new connection: [Errno 60] Operation timed out',))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/bin/anubis", line 11, in <module>
sys.exit(main())
File "/usr/local/lib/python3.6/site-packages/anubis/cli.py", line 115, in main
command.run()
File "/usr/local/lib/python3.6/site-packages/anubis/commands/target.py", line 76, in run
Thread(target=search_pkey(self, self.options["TARGET"][i])),
File "/usr/local/lib/python3.6/site-packages/anubis/scanners/pkey.py", line 18, in search_pkey
headers=headers, data=data, verify=False)
File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 111, in post
return request('post', url, data=data, json=json, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 57, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 475, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 585, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 467, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='www.pkey.in', port=443): Max retries exceeded with url: /tools-i/search-subdomains (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x10f246828>: Failed to establish a new connection: [Errno 60] Operation timed out',))
macOS 10.13.2
It looks like https://pkey.in times out which causes urllib3 to get a timeout exception. Once this exception occurs the scan times out and quits. I suggest removing pkey or creating an easy way to disable that specific searcher through a command line flag. Additionally there may need to be additional error handling to recover from these issues.
Note: pkey.in is also timing out in my browser.
Make sure you include the following:
Error connecting to target! Make sure you spelled it correctly and it is a resolvable address Traceback (most recent call last): File "C:\Python39\lib\runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Python39\lib\runpy.py", line 87, in _run_code exec(code, run_globals) File "C:\Python39\Scripts\anubis.exe\__main__.py", line 7, in <module> File "C:\Python39\lib\site-packages\anubis\cli.py", line 127, in main command.run() File "C:\Python39\lib\site-packages\anubis\commands\target.py", line 68, in run self.init() File "C:\Python39\lib\site-packages\anubis\commands\target.py", line 64, in init raise e File "C:\Python39\lib\site-packages\anubis\commands\target.py", line 59, in init domain_str = socket.gethostbyname(host) socket.gaierror: [Errno 11001] getaddrinfo failed
Tried to enumerate a list of domains with Anubis in Powershell.
Tried both:
example.com
https://example.com
To confirm the issue I asked a friend and he gets the same error. So Im assuming its code or format of the list related not a missing dependency.
Consider adding Hudson Rock's complimentary data to receive additional URLs associated with a specific domain.
Data was retrieved from infected computers with the Infostealer malware and contains URLs that usually cannot be found conventionally.
Domain sample: https://cavalier.hudsonrock.com/api/json/v2/osint-tools/search-by-domain?domain=tesla.com
Thank you.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.