nsonaniya2010 / subdomainizer Goto Github PK

Describe the bug
When I run python3 SubDomainizer.py -h, I get the following error message:

Traceback (most recent call last):
File "SubDomainizer.py", line 13, in
import termcolor
ImportError: No module named 'termcolor'

I have verified that the module 'termcolor' is installed

This is a really annoying bug, after I gather subdomains and put the URLS in a list to scan them, subdomainizer completely freezes at specific urls for whatever reasons despite being alive hosts, I delete the URL from the txt file that's causing this and rescan, another URL that causes the tool to completely freeze and I had to remove that too..etc until the output finally shows when there are no more URLs causing a permanent freeze

other than that the tool is an exception and is very useful despite simplicity, thank you very much for publicly posting the project

Hi ,

Thanks for the great tool .

• Given the tool fetches many js files , regex prints the secrets on hits but does not print the URL where it was found . Printing js url wouldn't be a good enhancement ? it can save extra effort back-tracing the url .

• It is slow even when given list of 10-20 domains . Can multiprocessing be applied here to make it faster ?

Thanks

Would be great if when reading from the subdomain list http or https is pre-appended, so it's easier to import from other tools.

Will be happy to send a PR later if needed.

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

y run the tool sudo python3 SubDomainizer.py -u target.com

I get

Traceback (most recent call last):
File "/home/bug/Desktop/tool2/SubDomainizer/SubDomainizer.py", line 13, in
import termcolor
ModuleNotFoundError: No module named 'termcolor'

the module is already installed in my linux mint.

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Smartphone (please complete the following information):

• Device: [e.g. iPhone6]
• OS: [e.g. iOS8.1]
• Browser [e.g. stock browser, safari]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

###My Result:

`Got Some IPv4 addresses:

2.343.543.543
1.031.844.254
28.22.5.5
7.2.2.5
22.5.5.5
05.7.18.2
5.24.7.06
2.5.5.5
02.02.04.04
24.33.02.04
2.343.516.516
16.53.05.7
394.394.909.59
2.2.5.2
45.01.81.39
3.996.89.89
382.167.814.178
2.18.52.15
2.343.544.544
5.22.5.5
`

Hi,

Firstly, a great tool, so glad I found it!

I have noticed that savesecretsresults() isn't begin called in the SubDomainizer.py. As a result, nothing is getting saved to -sop --SECRETOP.

I don't like to raise an issue without giving a possible solution but unfortunately, my coding isn't up to scratch.

Option to have a timeout, some websites keep parsing for 30 min.. Timeout option would be a solution to skip such sites

i use mitm for check requests, so sometimes script make request with joined strings.
i try using sript for github.com , and i see this:

172.18.0.11:37246: GET https://github.com/
200 OK 39.23k
172.18.0.11:37250: GET https://github.com/
200 OK 39.25k
GET https://github.githubassets.com/assets/compat-bootstrap-edde9aec.jshttps://github.githubassets.com/assets/compat-bootstrap-edde9aec.js
404 Not Found 30b
GET https://github.githubassets.com/assets/github-bootstrap-e9911286.js
200 OK 122.5k
GET https://github.githubassets.com/assets/compat-bootstrap-edde9aec.js
200 OK 9.42k
GET https://github.githubassets.com/assets/frameworks-564f5a6e.jshttps://github.githubassets.com/assets/frameworks-564f5a6e.js
404 Not Found 30b
GET https://github.githubassets.com/assets/frameworks-564f5a6e.js
200 OK 48.33k
GET https://github.githubassets.com/assets/github-bootstrap-e9911286.jshttps://github.githubassets.com/assets/github-bootstrap-e9911286.js
404 Not Found 30b

So few request contains of 2 urls

I'm running linux Mint tina

uname:
Linux d0urd3n 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

This is the error I'm getting:

Traceback (most recent call last):
File "/home/n0w0nd3r/tools/SubDomainizer/SubDomainizer.py", line 869, in
gitThread.map(getGithubData, contentApiURLs)
File "/usr/lib/python3.6/multiprocessing/pool.py", line 266, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/usr/lib/python3.6/multiprocessing/pool.py", line 644, in get
raise self._value
File "/usr/lib/python3.6/multiprocessing/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "/usr/lib/python3.6/multiprocessing/pool.py", line 44, in mapstar
return list(map(*args))
File "/home/n0w0nd3r/tools/SubDomainizer/SubDomainizer.py", line 617, in getGithubData
_data = base64.b64decode(jsonData['content'])
KeyError: 'content'

Is your feature request related to a problem? Please describe.
I know what the secrets are but I need to know where were they found so I can remove the data leak from my server

Describe the solution you'd like
I'd like to see where the items in secretList are found, to make it easier to understand which files/data is their source

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Currently the output is logged into one file which we can specify with the "-o" flag. Ideally, we could use that same flag and two different files would be generated: one with the results from subdomains another with cloud storage results. The names could be pre-appended with the parameter from the output flag.

This would make much easier to parse results from the tool and integration into different workflows.

Cheres

I am getting this error on Raspberry Pi 3. Any ideas on how to fix it?

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Smartphone (please complete the following information):

• Device: [e.g. iPhone6]
• OS: [e.g. iOS8.1]
• Browser [e.g. stock browser, safari]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

Hi @nsonaniya2010
It would be much more useful if you display associated js file from which the secret is exposed.

The following error is thrown:

`Traceback (most recent call last):
File "C:\Python310\lib\site-packages\urllib3\connectionpool.py", line 384, in _make_request
six.raise_from(e, None)
File "", line 2, in raise_from
File "C:\Python310\lib\site-packages\urllib3\connectionpool.py", line 380, in _make_request
httplib_response = conn.getresponse()
File "C:\Python310\lib\http\client.py", line 1374, in getresponse
response.begin()
File "C:\Python310\lib\http\client.py", line 318, in begin
version, status, reason = self._read_status()
File "C:\Python310\lib\http\client.py", line 279, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "C:\Python310\lib\socket.py", line 705, in readinto
return self._sock.recv_into(b)
File "C:\Python310\lib\site-packages\urllib3\contrib\pyopenssl.py", line 312, in recv_into
return self.recv_into(*args, **kwargs)
File "C:\Python310\lib\site-packages\urllib3\contrib\pyopenssl.py", line 310, in recv_into
raise timeout('The read operation timed out')
TimeoutError: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Python310\lib\site-packages\requests\adapters.py", line 439, in send
resp = conn.urlopen(
File "C:\Python310\lib\site-packages\urllib3\connectionpool.py", line 637, in urlopen
retries = retries.increment(method, url, error=e, _pool=self,
File "C:\Python310\lib\site-packages\urllib3\util\retry.py", line 368, in increment
raise six.reraise(type(error), error, _stacktrace)
File "C:\Python310\lib\site-packages\urllib3\packages\six.py", line 686, in reraise
raise value
File "C:\Python310\lib\site-packages\urllib3\connectionpool.py", line 597, in urlopen
httplib_response = self._make_request(conn, method, url,
File "C:\Python310\lib\site-packages\urllib3\connectionpool.py", line 386, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File "C:\Python310\lib\site-packages\urllib3\connectionpool.py", line 306, in _raise_timeout
raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='www.fisglobal.com', port=443): Read timed out. (read timeout=20)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\ricky\Tools\SubDomainizer\SubDomainizer.py", line 785, in
subextractor(compiledRegexCloud, compiledRegexSecretList,
File "C:\Users\ricky\Tools\SubDomainizer\SubDomainizer.py", line 618, in subextractor
jsfile.IntJsExtract(url, heads)
File "C:\Users\ricky\Tools\SubDomainizer\SubDomainizer.py", line 201, in IntJsExtract
req = requests.get(url, headers=heads, verify=False, timeout=(20, 20))
File "C:\Python310\lib\site-packages\requests\api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "C:\Python310\lib\site-packages\requests\api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Python310\lib\site-packages\requests\sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "C:\Python310\lib\site-packages\requests\sessions.py", line 668, in send
history = [resp for resp in gen] if allow_redirects else []
File "C:\Python310\lib\site-packages\requests\sessions.py", line 668, in
history = [resp for resp in gen] if allow_redirects else []
File "C:\Python310\lib\site-packages\requests\sessions.py", line 239, in resolve_redirects
resp = self.send(
File "C:\Python310\lib\site-packages\requests\sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "C:\Python310\lib\site-packages\requests\adapters.py", line 529, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='www.example.com', port=443): Read timeout.`

Please help!

Describe the bug
When testing against an internal site with invalid SSL I get this error:
An error occured while fetching URL, Might be server is down, or domain does not exist, Please check!

Suggested Fix
Add a flag which ignores all ssl errors (-k in this example)

1. Have site with invalid ssl
2. Run python3 SubDomainizer.py -u -k https://mysite.com
3. ???
4. Profit

Hello @nsonaniya2010 ,
Getting this after scanning 10+ subdomains using list function:

Searching for External Javascript links in page..... Successfully got all the external js links SubDomainizer.py:132: DeprecationWarning: invalid escape sequence '\|' content = unquote(requests.get(js).content.decode('unicode-escape'))

Traceback (most recent call last): File "SubDomainizer.py", line 360, in <module> subextractor(compiledRegexCloud, compiledRegexSecretList, compiledRegexDomain,compiledRegexIP, i) File "SubDomainizer.py", line 290, in subextractor threads = ThreadPool(300) File "/usr/lib/python3.7/multiprocessing/dummy/__init__.py", line 124, in Pool return ThreadPool(processes, initializer, initargs) File "/usr/lib/python3.7/multiprocessing/pool.py", line 802, in __init__ Pool.__init__(self, processes, initializer, initargs) File "/usr/lib/python3.7/multiprocessing/pool.py", line 176, in __init__ self._repopulate_pool() File "/usr/lib/python3.7/multiprocessing/pool.py", line 241, in _repopulate_pool w.start() File "/usr/lib/python3.7/multiprocessing/dummy/__init__.py", line 51, in start threading.Thread.start(self) File "/usr/lib/python3.7/threading.py", line 847, in start _start_new_thread(self._bootstrap, ()) RuntimeError: can't start new thread

Hello @nsonaniya2010
SubDomainizer is not parsing amazon bucket from url as the source is like

<a href="https://example.com" target="NEW"><img src="https://s3-us-west-2.amazonaws.com/domain/image.jpg"></a>

Hello @nsonaniya2010 ,
Can you please enhance SubDomainizer by adding one more switch to scan already downloaded js files from a folder.

I love this tools, it's quite faster, and it has a features particularly i love that is

• Github data checking scan

• External Javascript examination

But particulary, Github scan, so It is possible to add just this few features

• Json output

• Query subdomain IP

And also add support for module import, that way it will support this tool from being used in
other python scripts. I wanted to intergrate this tool at https://www.nmmapper.com and particulary at it's collection of subdomain tools

So to be something like this

import subdomainizer
scan_result = subdomainizer.subdomain_scan("example.com")

git_scan_result = subdomainizer.git_scan("whatever")

Thank you.

It would be good if you add some more cloud storage services like digitaloceanspaces.com,windows.net,storage.googleapis.com etc.

By switched on search in GitHub such error appears here

Hi, this is awesome project for recon, May I ask, you have any plan add headless support? Headless mode can detect more domain and js files.

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Smartphone (please complete the following information):

• Device: [e.g. iPhone6]
• OS: [e.g. iOS8.1]
• Browser [e.g. stock browser, safari]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

As you mentioned in #13 you'll be re-writing the code to support showing the URL's that secrets are found from. Is it possible to also increase the parallelism of this script?

When running this to scan 100~ url's it takes hours. I created a quick wrapper in golang to have an instance of the script run on all of my cpu's cores and it finished in 15 minutes so there are definitely some bottlenecks in the code slowing it down that could likely be threaded.

Thanks for making this amazing tool!

func SubDomainizer(dir string) {
	println("starting SubDomainizer")

	if _, err := os.Stat(dir + "/" + date + "/" + "subdomainizer/domains"); os.IsNotExist(err) {
		os.MkdirAll(dir+"/"+date+"/"+"subdomainizer/domains", os.ModePerm)
	}
	if _, err := os.Stat(dir + "/" + date + "/" + "subdomainizer/cloud"); os.IsNotExist(err) {
		os.MkdirAll(dir+"/"+date+"/"+"subdomainizer/cloud", os.ModePerm)
	}
	if _, err := os.Stat(dir + "/" + date + "/" + "subdomainizer/secrets"); os.IsNotExist(err) {
		os.MkdirAll(dir+"/"+date+"/"+"subdomainizer/secrets", os.ModePerm)
	}

	var wg = sync.WaitGroup{}
	maxGoroutines := 10
	guard := make(chan struct{}, maxGoroutines)

	domains := ReadFile(dir + "[redacted]")
	for _, domain := range domains {
		guard <- struct{}{}
		wg.Add(1)
		go func(dir string, date string, domain string) {
			hash := GenerateRandomString()

			cmd := exec.Command("python3", "[redacted]]tools/SubDomainizer/SubDomainizer.py", "-u", domain,
				"-o", dir+"/"+date+"/subdomainizer/domains/"+hash+"_domains.txt", "-cop", dir+"/"+date+"/subdomainizer/cloud/"+hash+"_cloud.txt", "-sop", dir+"/"+date+"/subdomainizer/secrets/"+hash+"_secrets.txt",
				"-g", "-gt", "[redacted]")

			println(cmd.String())
			cmd.Start()
			cmd.Wait()
			<-guard
			wg.Done()
		}(dir, date, domain)

	}
	wg.Wait()

}