opsdisk / metagoofil Goto Github PK
View Code? Open in Web Editor NEWSearch Google and download specific file types
License: Other
Search Google and download specific file types
License: Other
pip install -r requirements.txt
pip install -r requirements.txt
Collecting google==3.0.0
Using cached google-3.0.0-py2.py3-none-any.whl (45 kB)
ERROR: Could not find a version that satisfies the requirement requests==2.28.1 (from versions: 0.2.0, 0.2.1, 0.2.2, 0.2.3, 0.2.4, 0.3.0, 0.3.1, 0.3.2, 0.3.3, 0.3.4, 0.4.0, 0.4.1, 0.5.0, 0.5.1, 0.6.0, 0.6.1, 0.6.2, 0.6.3, 0.6.4, 0.6.5, 0.6.6, 0.7.0, 0.7.1, 0.7.2, 0.7.3, 0.7.4, 0.7.5, 0.7.6, 0.8.0, 0.8.1, 0.8.2, 0.8.3, 0.8.4, 0.8.5, 0.8.6, 0.8.7, 0.8.8, 0.8.9, 0.9.0, 0.9.1, 0.9.2, 0.9.3, 0.10.0, 0.10.1, 0.10.2, 0.10.3, 0.10.4, 0.10.6, 0.10.7, 0.10.8, 0.11.1, 0.11.2, 0.12.0, 0.12.1, 0.13.0, 0.13.1, 0.13.2, 0.13.3, 0.13.4, 0.13.5, 0.13.6, 0.13.7, 0.13.8, 0.13.9, 0.14.0, 0.14.1, 0.14.2, 1.0.0, 1.0.1, 1.0.2, 1.0.3, 1.0.4, 1.1.0, 1.2.0, 1.2.1, 1.2.2, 1.2.3, 2.0.0, 2.0.1, 2.1.0, 2.2.0, 2.2.1, 2.3.0, 2.4.0, 2.4.1, 2.4.2, 2.4.3, 2.5.0, 2.5.1, 2.5.2, 2.5.3, 2.6.0, 2.6.1, 2.6.2, 2.7.0, 2.8.0, 2.8.1, 2.9.0, 2.9.1, 2.9.2, 2.10.0, 2.11.0, 2.11.1, 2.12.0, 2.12.1, 2.12.2, 2.12.3, 2.12.4, 2.12.5, 2.13.0, 2.14.0, 2.14.1, 2.14.2, 2.15.1, 2.16.0, 2.16.1, 2.16.2, 2.16.3, 2.16.4, 2.16.5, 2.17.0, 2.17.1, 2.17.2, 2.17.3, 2.18.0, 2.18.1, 2.18.2, 2.18.3, 2.18.4, 2.19.0, 2.19.1, 2.20.0, 2.20.1, 2.21.0, 2.22.0, 2.23.0, 2.24.0, 2.25.0, 2.25.1, 2.26.0, 2.27.0, 2.27.1)
ERROR: No matching distribution found for requests==2.28.1
Can't pinpoint what is causing this, it usually happens after a big search (-l 500 for example), but after that if persists to even the smallest searches (as seen below)
after that it keeps giving that error no matter what i do and eventually it works for one more run.
# python metagoofil.py -d apple.com -t pdf,doc -l 50 -n 5 -o test -w -f
[*] Downloaded files will be saved here: test
[*] Searching for 50 .pdf files and waiting 30.0 seconds between searches
Traceback (most recent call last):
File "metagoofil.py", line 226, in <module>
mg.go()
File "metagoofil.py", line 124, in go
for url in googlesearch.search(query, start=0, stop=self.search_max, num=100, pause=self.delay, extra_params={'filter': '0'}, user_agent=self.user_agent):
File "/usr/local/lib/python2.7/dist-packages/googlesearch/__init__.py", line 359, in search
html = get_page(url)
File "/usr/local/lib/python2.7/dist-packages/googlesearch/__init__.py", line 147, in get_page
response = urlopen(request)
File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 435, in open
response = meth(req, response)
File "/usr/lib/python2.7/urllib2.py", line 548, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python2.7/urllib2.py", line 467, in error
result = self._call_chain(*args)
File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 654, in http_error_302
return self.parent.open(new, timeout=req.timeout)
File "/usr/lib/python2.7/urllib2.py", line 435, in open
response = meth(req, response)
File "/usr/lib/python2.7/urllib2.py", line 548, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python2.7/urllib2.py", line 473, in error
return self._call_chain(*args)
File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 556, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 503: Service Unavailable
Any ideas?
Hello,
When you run this command(in kali):
metagoofil -d https://sans.org -t doc,pdf,xls -l 200 -o sans_files -f
It fails instead of correctly handling this exception:
urllib.error.HTTPError: HTTP Error 429: Too Many Requests
Issue was first reported here:
https://bugs.kali.org/view.php?id=7005
Both of your functions for positive_int and positive_float are broken. I think the issue is that you are calling type on the value which can't be greater than 0 since type is not an integer or float.
Can we make this compatible with other operating systems such as macOS, or Windows? I am a really big fan of this tool, and would love to see its availability globalized. I have tried running it on my macOS system, and, no matter how much I modify the code, I cannot get it to work. I figured I would reach out to the devs! Huge props, by the way.
Hi,
Was getting this error:
Traceback (most recent call last):
File "./metagoofil.py", line 226, in
mg.go()
File "./metagoofil.py", line 146, in go
self.html_links.close()
AttributeError: Metagoofil instance has no attribute 'html_links'
Fix
From:
if self.save_links:
for f in self.files:
self.html_links.write(f + "\n")
self.html_links.close()
To:
if self.save_links:
for f in self.files:
self.html_links.write(f + "\n")
if self.save_links:
self.html_links.close()
It seems like you are trying to continue the original work and this would be really cool. However, your "fork" has much lesser code. it's 550kb (old) vs 15kb (new).
Could you clarify it please?
Hi,
When doing a search and google does not show all files but instead he show 1 or a few and then there is the following message:
In order to show you the most relevant results, we have omitted some entries very similar to the 1 already displayed.
If you like, you can repeat the search with the omitted results included.
The program only see the first item found by Google and will stop there. It is a problem as I've got some example where 4 to 5 pages of results does offers PDF but only one is shown.
Let me know if you know a way to fix it or if you need help for the code/test.
Thanks
Might it be possible to add a user-agent impersonation option for accessing sites? I've run across things like this a few times: https://stackoverflow.com/questions/25936072/python-urllib2-httperror-http-error-503-service-unavailable-on-valid-website and having a way to set a user agent with a flag would be phenomenal
I encountered this problem after running.
$ proxychains -q python3 metagoofil.py -d nsfocus.com -f -t all -e 60
[*] Searching for 100 .all files and waiting 60.0 seconds between searches
[-] EXCEPTION: <urlopen error EOF occurred in violation of protocol (_ssl.c:997)>
Traceback (most recent call last):
File "/usr/local/python3/lib/python3.10/urllib/request.py", line 1348, in do_open
h.request(req.get_method(), req.selector, req.data, headers,
File "/usr/local/python3/lib/python3.10/http/client.py", line 1282, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/python3/lib/python3.10/http/client.py", line 1328, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/python3/lib/python3.10/http/client.py", line 1277, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/python3/lib/python3.10/http/client.py", line 1037, in _send_output
self.send(msg)
File "/usr/local/python3/lib/python3.10/http/client.py", line 975, in send
self.connect()
File "/usr/local/python3/lib/python3.10/http/client.py", line 1454, in connect
self.sock = self._context.wrap_socket(self.sock,
File "/usr/local/python3/lib/python3.10/ssl.py", line 512, in wrap_socket
return self.sslsocket_class._create(
File "/usr/local/python3/lib/python3.10/ssl.py", line 1070, in _create
self.do_handshake()
File "/usr/local/python3/lib/python3.10/ssl.py", line 1341, in do_handshake
self._sslobj.do_handshake()
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:997)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/osint/osint/domain_col/tool/metagoofil/metagoofil.py", line 162, in go
for url in googlesearch.search(
File "/usr/local/python3/lib/python3.10/site-packages/googlesearch/__init__.py", line 272, in search
get_page(url_home % vars(), user_agent, verify_ssl)
File "/usr/local/python3/lib/python3.10/site-packages/googlesearch/__init__.py", line 174, in get_page
response = urlopen(request)
File "/usr/local/python3/lib/python3.10/urllib/request.py", line 216, in urlopen
return opener.open(url, data, timeout)
File "/usr/local/python3/lib/python3.10/urllib/request.py", line 519, in open
response = self._open(req, data)
File "/usr/local/python3/lib/python3.10/urllib/request.py", line 536, in _open
result = self._call_chain(self.handle_open, protocol, protocol +
File "/usr/local/python3/lib/python3.10/urllib/request.py", line 496, in _call_chain
result = func(*args)
File "/usr/local/python3/lib/python3.10/urllib/request.py", line 1391, in https_open
return self.do_open(http.client.HTTPSConnection, req,
File "/usr/local/python3/lib/python3.10/urllib/request.py", line 1351, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error EOF occurred in violation of protocol (_ssl.c:997)>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/osint/osint/domain_col/tool/metagoofil/metagoofil.py", line 381, in <module>
mg.go()
File "/home/osint/osint/domain_col/tool/metagoofil/metagoofil.py", line 175, in go
if e.code == 429:
AttributeError: 'URLError' object has no attribute 'code'
Having installed pip install google
, this version of metagoofil errors with "ImportError: No module named google"
My fix was to replace "google" by "googlesearch" 3 times in lines 7 and 105 of metagoofil.py
.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.