Giter Club home page Giter Club logo

Comments (12)

opsdisk avatar opsdisk commented on June 1, 2024 1

Thanks for that info @halimB8 I'll see if I can replicate it on my end. It may be a week or two though until I can get to it.

from pagodo.

opsdisk avatar opsdisk commented on June 1, 2024 1

As a heads up @halimB8 , I released yagooglesearch yesterday. I rewrote the entire underlying library that powers pagodo. It supports HTTP 429 auto detection/backoff and has native proxy support (https://github.com/opsdisk/yagooglesearch#http-and-socks5-proxy-support). pagodo v2 should be released shortly!

from pagodo.

opsdisk avatar opsdisk commented on June 1, 2024 1

Just released v2! https://github.com/opsdisk/pagodo/releases/tag/v2.0.0

Let me know if you're still running into this issue.

from pagodo.

opsdisk avatar opsdisk commented on June 1, 2024

Thanks for submitting an issue @halimB8

It's hard to pinpoint exactly what's happening and if the traffic is being properly routed. Here are my thoughts:

  • When round-robining traffic, a lower -e value should be permitted, but the 60.0 was higher than the default...and that seemed odd that it immediately returned an HTTP 429 on the first search (unless you'd been running pagodo through it earlier and "burned" those IPs).
  • Google may be returning an HTTP 429 for traffic coming from Tor exit nodes. You might need to spin up your own Virtual Private Servers on a platform like DigitalOcean, Linode, etc.
  • Run tcpdump on your primary interface (eth0 or something like that) before running pagodo to ensure no traffic is bypassing the tunnels
  • Are you able to still search Google using the GUI browser after you get the HTTP 429 errors? If not, the pagodo traffic may be sourcing from your home IP and not going through the tunnels.

I'm leaning towards a misconfigured proxy setup and not an issue with pagodo, but for now, can you provide how you are setting up the Tor proxies? That would assist me in troubleshooting.

from pagodo.

halimB8 avatar halimB8 commented on June 1, 2024

Thanks for your answer @opsdisk,

  • Even after having HTTP 429 errors I still can access google using my browser, which means that pagodo is not using my IP.
  • To setup my Tor proxy I have created in /etc/tor/ 3 tor configuration files torrc.1 torrc.2 and torrc.3, in each one I used a different port, here is an example of one of my configuration files :
SocksPort 9060
DataDirectory /var/lib/tor1
ControlPort 9061

and then in the proxychains4 file configuration I added all my tor proxies like that:

round_robin_chain
chain_len = 1
proxy_dns
remote_dns_subnet 224
tcp_read_time_out 15000
tcp_connect_time_out 8000
[ProxyList]
socks4  127.0.0.1 9050
socks4  127.0.0.1 9060
socks4  127.0.0.1 9062
socks4  127.0.0.1 9064

  • Then I start my proxies in different terminals (for example the proxy1) : tor -f /etc/tor/torrc.1

  • To confirm that it's working, I run my browser with proxychains4 : proxychains4 Firefox google.com
    and logs of proxychains4 looks like that which confirm that everything is working fine and it's doing round robin:

[proxychains] config file found: /etc/proxychains4.conf
[proxychains] preloading /usr/lib/x86_64-linux-gnu/libproxychains.so.4
[proxychains] DLL init: proxychains-ng 4.14
[proxychains] DLL init: proxychains-ng 4.14
[proxychains] DLL init: proxychains-ng 4.14
[proxychains] DLL init: proxychains-ng 4.14
[proxychains] DLL init: proxychains-ng 4.14
[proxychains] Round Robin chain  ...  127.0.0.1:9050  ...  google.com:80  ...  OK
[proxychains] Round Robin chain  ...  127.0.0.1:9060  ...  detectportal.firefox.com:80  ...  OK
[proxychains] Round Robin chain  ...  127.0.0.1:9062  ...  contile.services.mozilla.com:443 [proxychains] DLL init: proxychains-ng 4.14
 ...  OK
[proxychains] Round Robin chain  ...  127.0.0.1:9064  ...  www.google.com:443 [proxychains] DLL init: proxychains-ng 4.14
 ...  OK
[proxychains] Round Robin chain  ...  127.0.0.1:9050  ...  push.services.mozilla.com:443 [proxychains] DLL init: proxychains-ng 4.14
 ...  OK
[proxychains] Round Robin chain  ...  127.0.0.1:9060  ...  incoming.telemetry.mozilla.org:443  ...  OK
[proxychains] Round Robin chain  ...  127.0.0.1:9062  ...  firefox.settings.services.mozilla.com:443  ...  OK
[proxychains] Round Robin chain  ...  127.0.0.1:9064  ...  incoming.telemetry.mozilla.org:443  ...  OK
[proxychains] Round Robin chain  ...  127.0.0.1:9050  ...  r3.o.lencr.org:80  ...  OK
[proxychains] Round Robin chain  ...  127.0.0.1:9060  ...  ocsp.pki.goog:80  ...  OK
[proxychains] Round Robin chain  ...  127.0.0.1:9062  ...  detectportal.firefox.com:80  ...  OK
[proxychains] Round Robin chain  ...  127.0.0.1:9064  ...  ocsp.digicert.com:80  ...  OK
[proxychains] Round Robin chain  ...  127.0.0.1:9050  ...  ocsp.digicert.com:80  ...  OK
[proxychains] Round Robin chain  ...  127.0.0.1:9060  ...  detectportal.firefox.com:80  ...  OK
[proxychains] Round Robin chain  ...  127.0.0.1:9062  ...  ocsp.digicert.com:80  ...  OK
[proxychains] Round Robin chain  ...  127.0.0.1:9064  ...  www.gstatic.com:443  ...  OK
[proxychains] Round Robin chain  ...  127.0.0.1:9050  ...  www.gstatic.com:443  ...  OK
[proxychains] Round Robin chain  ...  127.0.0.1:9060  ...  ocsp.pki.goog:80  ...  OK
[proxychains] Round Robin chain  ...  127.0.0.1:9062  ...  adservice.google.com:443  ...  OK
[proxychains] Round Robin chain  ...  127.0.0.1:9064  ...  googleads.g.doubleclick.net:443  ...  OK
[proxychains] Round Robin chain  ...  127.0.0.1:9050  ...  www.google.com:443  ...  OK
[proxychains] Round Robin chain  ...  127.0.0.1:9060  ...  www.google.com:443  ...  OK
[proxychains] Round Robin chain  ...  127.0.0.1:9062  ...  ocsp.pki.goog:80  ...  OK
  • And when I search for my IP addressee, I find that it's not my real IP

from pagodo.

opsdisk avatar opsdisk commented on June 1, 2024

Hey @halimB8 - in the middle of rewriting a new Google search library right now for pagodo, but wanted to have you check something:

"Even after having HTTP 429 errors I still can access google using my browser" - have you tried executing a Google search through the browser after that? I can browse to google.com all day on Tor, but anytime I try and search, I'll get the reCAPTCHA screen. With the Tor exit nodes being public, I think Google uses that and will squash most searches through Tor without a reCAPTCHA verification (which pagodo can't currently do).

from pagodo.

halimB8 avatar halimB8 commented on June 1, 2024

Hey @opsdisk Thanks for your answers,

I just tried again and now before the HTTP 429 error, I am getting an error with dork like that :

[-] Error with dork: index.of.secret
[-] EXCEPTION: HTTP Error 429: Too Many Requests
[*] Google is blocking you, looks like you need to spread out the Google searches.  Don't know how to utilize SSH and dynamic socks proxies?  Do yourself a favor and pick up a copy of The Cyber Plumber's Handbook and interactive lab (https://gumroad.com/l/cph_book_and_lab) to learn all about Secure Shell (SSH) tunneling, port redirection, and bending traffic like a boss.

and yes I tried to search on google on my browser after getting that error and it didn't ask me for a reCAPTCHA, I even took some dorks from sensitive_directories.dorks and run them manually in my browser and worked fine

from pagodo.

halimB8 avatar halimB8 commented on June 1, 2024

Thanks @opsdisk for this fast and great work.
So I just tried again with the same config, and this time I got an error and then a warning that google is blocking my IP, and then it sleeps for 60min
Here is the command I run:

proxychains4 python3 pagodo.py -d myDomain.com -g dorks/files_containing_juicy_info.dorks -o -s

and here is what I got :

[proxychains] config file found: /etc/proxychains4.conf
[proxychains] preloading /usr/lib/x86_64-linux-gnu/libproxychains.so.4
[proxychains] DLL init: proxychains-ng 4.14
2021-09-01 17:24:25,472 [MainThread  ] [INFO] Initiation timestamp: 2021-09-01T17:24:25.472163
2021-09-01 17:24:25,472 [MainThread  ] [INFO] Search ( 1 / 942 ) for Google dork [ site:*******.com intitle:"Ganglia" "Cluster Report for" ] using User-Agent 'Mozilla/5.0 (X11; U; Linux i686; de; rv:1.9.1.1) Gecko/20090722 Gentoo Firefox/3.5.1' through proxy ''
2021-09-01 17:24:25,472 [MainThread  ] [INFO] Requesting URL: https://www.google.com/
[proxychains] Round Robin chain  ...  127.0.0.1:9050  ...  www.google.com:443 <--socket error or timeout!
2021-09-01 17:24:40,489 [MainThread  ] [ERROR] Error with dork: intitle:"Ganglia" "Cluster Report for"
2021-09-01 17:24:40,489 [MainThread  ] [ERROR] EXCEPTION: HTTPSConnectionPool(host='www.google.com', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f41145ed970>: Failed to establish a new connection: [Errno 111] Connection refused'))
2021-09-01 17:24:40,489 [MainThread  ] [INFO] Sleeping 53.6 seconds before executing the next dork search...
2021-09-01 17:25:34,143 [MainThread  ] [INFO] Search ( 2 / 942 ) for Google dork [ site:*****.com allinurl:/examples/jsp/snp/snoop.jsp ] using User-Agent 'Opera/9.80 (Windows NT 5.2; U; en) Presto/2.2.15 Version/10.00' through proxy ''
2021-09-01 17:25:34,144 [MainThread  ] [INFO] Requesting URL: https://www.google.com/
[proxychains] Round Robin chain  ...  127.0.0.1:9060  ...  www.google.com:443  ...  OK
2021-09-01 17:25:34,922 [MainThread  ] [INFO] Stats: start=0, num=100, total_valid_links_found=0 / max_search_result_urls_to_return=100
2021-09-01 17:25:34,923 [MainThread  ] [INFO] Requesting URL: https://www.google.com/search?hl=en&q=site%3A*******.com+allinurl%3A%2Fexamples%2Fjsp%2Fsnp%2Fsnoop.jsp&num=100&btnG=Google+Search&tbs=li:1&safe=off&cr=&filter=0
[proxychains] Round Robin chain  ...  127.0.0.1:9062  ...  www.google.com:443  ...  OK
2021-09-01 17:25:36,135 [MainThread  ] [WARNING] Google is blocking your IP for making too many requests in a specific time period.
2021-09-01 17:25:36,136 [MainThread  ] [INFO] Sleeping for 60 minutes...

from pagodo.

opsdisk avatar opsdisk commented on June 1, 2024

For grins, can you try using the native proxy support without proxychains4? I want to determine if it's a proxychains4 or Tor issue.

So instead of prepending the command with proxychains4, use:

python pagodo.py -g dorks.txt -p socks5h://127.0.0.1:9050,socks5h://127.0.0.1:9060,socks5h://127.0.0.1:9062,socks5h://127.0.0.1:9064

Unless they were updated, I used the proxies you specified here: #55 (comment)

from pagodo.

halimB8 avatar halimB8 commented on June 1, 2024

I Think it's tor issue, cause I just run the command you asked me for :

python3 pagodo.py -g dorks/web_server_detection.dorks -d myDOmain.com -p socks5h://127.0.0.1:9050,socks5h://127.0.0.1:9060,socks5h://127.0.0.1:9062,socks5h://127.0.0.1:9064

And from the first search I got a warning and it sleeps for 60min as you can see here :

2021-09-02 17:48:07,669 [MainThread  ] [INFO] Initiation timestamp: 2021-09-02T17:48:07.669896
2021-09-02 17:48:07,670 [MainThread  ] [INFO] Search ( 1 / 186 ) for Google dork [ site:myDomain.com "Novell, Inc" WEBACCESS Username Password "Version *.*" Copyright -inurl:help -guides|guide ] using User-Agent 'Opera/9.80 (X11; Linux i686; U; en) Presto/2.5.27 Version/10.60' through proxy 'socks5h://127.0.0.1:9050'
2021-09-02 17:48:07,670 [MainThread  ] [INFO] Requesting URL: https://www.google.com/
2021-09-02 17:48:09,667 [MainThread  ] [INFO] Stats: start=0, num=100, total_valid_links_found=0 / max_search_result_urls_to_return=100
2021-09-02 17:48:09,667 [MainThread  ] [INFO] Requesting URL: https://www.google.com/search?hl=en&q=site%3AmyDOmain.com+%22Novell%2C+Inc%22+WEBACCESS+Username+Password+%22Version+%2A.%2A%22+Copyright+-inurl%3Ahelp+-guides%7Cguide&num=100&btnG=Google+Search&tbs=li:1&safe=off&cr=&filter=0
2021-09-02 17:48:12,398 [MainThread  ] [WARNING] Google is blocking your IP for making too many requests in a specific time period.
2021-09-02 17:48:12,399 [MainThread  ] [INFO] Sleeping for 60 minutes...

from pagodo.

opsdisk avatar opsdisk commented on June 1, 2024

The sleeping is because an HTTP 429 was received by pagodo from Google. I haven't been able to set up a Tor test environment to confirm that it's Tor, but that's still my suspicion.

from pagodo.

opsdisk avatar opsdisk commented on June 1, 2024

You still want me to keep this issue open @halimB8 ?

from pagodo.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.