Comments (5)
Yes, this feature is in the TODO. I think the only way to implement this is to compare a checksum. If anyone have other options I'm ready to discuss =)
from proxybroker.
Yes, this feature is in the TODO.
Well I completely missed it, sorry about that.
I think the only way to implement this is to compare a checksum.
You might also be able to do size and number of files returned as well. Also sending data thru the proxy and seeing if it was changed on it's way to it's destination would be another way to tell if the proxy was messy with the traffic
from proxybroker.
Theres no real way to compare websites 100% because each html will be rendered different in each request. Checksum will fail so often.
An idea will be to save a list of web elements (the html dom names, not their content) and compare the main structure. You will notice changes for ads or div created between direct and proxy requests.
from proxybroker.
@erm3nda or you could just request a static web page retard
from proxybroker.
@erm3nda I partially agree with this comment as checksum doesn't have to be index.html
specific. Another method would be to do a checksum on the website's resources; i.e., its main .css file, js files, etc. Whatever the case -- whether it is checksum or specific HTML -- there are only two ways this can be achieved:
- Statistically, by verifying > ~10% of proxies respond with the same DOM/checksum request.
- Request from the local machine that saves either checksum or DOM state to be verified against all proxies. This method, however, exposes the IP of the user using ProxyBroker.
Another thing to mention is the fact that not all websites are rendered the same depending on the language. Take NYTimes.com, for example:
Both websites look fairly similar, but it seems that their HTML is rendered differently. Therefore, I believe, it would be preferred the client serves a proper Accept-Language
header and verifies the Content-Language
in the checking of the page on top of whatever other mechanism is being done to verify the page.
from proxybroker.
Related Issues (20)
- Start server in new thread HOT 1
- Log/Output File Change HOT 1
- new free proxy site: https://pzzqz.com with free api HOT 5
- bash: proxybroker: command not found HOT 3
- is it possible to create a docker container with broker.serve()
- set_wakeup_fd only works in main thread of the main interpreter HOT 1
- Unable to start Proxybroker HOT 2
- I'm not sure what i'm doing wrong HOT 1
- Windows: Python-CFFI error HOT 4
- How to convert this ProxyBroker script to exe? HOT 2
- TypeError: function() argument 'code' must be code, not str HOT 1
- I managed to run this on Windows HOT 4
- trabl install end start HOT 1
- TypeError: As of 3.10, the *loop* parameter was removed from Queue() since it is no longer necessary HOT 4
- Not maintained. USE `proxybroker2` INSTEAD. HOT 1
- Error
- /home/user/.local/lib/python3.10/site-packages/proxybroker/cli.py:379: DeprecationWarning: There is no current event loop loop = asyncio.get_event_loop() Traceback (most recent call last): File "/home/user/.local/bin/proxybroker", line 8, in <module> sys.exit(cli()) File "/home/user/.local/lib/python3.10/site-packages/proxybroker/cli.py", line 380, in cli proxies = asyncio.Queue(loop=loop) File "/usr/lib/python3.10/asyncio/queues.py", line 34, in __init__ super().__init__(loop=loop) File "/usr/lib/python3.10/asyncio/mixins.py", line 17, in __init__ raise TypeError( TypeError: As of 3.10, the *loop* parameter was removed from Queue() since it is no longer necessary
- TypeError: Queue.__init__() got an unexpected keyword argument 'loop' HOT 3
- What is judges? HOT 1
- TypeError: As of 3.10, the *loop* parameter was removed from Queue() since it is no longer necessary
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from proxybroker.