constverum / proxybroker Goto Github PK

Using Kali Linux 2.0

/usr/local/lib/python3.5/dist-packages/aiohttp/client.py:479: DeprecationWarning: Use async with instead
warnings.warn("Use async with instead", DeprecationWarning)
Exception ignored in: <bound method DNSResolver._sock_state_cb of <aiodns.DNSResolver object at 0x7ffff355f978>>
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/aiodns/init.py", line 85, in _sock_state_cb
self.loop.add_reader(fd, self._handle_event, fd, READ)
File "/usr/lib/python3.5/asyncio/selector_events.py", line 336, in add_reader
self._ensure_fd_no_transport(fd)
File "/usr/lib/python3.5/asyncio/selector_events.py", line 258, in _ensure_fd_no_transport
fd, transport))
RuntimeError: File descriptor 87 is used by transport <_SelectorSocketTransport fd=87 read=polling write=<idle, bufsize=0>>
Exception ignored in: <bound method DNSResolver._sock_state_cb of <aiodns.DNSResolver object at 0x7ffff355fa58>>
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/aiodns/init.py", line 85, in _sock_state_cb
self.loop.add_reader(fd, self._handle_event, fd, READ)
File "/usr/lib/python3.5/asyncio/selector_events.py", line 336, in add_reader
self._ensure_fd_no_transport(fd)
File "/usr/lib/python3.5/asyncio/selector_events.py", line 258, in _ensure_fd_no_transport
fd, transport))
RuntimeError: File descriptor 88 is used by transport <_SelectorSocketTransport fd=88 read=polling write=<idle, bufsize=0>>

proxybroker --max-conn 10 --providers ['http://txt.proxyspy.net/proxy.txt', 'http://skypegrab.net/proxy/http.txt'] find --types HTTP HTTPS --lvl High --countries US --strict -l 10

or

proxybroker --max-conn 10 --providers 'http://txt.proxyspy.net/proxy.txt', 'http://skypegrab.net/proxy/http.txt' find --types HTTP HTTPS --lvl High --countries US --strict -l 10

Getting Below Error:
proxybroker: error: argument command: invalid choice: 'HTTP' (choose from 'find', 'grab', 'serve')

C:\Python34\Scripts>proxybroker.exe Traceback (most recent call last): File "c:\python34\lib\runpy.py", line 170, in _run_module_as_main "__main__", mod_spec) File "c:\python34\lib\runpy.py", line 85, in _run_code exec(code, run_globals) File "C:\Python34\Scripts\proxybroker.exe\__main__.py", line 5, in <module> File "c:\python34\lib\site-packages\proxybroker\__init__.py", line 28, in <mod ule> from .proxy import Proxy File "c:\python34\lib\site-packages\proxybroker\proxy.py", line 35 async def create(cls, host, *args, **kwargs): ^ SyntaxError: invalid syntax

Getting this Error when i try to start Proxybroker from Script Folder under Windows XP - Any Solution on this?

Could it use maxminds geo ip database or another source to show which city/state of the proxy in addition to country?

As far as I understand reading proxybroker/resolver.py it only uses http://httpbin.org/ to check IP. Maybe you should use some backup IP checking sites in case httpbin.org is down. I suggest http://freegeoip.net/ and some proxy judges:http://www.proxyjudge.info/, http://proxyjudge.us/, http://azenv.net/

This is how I run proxy server
proxybroker serve --host 127.0.0.1 --port 8888 --types HTTP HTTPS --lvl High

I configure 127.0.0.1 and 8888 as proxy in fireforx
and try to access the site google.com or github.com or any https site.
I get this error
any suggestion?

Detail of error.

Your connection is not secure

The owner of www.google.com has configured their website improperly. To protect your information from being stolen, Firefox has not connected to this website.

This site uses HTTP Strict Transport Security (HSTS) to specify that Firefox may only connect to it securely. As a result, it is not possible to add an exception for this certificate.

It'd be awesome if you could add a judge for SMTP (supply a host as param).
You might be able to use aiosmtplib...
Tks!

I tried to install through pip and received the following error:

Collecting proxybroker
Using cached proxybroker-0.1.2.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "", line 1, in
File "/tmp/pip-build-CX9FBL/proxybroker/setup.py", line 10, in
VERSION = import(PACKAGE).version
File "proxybroker/init.py", line 2
SyntaxError: Non-ASCII character '\xc2' in file proxybroker/init.py on line 3, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details

----------------------------------------

Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-CX9FBL/proxybroker

My run of it:
$ proxybroker find --types HTTP HTTPS --lvl High --countri DE CH AT --strict

.. /python3.6/site-packages/aiohttp/client.py:576: DeprecationWarning: Use async with instead
  warnings.warn("Use async with instead", DeprecationWarning)
.. /python3.6/site-packages/aiohttp/helpers.py:129: DeprecationWarning: ClientSession.close() is a coroutine
  warnings.warn(self._msg, DeprecationWarning)

...and the program hangs.
Errors like it with other commands, and on run it as framework from a script.

Used Command:
$ proxybroker find --types HTTP HTTPS --lvl High --countries DE CH AT --strict

Error Message:

/usr/local/lib/python3.5/dist-packages/aiohttp/client.py:565: DeprecationWarning: Use async with instead
  warnings.warn("Use async with instead", DeprecationWarning)

ProxyBroker is starting & everything is just working fine, but it always gives me this Error, but why?

While trying any example from your website i get the following output:

`(python_virtual) C:\Users\Admin\Desktop\python_virtual>python helloworld.py
C:\Users\Admin\Desktop\python_virtual\lib\site-packages\aiohttp\client.py:526: D
eprecationWarning: Use async with instead
warnings.warn("Use async with instead", DeprecationWarning)
C:\Users\Admin\Desktop\python_virtual\lib\site-packages\aiohttp\client.py:526: D
eprecationWarning: Use async with instead
warnings.warn("Use async with instead", DeprecationWarning)
C:\Users\Admin\Desktop\python_virtual\lib\site-packages\aiohttp\client.py:526: D
eprecationWarning: Use async with instead
warnings.warn("Use async with instead", DeprecationWarning)
C:\Users\Admin\Desktop\python_virtual\lib\site-packages\aiohttp\client.py:526: D
eprecationWarning: Use async with instead
warnings.warn("Use async with instead", DeprecationWarning)
C:\Users\Admin\Desktop\python_virtual\lib\site-packages\aiohttp\client.py:526: D
eprecationWarning: Use async with instead
warnings.warn("Use async with instead", DeprecationWarning)
C:\Users\Admin\Desktop\python_virtual\lib\site-packages\aiohttp\client.py:526: D
eprecationWarning: Use async with instead
warnings.warn("Use async with instead", DeprecationWarning)
C:\Users\Admin\Desktop\python_virtual\lib\site-packages\aiohttp\client.py:526: D
eprecationWarning: Use async with instead
warnings.warn("Use async with instead", DeprecationWarning)
C:\Users\Admin\Desktop\python_virtual\lib\site-packages\aiohttp\client.py:526: D
eprecationWarning: Use async with instead
warnings.warn("Use async with instead", DeprecationWarning)
C:\Users\Admin\Desktop\python_virtual\lib\site-packages\aiohttp\client.py:526: D
eprecationWarning: Use async with instead
warnings.warn("Use async with instead", DeprecationWarning)
C:\Users\Admin\Desktop\python_virtual\lib\site-packages\aiohttp\client.py:526: D
eprecationWarning: Use async with instead
warnings.warn("Use async with instead", DeprecationWarning)
Traceback (most recent call last):
File "helloworld.py", line 29, in
main()
File "helloworld.py", line 25, in main
loop.run_until_complete(tasks)
File "c:\users\admin\appdata\local\programs\python\python36-32\Lib\asyncio\bas
e_events.py", line 466, in run_until_complete
return future.result()
File "C:\Users\Admin\Desktop\python_virtual\lib\site-packages\proxybroker\api.
py", line 108, in find
await self._run(self._checker.check_judges(), action)
File "C:\Users\Admin\Desktop\python_virtual\lib\site-packages\proxybroker\api.
py", line 114, in _run
await tasks
File "C:\Users\Admin\Desktop\python_virtual\lib\site-packages\proxybroker\chec
ker.py", line 26, in check_judges
await asyncio.gather(*[j.check() for j in self._judges])
File "C:\Users\Admin\Desktop\python_virtual\lib\site-packages\proxybroker\judg
e.py", line 62, in check
page = await resp.text()
File "C:\Users\Admin\Desktop\python_virtual\lib\site-packages\aiohttp\client_r
eqrep.py", line 764, in text
return self._content.decode(encoding, errors=errors)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf6 in position 5569: inval
id start byte

(python_virtual) C:\Users\Admin\Desktop\python_virtual>`

I understand that ProxyBroker is a proper proxy that uses Connect Tunneling for HTTPS requests. Is there any intention to allow proxybroker to terminate ssl such that HTTPS response codes can indicate whether a proxy is not functioning properly.

Hello,

I tried your code and there is an error: TypeError: init() got multiple values for argument 'judge'. The traceback as follows:

Traceback (most recent call last):
File "./test.py", line 77, in
start(500)
File "./test.py", line 59, in start
worker.start()
File "/proxychecker/pyproxychecker/checker.py", line 100, in start
self.loop.run_until_complete(self.run())
File "/anaconda3/lib/python3.5/asyncio/base_events.py", line 342, in run_until_complete
return future.result()
File "/anaconda3/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/anaconda3/lib/python3.5/asyncio/tasks.py", line 239, in _step
result = coro.send(value)
File "/proxychecker/pyproxychecker/checker.py", line 109, in run
await self.clear_proxies()
File "/proxychecker/pyproxychecker/checker.py", line 171, in clear_proxies
for p in self.proxies['input']]
File "/proxychecker/pyproxychecker/checker.py", line 171, in
for p in self.proxies['input']]
TypeError: init() got multiple values for argument 'judge'

import asyncio
from proxybroker import Broker
outFile=open('result.txt','w')
loop = asyncio.get_event_loop()

proxies = asyncio.Queue(loop=loop)
broker = Broker(proxies, loop=loop)
loop.run_until_complete(broker.grab(limit=1000))
while True:
proxy = proxies.get_nowait()
if proxy is None: break
print('Found proxy: %s' % proxy)
outFile.write(str(proxy)+'\n')

hi i this is my code ,when limit over 10K ,it is error ,how can i get all proxys without check

Detailed description and possible solution in the following Stackoverflow question:

http://stackoverflow.com/questions/39018969/using-proxybroker-in-python3-5-throws-an-encoding-error/39053948#39053948

I get:

$ pip install proxybroker
Collecting proxybroker
Using cached proxybroker-0.2.0.tar.gz
Complete output from command python setup.py egg_info:
zip_safe flag not set; analyzing archive contents...

Installed /tmp/pip-build-FKQ8Z_/proxybroker/.eggs/pytest_runner-2.12.1-py2.7.egg
error in proxybroker setup command: package_data must be a dictionary mapping package names to lists of wildcard patterns

----------------------------------------

Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-FKQ8Z_/proxybroker/

and

$ pip install -U git+https://github.com/constverum/ProxyBroker.git
Collecting git+https://github.com/constverum/ProxyBroker.git
Cloning https://github.com/constverum/ProxyBroker.git to /tmp/pip-zcYk3h-build
Complete output from command python setup.py egg_info:
zip_safe flag not set; analyzing archive contents...

Installed /tmp/pip-zcYk3h-build/.eggs/pytest_runner-2.12.1-py2.7.egg
error in proxybroker setup command: package_data must be a dictionary mapping package names to lists of wildcard patterns

----------------------------------------

Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-zcYk3h-build/

What should i do?

I have proxy list like this:
socks5://login1:pass1@ip1:port1
socks5://login2:pass2@ip1:port1
socks5://login3:pass3@ip1:port1
socks5://login4:pass4@ip2:port1
socks5://login5:pass5@ip2:port1
socks5://login6:pass6@ip2:port1
socks5://login7:pass7@ip2:port1

But i see proxybroker can not work with proxies with auth!
How i can fix it?

I've tried to install proxybroker and got following error:

error in proxybroker setup command: package_data must be a dictionary mapping package names to lists of wildcard patterns

----------------------------------------

Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/w5/nwdvwf2j6mz1dp95y2fwtjb00000gn/T/pip-build-4gORWY/proxybroker/

Is it Possible to add an command to execlude specific ports? So it wouldn't grab Proxys with for example: Port 80, Port 8080?

If I install the project with pip install proxybroker, the command proxybroker is missing. If I install the dev. version directly from github, the proxybroker is there.

Either update the pypi version or mention this strangeness in the documentation. BTW, thanks for the project, it'll be useful for me.

returning true from eof_received() has no effect when using ssl, why returun this? plz help me, thx :)

Python Version: Python 3.5.2
Proxybroker Version: proxybroker 0.2.0b3

Current Behavior

proxybroker find --types socks4 --countries US --limit 10
proxybroker find: error: argument --types: invalid choice: 'socks4' (choose from 'HTTP', 'HTTPS', 'SOCKS4', 'SOCKS5', 'CONNECT:80', 'CONNECT:25')

Expected Behavior

proxybroker will detect the type from CLI without parsing case sensitivity.

I ran command:

proxybroker find --types HTTPS --lvl High --countries US --strict -l 10

Exception ignored in: <bound method _SSLProtocolTransport.del of <asyncio.sslproto._SSLProtocolTransport object at 0x00000279F0E28470>>
Traceback (most recent call last):
File "c:\users\rahul\anaconda3\lib\asyncio\sslproto.py", line 328, in del
self.close()
File "c:\users\rahul\anaconda3\lib\asyncio\sslproto.py", line 319, in close
self._ssl_protocol._start_shutdown()
File "c:\users\rahul\anaconda3\lib\asyncio\sslproto.py", line 542, in _start_shutdown
self._write_appdata(b'')
File "c:\users\rahul\anaconda3\lib\asyncio\sslproto.py", line 547, in _write_appdata
self._process_write_backlog()
File "c:\users\rahul\anaconda3\lib\asyncio\sslproto.py", line 651, in _process_write_backlog
self._fatal_error(exc, 'Fatal error on SSL transport')
File "c:\users\rahul\anaconda3\lib\asyncio\sslproto.py", line 658, in _fatal_error
if isinstance(exc, base_events._FATAL_ERROR_IGNORE):
NameError: name 'base_events' is not defined
Exception ignored in: <bound method _SSLProtocolTransport.del of <asyncio.sslproto._SSLProtocolTransport object at 0x00000279F4534B70>>
Traceback (most recent call last):
File "c:\users\rahul\anaconda3\lib\asyncio\sslproto.py", line 328, in del
self.close()
File "c:\users\rahul\anaconda3\lib\asyncio\sslproto.py", line 319, in close
self._ssl_protocol._start_shutdown()
File "c:\users\rahul\anaconda3\lib\asyncio\sslproto.py", line 542, in _start_shutdown
self._write_appdata(b'')
File "c:\users\rahul\anaconda3\lib\asyncio\sslproto.py", line 547, in _write_appdata
self._process_write_backlog()
File "c:\users\rahul\anaconda3\lib\asyncio\sslproto.py", line 651, in _process_write_backlog
self._fatal_error(exc, 'Fatal error on SSL transport')
File "c:\users\rahul\anaconda3\lib\asyncio\sslproto.py", line 658, in _fatal_error
if isinstance(exc, base_events._FATAL_ERROR_IGNORE):
NameError: name 'base_events' is not defined

Python Version: 3.5.2
OS: Windows 10

A neat feature would be to check if the proxy was injecting ads/javascript (or in any other way modifying traffic)

Here's another source to add:
link: http://pubproxy.com/
api: http://pubproxy.com/api/proxy?limit=20&format=txt

When I execute

proxybroker find --types HTTP HTTPS --lvl High --countries US --strict -l 10

I receive

Fatal Python error: ffi.from_handle() detected that the address passed points to garbage. If it is really the result of ffi.new_handle(), then the Python object has already been garbage collected

Thread 0x00000d08 (most recent call first):
  File "c:\python35\lib\threading.py", line 293 in wait
  File "c:\python35\lib\queue.py", line 164 in get
  File "c:\python35\lib\concurrent\futures\thread.py", line 64 in _worker
  File "c:\python35\lib\threading.py", line 862 in run
  File "c:\python35\lib\threading.py", line 914 in _bootstrap_inner
  File "c:\python35\lib\threading.py", line 882 in _bootstrap

Current thread 0x000052f4 (most recent call first):
  File "c:\python35\lib\site-packages\pycares\_cfficore\__init__.py", line 122 in _sock_state_cb
  File "c:\python35\lib\site-packages\pycares\_cfficore\__init__.py", line 455 in process_fd
  File "c:\python35\lib\site-packages\pycares\_cfficore\__init__.py", line 113 in wrapper
  File "c:\python35\lib\site-packages\aiodns\__init__.py", line 103 in _handle_event
  File "c:\python35\lib\asyncio\events.py", line 125 in _run
  File "c:\python35\lib\asyncio\base_events.py", line 1254 in _run_once
  File "c:\python35\lib\asyncio\base_events.py", line 295 in run_forever
  File "c:\python35\lib\asyncio\base_events.py", line 325 in run_until_complete
  File "c:\python35\lib\site-packages\proxybroker\cli.py", line 310 in cli
  File "C:\Python35\Scripts\proxybroker-script.py", line 9 in <module>

on last version from pip and from this repository.

Windows 8.1 x64, Python 3.5.1
cffi versions 1.5, 1.6

Whenever I run proxybroker I get this exact error:

python/lib/python3.5/site-packages/aiohttp/client.py:575:`
 DeprecationWarning: Use async with instead
  warnings.warn("Use async with instead", DeprecationWarning)

I've installed it on 2 machines and still get the same error. If I install proxybroker directly from github the command works fine. If I install proxybroker through pip, the command isn't found.

I encountered a new error stating: "Could not get a external IP.".

I took a look around to figure out if I can fix it or if I caused the error somehow. To me it looks like proxybroker tries to connect to "http://httpbin.org/ip" but the timeout of 3 seconds fails. I checked whether the mentioned website was working properly. It is up and running with fast response times.
I tried running proxybroker on a server, to check whether or not it is a local problem with the same error popping up.
The funny thing is sometimes proxybroker actually fetched a few proxies before the following error was thrown.

Do you have any idea what else could cause this error, but the website? And more important, how to fix this issue?

Any feedback is appreciated!

/home/sebastian/PycharmProjects/STW/venv/lib/python3.5/site-packages/aiohttp/client.py:494: DeprecationWarning: Use async with instead
  warnings.warn("Use async with instead", DeprecationWarning)
Traceback (most recent call last):
  File "/home/sebastian/PycharmProjects/STW/venv/lib/python3.5/site-packages/proxybroker/utils.py", line 101, in set_my_ip
    data = await resp.json()
  File "/home/sebastian/PycharmProjects/STW/venv/lib/python3.5/site-packages/async_timeout/__init__.py", line 44, in __exit__
    raise asyncio.TimeoutError from None
concurrent.futures._base.TimeoutError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/sebastian/PycharmProjects/testing/test/test_prox2.py", line 19, in <module>
    loop.run_until_complete(tasks)
  File "/usr/lib/python3.5/asyncio/base_events.py", line 387, in run_until_complete
    return future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/lib/python3.5/asyncio/tasks.py", line 241, in _step
    result = coro.throw(exc)
  File "/home/sebastian/PycharmProjects/STW/venv/lib/python3.5/site-packages/proxybroker/api.py", line 98, in find
    await set_my_ip(self._timeout, self._loop)
  File "/home/sebastian/PycharmProjects/STW/venv/lib/python3.5/site-packages/proxybroker/utils.py", line 103, in set_my_ip
    raise RuntimeError('Could not get an external IP. Error: %s' % e)
RuntimeError: Could not get an external IP. Error: 
Exception ignored in: <coroutine object show at 0x7f9b22cdd150>
Traceback (most recent call last):
  File "/home/sebastian/PycharmProjects/testing/test/test_prox2.py", line 7, in show
  File "/usr/lib/python3.5/asyncio/queues.py", line 170, in get
  File "/usr/lib/python3.5/asyncio/futures.py", line 227, in cancel
  File "/usr/lib/python3.5/asyncio/futures.py", line 242, in _schedule_callbacks
  File "/usr/lib/python3.5/asyncio/base_events.py", line 497, in call_soon
  File "/usr/lib/python3.5/asyncio/base_events.py", line 506, in _call_soon
  File "/usr/lib/python3.5/asyncio/base_events.py", line 334, in _check_closed
RuntimeError: Event loop is closed


Process finished with exit code 1

What do you think?

hi from python slack chat!

Hi, and thanks for ProxyBroker :)

I've been trying for a few days to get ProxyBroker to work but I've been unsuccessful. I installed as suggested and tried almost other apparent solution to get this working. Still nothing.

So, If I run this command I get these errors: I also tried via a mkvirtualenv.

me@root:~$ proxybroker find --types HTTP
"Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/proxybroker-0.2.0b3-py3.5.egg/proxybroker/resolver.py", line 68, in get_real_ext_ip
    data = await resp.json()
  File "/usr/local/lib/python3.5/dist-packages/aiohttp/helpers.py", line 564, in __exit__
    raise asyncio.TimeoutError from None
concurrent.futures._base.TimeoutError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/proxybroker", line 9, in <module>
    load_entry_point('proxybroker==0.2.0b3', 'console_scripts', 'proxybroker')()
  File "/usr/local/lib/python3.5/dist-packages/proxybroker-0.2.0b3-py3.5.egg/proxybroker/cli.py", line 310, in cli
    loop.run_until_complete(asyncio.gather(*tasks, loop=loop))
  File "/usr/lib/python3.5/asyncio/base_events.py", line 387, in run_until_complete
    return future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/lib/python3.5/asyncio/tasks.py", line 241, in _step
    result = coro.throw(exc)
  File "/usr/local/lib/python3.5/dist-packages/proxybroker-0.2.0b3-py3.5.egg/proxybroker/api.py", line 149, in find
    ip = await self._resolver.get_real_ext_ip()
  File "/usr/local/lib/python3.5/dist-packages/proxybroker-0.2.0b3-py3.5.egg/proxybroker/resolver.py", line 70, in get_real_ext_ip
    raise RuntimeError('Could not get a external IP. Error: %s' % e)
RuntimeError: Could not get a external IP. Error: 
Exception ignored in: <coroutine object handle at 0x7ff66f244f68>
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/proxybroker-0.2.0b3-py3.5.egg/proxybroker/cli.py", line 258, in handle
  File "/usr/lib/python3.5/asyncio/queues.py", line 170, in get
  File "/usr/lib/python3.5/asyncio/futures.py", line 227, in cancel
  File "/usr/lib/python3.5/asyncio/futures.py", line 242, in _schedule_callbacks
  File "/usr/lib/python3.5/asyncio/base_events.py", line 497, in call_soon
  File "/usr/lib/python3.5/asyncio/base_events.py", line 506, in _call_soon
  File "/usr/lib/python3.5/asyncio/base_events.py", line 334, in _check_closed
RuntimeError: Event loop is closed
me@root:~$

Please help

Thanks, I appreciate it.

Run basic example on Windows 8, python 3.5.1
aiodns==1.0.1
aiohttp==0.21.5
chardet==2.3.0
maxminddb==1.2.0
proxybroker==0.1.2
pycares==1.0.0

Error message:

D:\Temp\proxybroker\venv\Scripts\python.exe D:/Temp/proxybroker/example.py
Traceback (most recent call last):
File "D:/Temp/proxybroker/example.py", line 7, in
broker = Broker(proxies, loop=loop)
File "D:\Temp\proxybroker\venv\lib\site-packages\proxybroker\api.py", line 45, in init
self._setup(timeout, attempts_conn, max_concurrent_conn, verify_ssl)
File "D:\Temp\proxybroker\venv\lib\site-packages\proxybroker\api.py", line 53, in _setup
self._loop.add_signal_handler(signal.SIGINT, self._done)
File "D:\ProgramFiles\Python35\lib\asyncio\events.py", line 472, in add_signal_handler
raise NotImplementedError
NotImplementedError

On this gif ( https://raw.githubusercontent.com/constverum/ProxyBroker/master/docs/source/_static/cli_serve_example.gif ) the server prints an info line when a client connects.

The current version doesn't do that, though it would be very useful. I tried the command that is on the GIF.

proxybroker
│   ├── api.py
│   ├── checker.py
│   ├── cli.py
│   ├── data
GeoLite2-Country.mmdb

How to open GeoLite2-Country.mmdb in proxybroker/data?

I've see that outfile is not being written until whole process has finished.
I tried | tee piping but output did not work.

Tried many things but can't make it work.

The reason to fill this issue, is because i can't know in real time how many proxies do i have "in process" (to be written to file) ... so it's a quite blinded process...

Implementing one of the code examples (or rather just copy pasting it) I encountered the same error I am having for quite a while.

This is the code example I copied:

import asyncio from proxybroker import Broker


async def save(proxies, filename):
    """Save proxies to a file."""
    with open(filename, 'w') as f:
        while True:
            proxy = await proxies.get()
            if proxy is None:
                break
            proto = 'https' if 'HTTPS' in proxy.types else 'http'
            row = '%s://%s:%d\n' % (proto, proxy.host, proxy.port)
            f.write(row)


def main():
    proxies = asyncio.Queue()
    broker = Broker(proxies)
    tasks = asyncio.gather(broker.find(types=['HTTP', 'HTTPS'], limit=10),
                           save(proxies, filename='proxies.txt'))
    loop = asyncio.get_event_loop()
    loop.run_until_complete(tasks)


if __name__ == '__main__':
    main()

When I try to run the code the following error is thrown together with some deprecation warnings:

/home/sebastian/PycharmProjects/STW/venv/lib/python3.5/site-packages/aiohttp/client.py:494:
DeprecationWarning: Use async with instead   warnings.warn("Use async
with instead", DeprecationWarning)
/home/sebastian/PycharmProjects/STW/venv/lib/python3.5/site-packages/aiohttp/client.py:494:
DeprecationWarning: Use async with instead   warnings.warn("Use async
with instead", DeprecationWarning)
/home/sebastian/PycharmProjects/STW/venv/lib/python3.5/site-packages/aiohttp/client.py:494:
DeprecationWarning: Use async with instead   warnings.warn("Use async
with instead", DeprecationWarning)
/home/sebastian/PycharmProjects/STW/venv/lib/python3.5/site-packages/aiohttp/client.py:494:
DeprecationWarning: Use async with instead   warnings.warn("Use async
with instead", DeprecationWarning)
/home/sebastian/PycharmProjects/STW/venv/lib/python3.5/site-packages/aiohttp/client.py:494:
DeprecationWarning: Use async with instead   warnings.warn("Use async
with instead", DeprecationWarning)
/home/sebastian/PycharmProjects/STW/venv/lib/python3.5/site-packages/aiohttp/client.py:494:
DeprecationWarning: Use async with instead   warnings.warn("Use async
with instead", DeprecationWarning)
/home/sebastian/PycharmProjects/STW/venv/lib/python3.5/site-packages/aiohttp/client.py:494:
DeprecationWarning: Use async with instead   warnings.warn("Use async
with instead", DeprecationWarning)
/home/sebastian/PycharmProjects/STW/venv/lib/python3.5/site-packages/aiohttp/client.py:494:
DeprecationWarning: Use async with instead   warnings.warn("Use async
with instead", DeprecationWarning)
/home/sebastian/PycharmProjects/STW/venv/lib/python3.5/site-packages/aiohttp/client.py:494:
DeprecationWarning: Use async with instead   warnings.warn("Use async
with instead", DeprecationWarning)
/home/sebastian/PycharmProjects/STW/venv/lib/python3.5/site-packages/aiohttp/client.py:494:
DeprecationWarning: Use async with instead   warnings.warn("Use async
with instead", DeprecationWarning) https://getproxy.net/en/ is failed.
Error: ClientOSError(101, 'Cannot connect to host getproxy.net:443
ssl:True [Can not connect to getproxy.net:443 [Network is
unreachable]]'); https://getproxy.net/en/ is failed. Error:
ClientOSError(101, 'Cannot connect to host getproxy.net:443 ssl:True
[Can not connect to getproxy.net:443 [Network is unreachable]]');
https://getproxy.net/en/ is failed. Error: ClientOSError(101, 'Cannot
connect to host getproxy.net:443 ssl:True [Can not connect to
getproxy.net:443 [Network is unreachable]]'); Traceback (most recent
call last):   File
"/home/sebastian/PycharmProjects/testing/test/test_prox.py", line 27,
in 
    main()   File "/home/sebastian/PycharmProjects/testing/test/test_prox.py", line 23,
in main
    loop.run_until_complete(tasks)   File "/usr/lib/python3.5/asyncio/base_events.py", line 387, in
run_until_complete
    return future.result()   File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception   File "/usr/lib/python3.5/asyncio/tasks.py", line 241, in _step
    result = coro.throw(exc)   File "/home/sebastian/PycharmProjects/STW/venv/lib/python3.5/site-packages/proxybroker/api.py",
line 108, in find
    await self._run(self._checker.check_judges(), action)   File "/home/sebastian/PycharmProjects/STW/venv/lib/python3.5/site-packages/proxybroker/api.py",
line 114, in _run
    await tasks   File "/usr/lib/python3.5/asyncio/futures.py", line 361, in __iter__
    yield self  # This tells Task to wait for completion.   File "/usr/lib/python3.5/asyncio/tasks.py", line 296, in _wakeup
    future.result()   File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception   File "/usr/lib/python3.5/asyncio/tasks.py", line 241, in _step
    result = coro.throw(exc)   File "/home/sebastian/PycharmProjects/STW/venv/lib/python3.5/site-packages/proxybroker/checker.py",
line 26, in check_judges
    await asyncio.gather(*[j.check() for j in self._judges])   File "/usr/lib/python3.5/asyncio/futures.py", line 361, in __iter__
    yield self  # This tells Task to wait for completion.   File "/usr/lib/python3.5/asyncio/tasks.py", line 296, in _wakeup
    future.result()   File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception   File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)   File "/home/sebastian/PycharmProjects/STW/venv/lib/python3.5/site-packages/proxybroker/judge.py",
line 82, in check
    j=self, code=resp.status, page=page[0], IndexError: string index out of range

I use python3.5.2 and the up to date versions of proxybroker (0.1.4) aiohttp (1.0.2) asyncio (3.4.3).
I'm not sure what causes the error as I did not change the code example and as far as I know I installed all dependencies. Posting the issue on stackoverflow I was asked to create an issue as it seems to be a general problem with proxybroker at the moment.

Hi,
I am having below problem with both latest and stable versions installtion:
error in proxybroker setup command: package_data must be a dictionary mapping package names to lists of wildcard patterns

Screenshot:
http://puu.sh/ygTvv/5800ac1d57.png

Initially I need to check my local proxy list. So I set "http://127.0.0.1:8000/proxy.txt/" in my providers list. But ProxyBroker returns 0 proxies.

Hi, all. I'm interested in distribute vast of requests through a bunch of different proxies rather than the current mode.

The current mode is always pop out the highest priority proxy from the proxy pool. And as partially documented, current mode the Broker.serve will start to grab(w/o checking) at most limit proxies when the server starts. And after that the if the available proxy list is empty, it will resume to find "one working proxy and paused again" (refer to the doc).

I have found self._limit -= 1 but not self._limit += 1 in the project.

So, it's this strategy that makes it hard to maintain say a fixed (100) number of working proxies to be randomly popped when a http/https request arrive in.

It's simple to get a workaround and make it work just for me. But I think it would be better to provide this functionality and integrated into this project.

How do you think about inheriting some child classes from such as Broker, Server, ProxyPool to implement that?

With lists from 2k proxies it sleeps for a long time before to finish even with limit=1.

If you get --lvl High proxies, the country flag is kinda useless, because most of them are backconnect proxies, so, even if you use -c US, the final IP used for the connection will not match.

On my own, i just added a real check using requests and pointing to https://ipinfo.io/IP/json to ensure the effective country.

Regards.

Got an error:

Traceback (most recent call last): File "/usr/local/bin/proxybroker", line 11, in <module> load_entry_point('proxybroker==0.2.0b3', 'console_scripts', 'proxybroker')() File "/usr/local/lib/python3.4/dist-packages/pkg_resources/__init__.py", line 561, in load_entry_point return get_distribution(dist).load_entry_point(group, name) File "/usr/local/lib/python3.4/dist-packages/pkg_resources/__init__.py", line 2631, in load_entry_point return ep.load() File "/usr/local/lib/python3.4/dist-packages/pkg_resources/__init__.py", line 2291, in load return self.resolve() File "/usr/local/lib/python3.4/dist-packages/pkg_resources/__init__.py", line 2297, in resolve module = __import__(self.module_name, fromlist=['__name__'], level=0) File "/usr/local/lib/python3.4/dist-packages/proxybroker/__init__.py", line 28, in <module> from .proxy import Proxy File "/usr/local/lib/python3.4/dist-packages/proxybroker/proxy.py", line 35 async def create(cls, host, *args, **kwargs): ^ SyntaxError: invalid syntax
What might be the reason?

import asyncio
from proxybroker import Broker

async def find(proxies, loop):
    broker = Broker(queue=proxies,
                    # If I comment judges or/and providers, some errors occur at 2nd execution of get_proxies()
                    judges=['https://httpheader.net/', 'http://httpheader.net/'],
                    providers=['http://www.proxylists.net/', 'http://fineproxy.org/eng/'],
                    loop=loop)

    types = [('HTTP', ('High', ))]
    limit = 10

    await broker.find(types=types, limit=limit)


def get_proxies():
    loop = asyncio.get_event_loop()
    proxies = asyncio.Queue(loop=loop)
    loop.run_until_complete(find(proxies, loop))
    proxy_list = []
    while True:
        proxy = proxies.get_nowait()
        if proxy is None:
            break
        p = 'http://{}:{}'.format(proxy.host, proxy.port)
        if p not in proxy_list:
            proxy_list.append(p)
    print(proxy_list)
    return proxy_list

if __name__ == '__main__':
    plist = get_proxies()

    # Get some web pages with proxies. Some proxies might be blocked by web servers after mass connections.
    # Keep a record of blocked proxies here, avoid to use them for several moments.
    # However, if the number of available proxies was declined too much, retrieve some new proxies.

    plist = get_proxies()  # Some errors occur if I comment judges or/and providers.

    # Other tasks ...

If I comment judges/providers, something happens:

D:\Temp\proxybroker\venv\Scripts\python.exe D:/Temp/proxybroker/renew_proxy.py
['http://198.62.109.139:3128', 'http://117.135.251.132:80', 'http://212.224.70.124:8080', 'http://108.59.10.138:55555', 'http://120.26.213.55:9999', 'http://108.59.10.129:55555', 'http://108.59.10.135:55555', 'http://216.220.165.62:8080', 'http://51.254.103.206:3128', 'http://138.36.27.5:80']
Traceback (most recent call last):
File "D:/Temp/proxybroker/renew_proxy.py", line 90, in
plist = get_proxies() # Some errors occur if I comment judges or/and providers.
File "D:/Temp/proxybroker/renew_proxy.py", line 71, in get_proxies
loop.run_until_complete(find(proxies, loop))
File "D:\ProgramFiles\Python35\lib\asyncio\base_events.py", line 337, in run_until_complete
return future.result()
File "D:\ProgramFiles\Python35\lib\asyncio\futures.py", line 274, in result
raise self._exception
File "D:\ProgramFiles\Python35\lib\asyncio\tasks.py", line 241, in _step
result = coro.throw(exc)
File "D:/Temp/proxybroker/renew_proxy.py", line 65, in find
await broker.find(types=types, limit=limit)
File "D:\Temp\proxybroker\venv\lib\site-packages\proxybroker\api.py", line 108, in find
await self._run(self._checker.check_judges(), action)
File "D:\Temp\proxybroker\venv\lib\site-packages\proxybroker\api.py", line 114, in _run
await tasks
File "D:\ProgramFiles\Python35\lib\asyncio\futures.py", line 358, in iter
yield self # This tells Task to wait for completion.
File "D:\ProgramFiles\Python35\lib\asyncio\tasks.py", line 290, in _wakeup
future.result()
File "D:\ProgramFiles\Python35\lib\asyncio\futures.py", line 274, in result
raise self._exception
File "D:\ProgramFiles\Python35\lib\asyncio\tasks.py", line 241, in _step
result = coro.throw(exc)
File "D:\Temp\proxybroker\venv\lib\site-packages\proxybroker\checker.py", line 26, in check_judges
await asyncio.gather(*[j.check() for j in self._judges])
File "D:\ProgramFiles\Python35\lib\asyncio\futures.py", line 358, in iter
yield self # This tells Task to wait for completion.
File "D:\ProgramFiles\Python35\lib\asyncio\tasks.py", line 290, in _wakeup
future.result()
File "D:\ProgramFiles\Python35\lib\asyncio\futures.py", line 274, in result
raise self._exception
File "D:\ProgramFiles\Python35\lib\asyncio\tasks.py", line 239, in _step
result = coro.send(None)
File "D:\Temp\proxybroker\venv\lib\site-packages\proxybroker\judge.py", line 61, in check
allow_redirects=False) as resp:
File "D:\Temp\proxybroker\venv\lib\site-packages\aiohttp\client.py", line 538, in aenter
self._resp = yield from self._coro
File "D:\Temp\proxybroker\venv\lib\site-packages\aiohttp\client.py", line 150, in _request
raise RuntimeError('Session is closed')
RuntimeError: Session is closed

Process finished with exit code 1

im got error while installing.

Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-sAN7dE-build/

I have scratch through the code and find the aiohttp is only used in resolver.py, judge.py and providers.py among the library.

And I'm curious the progress of replacing the Server.serve portion from socket to http/https via aiohttp? Is it aiohttp doesn't support socks that prevent you to use low-level api rather than high-level api?

So, the current Broker.serve code seems to generate proxies once from all the providers.

I mean if we run the server for a long time, say a week, as you know those proxies' lifetime is rather short. The majority or even whole of them will be dead a week later.

This means we can only serve by an uncertain amount of time, maybe one day, or another day.

So, when the Broker._proxies, ProxyPool._pool is both empty, the server is dead.

Any suggestion to add some code to prevent this? Actually, the project structure is a bit hard to get well understood in a short time. Maybe you @constverum , as the author, can supply some hint?

ProxyBroker doesn't works after updating aiohttp to 2.1.0...
OS: macOS Sierra 10.12.4.
Python: 3.5, 3.6.

http://proxyb.net
Site is closed.

C:\Users\Anaconda3\python.exe "C:/Users/PycharmProjects/untitled4/Proxy Checker.py"
C:\Users\Anaconda3\lib\site-packages\aiohttp\client.py:576: DeprecationWarning: Use async with instead
warnings.warn("Use async with instead", DeprecationWarning)
C:\Users\Anaconda3\lib\site-packages\aiohttp\helpers.py:139: DeprecationWarning: ClientSession.close() is a coroutine
warnings.warn(self._msg, DeprecationWarning)

Line 41 error - unexpected key argument -> max_tries

""Run a local proxy server that distributes
incoming requests to external proxies."""

import asyncio
import logging
import aiohttp

from proxybroker import Broker

async def get_pages(urls, proxy_url):
tasks = [
fetch_page(url, aiohttp.ProxyConnector(proxy_url)) for url in urls]
for task in asyncio.as_completed(tasks):
url, content = await task
print('url: %s; content: %.100s' % (url, content))

async def fetch_page(url, conn):
resp = None
try:
with aiohttp.ClientSession(connector=conn) as session:
async with session.get(url) as response:
logger.info('url: %s; status: %d' % (url, response.status))
resp = await response.read()
except (aiohttp.errors.ClientOSError, aiohttp.errors.ClientResponseError,
aiohttp.errors.ServerDisconnectedError) as e:
logger.error('url: %s; error: %r' % (url, e))
finally:
return (url, resp)

def main():
host, port = '127.0.0.1', 8888 # by default

loop = asyncio.get_event_loop()

types = [('HTTP', 'High'), 'HTTPS', 'CONNECT:80']
codes = [200, 301, 302]

broker = Broker(max_tries=1, loop=loop)

# Broker.serve() also supports all arguments that are accepted
# Broker.find() method: data, countries, post, strict, dnsbl.
broker.serve(host=host, port=port, types=types, limit=10, max_tries=3,
             prefer_connect=True, min_req_proxy=5, max_error_rate=0.5,
             max_resp_time=8, http_allowed_codes=codes, backlog=100)

urls = ['http://httpbin.org/get', 'https://httpbin.org/get',
        'http://httpbin.org/redirect/1', 'http://httpbin.org/status/404']

proxy_url = 'http://%s:%d' % (host, port)
loop.run_until_complete(get_pages(urls, proxy_url))

broker.stop()

if name == 'main':
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger('Parser')

main()