fmoo / twisted-connect-proxy Goto Github PK

View Code? Open in Web Editor NEW

51.0 8.0 22.0 12 KB

Default Twisted does not ship with a CONNECT-enabled HTTP(s) proxy. This code provides one.

License: Other

Python 100.00%

twisted-connect-proxy's Introduction

twisted-connect-proxy

Default Twisted does not ship with a CONNECT-enabled HTTP(s) proxy. This code provides one.

This code also provides an HTTP CONNECT proxy client that implements IReactorTCP and IReactorSSL

Proxy Server

To run an HTTP CONNECT proxy server on port 8080, run:

./server.py

That was easy.

Proxy Client

The HTTP CONNECT proxy reactor can be used like this:

proxy = HTTPProxyConnector(proxy_host, proxy_port)

twisted-connect-proxy's People

Contributors

Stargazers

Watchers

Forkers

lvh nhanb deese akashsharma95 icersong bollustrado aoden2 dbunskoek twang-rs oefenweb dc3l1ne foxpass unzhanun ibonxalopa sekogan vazkir proudduck chenyilu outiy5 iq-scm

twisted-connect-proxy's Issues

Only working for twisted version > 18

I tested with both python2 and python3, on both macOS and linux, and I happended to find that for twisted version under 18 (e.g., 17.5.0), the connect proxy doesn't work (without any error shown though).
Thought I should report this.

Unit tests

Unittests. How do they work?!

Import connector logic into repository

from the gist at https://gist.github.com/2018384

HTTPS simply doesn't work.

Hello.

When I do these, it works (except for (http://)google.com -- it is giving me HTTP 301):
curl -x localhost:8080 http://example.com
curl -x localhost:8080 example.com

But when I do this, it simply doesn't work and I put the error text below it:
curl -x localhost:8080 https://example.com
curl: (56) Proxy CONNECT aborted

What am I doing wrong? I run the server.py directly. My twisted version is 22.10.0
Here is what I see in the terminal as soon as I do the request:

	Traceback (most recent call last):
	  File "C:\Users\useruser\Desktop\nvs\venv\lib\site-packages\twisted\python\log.py", line 96, in callWithLogger
	    return callWithContext({"system": lp}, func, *args, **kw)
	  File "C:\Users\useruser\Desktop\nvs\venv\lib\site-packages\twisted\python\log.py", line 80, in callWithContext
	    return context.call({ILogContext: newCtx}, func, *args, **kw)
	  File "C:\Users\useruser\Desktop\nvs\venv\lib\site-packages\twisted\python\context.py", line 117, in callWithContext
	    return self.currentContext().callWithContext(ctx, func, *args, **kw)
	  File "C:\Users\useruser\Desktop\nvs\venv\lib\site-packages\twisted\python\context.py", line 82, in callWithContext
	    return func(*args, **kw)
	--- <exception caught here> ---
	  File "C:\Users\useruser\Desktop\nvs\venv\lib\site-packages\twisted\internet\selectreactor.py", line 148, in _doReadOrWrite
	    why = getattr(selectable, method)()
	  File "C:\Users\useruser\Desktop\nvs\venv\lib\site-packages\twisted\internet\tcp.py", line 248, in doRead
	    return self._dataReceived(data)
	  File "C:\Users\useruser\Desktop\nvs\venv\lib\site-packages\twisted\internet\tcp.py", line 253, in _dataReceived
	    rval = self.protocol.dataReceived(data)
	  File "C:\Users\useruser\Desktop\nvs\forward_proxy.py", line 90, in dataReceived
	    Proxy.dataReceived(self, data)
	  File "C:\Users\useruser\Desktop\nvs\venv\lib\site-packages\twisted\web\http.py", line 2465, in dataReceived
	    return basic.LineReceiver.dataReceived(self, data)
	  File "C:\Users\useruser\Desktop\nvs\venv\lib\site-packages\twisted\protocols\basic.py", line 549, in dataReceived
	    why = self.lineReceived(line)
	  File "C:\Users\useruser\Desktop\nvs\venv\lib\site-packages\twisted\web\http.py", line 2319, in lineReceived
	    self.allContentReceived()
	  File "C:\Users\useruser\Desktop\nvs\venv\lib\site-packages\twisted\web\http.py", line 2447, in allContentReceived
	    req.requestReceived(command, path, version)
	  File "C:\Users\useruser\Desktop\nvs\venv\lib\site-packages\twisted\web\http.py", line 1032, in requestReceived
	    self.process()
	  File "C:\Users\useruser\Desktop\nvs\forward_proxy.py", line 29, in process
	    ProxyRequest.process(self)
	  File "C:\Users\useruser\Desktop\nvs\venv\lib\site-packages\twisted\web\proxy.py", line 137, in process
	    port = self.ports[protocol]
	builtins.KeyError: b'example.com'``

AttributeError: 'module' object has no attribute 'SSL_ST_INIT'

I don't know if this is something you can fix on your side, but we get this error in Ubuntu 16.04.

Traceback (most recent call last):
  File "/usr/local/bin/twisted-connect-proxy", line 8, in <module>
    from twisted.web.proxy import Proxy, ProxyRequest
  File "/usr/lib/python2.7/dist-packages/twisted/web/proxy.py", line 24, in <module>
    from twisted.internet import reactor
  File "/usr/lib/python2.7/dist-packages/twisted/internet/reactor.py", line 38, in <module>
    from twisted.internet import default
  File "/usr/lib/python2.7/dist-packages/twisted/internet/default.py", line 56, in <module>
    install = _getInstallFunction(platform)
  File "/usr/lib/python2.7/dist-packages/twisted/internet/default.py", line 44, in _getInstallFunction
    from twisted.internet.epollreactor import install
  File "/usr/lib/python2.7/dist-packages/twisted/internet/epollreactor.py", line 24, in <module>
    from twisted.internet import posixbase
  File "/usr/lib/python2.7/dist-packages/twisted/internet/posixbase.py", line 18, in <module>
    from twisted.internet import error, udp, tcp
  File "/usr/lib/python2.7/dist-packages/twisted/internet/tcp.py", line 29, in <module>
    from twisted.internet._newtls import (
  File "/usr/lib/python2.7/dist-packages/twisted/internet/_newtls.py", line 21, in <module>
    from twisted.protocols.tls import TLSMemoryBIOFactory, TLSMemoryBIOProtocol
  File "/usr/lib/python2.7/dist-packages/twisted/protocols/tls.py", line 41, in <module>
    from OpenSSL.SSL import Error, ZeroReturnError, WantReadError
  File "/usr/lib/python2.7/dist-packages/OpenSSL/__init__.py", line 8, in <module>
    from OpenSSL import rand, crypto, SSL
  File "/usr/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 118, in <module>
    SSL_ST_INIT = _lib.SSL_ST_INIT
AttributeError: 'module' object has no attribute 'SSL_ST_INIT'

Upgrading pyopenssl (pip install 'pyopenssl>=16.2.0') fixes the problem, but I rather not do that (system wide).

code urlparse.urlparse(self.uri) has problem in twisted-13.10(windows x64)

When I set the proxy with server.py and browse https://www.google.com, it outputs the 501 error.
I find the problem is causing by the below

    # when self.uri = www.google.com:443, 
    # parsed is ParseResult(scheme='www.google.com', netloc='', path='443', params='', query='', fragment='')
    parsed = urlparse.urlparse(self.uri)

And I change the code to below, it runs without any problem.

    uri = self.uri if self.uri.startswith('https://') else 'https://' + self.uri
    parsed = urlparse.urlparse(uri)

Make this an importable package

setup.py, license, etc etc.

SOCKS support

Out of scope for this project, so may need a rename.

Useful though.

SSH proxy support

Either bundle with SOCKS and rely on it, or configure port forwards via appropriate negotiations on the fly.

Track Data after successful CONNECT

In HTTPS when connection has established, Is there any method (here or in twisted library) to track the content of each request and its response? I have sucessfully done it for HTTP but when an HTTPS website is connected the data is not passing through that mehtod (ProxyClient.handleReponsePart() in twisted/web/proxy.py) but same method is not handling the HTTPS connections.

Any hint or help will be appreciated.

Broken in Twisted 16.3.0

I've been debugging this for a bit, but alas I'm just not advanced enough in Twisted to figure out the problem. This code is broken in twisted-16.3.0. I've bisected it down to this specific commit:

twisted/twisted@2d39cca

Which relates to HTTP/2 support in this issue:

https://twistedmatrix.com/trac/ticket/8320

The symptoms are basically that after receiving the CONNECT from the client; the proxy does send the 200 CONNECT OK; but, the client never continues to send to GET request.

The problem is somewhere in the server code, as using the below curl command works for twisted-16.2.0, but does not for twisted-16.3.0.

curl -x http://127.0.0.1:8080 https://www.google.com/robots.txt

The output when running client.py is:

2017-02-09 01:38:25-0600 [-] Log opened.
2017-02-09 01:38:25-0600 [-] Starting factory <__main__.HTTPProxiedClientFactory instance at 0x109922c68>
2017-02-09 01:38:25-0600 [HTTPConnectTunneler,client] 78 bytes for proxy None
2017-02-09 01:38:25-0600 [HTTPConnectTunneler,client] Got Status :: 200 CONNECT OK HTTP/1.0
2017-02-09 01:38:25-0600 [HTTPConnectTunneler,client] Got Header :: X-Connected-Ip: 216.58.218.132
2017-02-09 01:38:25-0600 [HTTPConnectTunneler,client] Got Header :: Content-Length: 0
2017-02-09 01:38:25-0600 [HTTPConnectTunneler,client] End Headers
2017-02-09 01:38:35-0600 [HTTPConnectTunneler,client] HTTPConnectTunneler connectionLost [Failure instance: Traceback (failu
re with no frames): <class 'twisted.internet.error.ConnectionDone'>: Connection was closed cleanly.
        ]
2017-02-09 01:38:35-0600 [HTTPConnectTunneler,client] Got Response ::
2017-02-09 01:38:35-0600 [-] Stopping factory <__main__.HTTPProxiedClientFactory instance at 0x109922c68>