Getting stuck at streamConn about go-tproxy HOT 15 CLOSED

The core problem with asymmetric routing is that if the upstream router has a direct connection to the client PC, rather than sending response packets to the TProxy server it will send it directly to the client PC.

For TProxy to work, all network traffic has to flow through it so the upstream router can't send it packets directly to the client.

There are two ways to accomplish this:

1. Use two networks so that the upstream router is on one and the client is on another, the TProxy server is setup to be the router for the client's network (this is how the vagrant example works)

2. Setup the TProxy server with two network interfaces, one attached to the network that hosts the router, the other directly connected to the client PC. From there, setup a bridge with the two interface and configure the TProxy rule on the bride interface.

from go-tproxy.

Are you able to provide some sample code that you are working with?

Also what operating system is the code running on (e.g. Ubuntu 16.04, etc)?

from go-tproxy.

I am currently exactly the example code that you provide.
I am running it on my server Ubuntu 16.04 and 1 more PC desktop also running Ubuntu 16.04
Do you need some more information

from go-tproxy.

On the ubuntu server, is IP forwarding enabled along with the appropriate IPTables rules for TProxy?

from go-tproxy.

Yes I think I have all the iptables rules added correctly. I try to keep it as correctly as the example without modifying anything, even my binding port.
My iptables rules are:

iptables -t mangle -N DIVERT
iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
iptables -t mangle -A DIVERT -j MARK --set-mark 1
iptables -t mangle -A DIVERT -j ACCEPT
iptables -t mangle -A PREROUTING -p tcp --dport 80 -j TPROXY --tproxy-mark 0x1/0x1 --on-port 8080

I am a little uncertain if I have to modify any of these to fit my environment:

ip rule add fwmark 1 lookup 100
ip route add local 0.0.0.0/0 dev lo table 100

In addition to your instruction. I also added:

sudo sysctl -w net.ipv4.ip_forward=1
sudo sysctl -w net.ipv6.conf.all.forwarding=1

from go-tproxy.

In addition, as the the example iptables rules above, I think it only forward port 80 right?
I tested on my browser, any HTTPS url can still work correctly.
Currently only HTTP websites on port 80 are not working.
If I add forward port 443 in the iptables, then I HTTPS websites are not working anymore

from go-tproxy.

I put some log printing in the example and it seems like it got stuck in the streamConn (copy 0 bytes and nil error)

Do you know which stream this was that had a 0 byte copy, was it conn->remoteConn or remoteConn->conn?

from go-tproxy.

I have modified this part in "handleTCPConn" in the example:

streamConn := func(name string, dst io.Writer, src io.Reader) {
	n, err := io.Copy(dst, src)
	fmt.Println("Streaming", name, n, err)
	streamWait.Done()
}

go streamConn("remote to conn", remoteConn, conn)
go streamConn("conn to remote", conn, remoteConn)

And the result is the following. The websites that I tested on is still live and good.

Streaming remote to conn 0 readfrom tcp 10.3.4.151:60554: write tcp 10.3.4.151:60554: write: connection timed out
Streaming conn to remote 0 <nil>
Streaming remote to conn 0 readfrom tcp 10.3.4.151:60551: write tcp 10.3.4.151:60551: write: connection timed out
Streaming conn to remote 0 <nil>
Streaming remote to conn 0 readfrom tcp 10.3.4.151:60552: write tcp 10.3.4.151:60552: write: connection timed out
Streaming conn to remote 0 <nil>
Streaming remote to conn 0 readfrom tcp 10.3.4.151:60555: write tcp 10.3.4.151:60555: write: connection timed out
Streaming conn to remote 0 <nil>

from go-tproxy.

Awesome, that helps a lot more, one thing to note though about the streamConn function, the first argument is the destination and the second is the source, so that error is while streaming data from the client to the remote.

That being said, it feels like there may be asymmetric routing happening on your network, so the data is leaving TProxy with the address of your client machine but when it is coming back from the remote host your router is sending it directly to your client rather than sending it to the TProxy server.

Do you possibly have a diagram of the network layout for this test?

from go-tproxy.

Awesome, that helps a lot more, one thing to note though about the streamConn function, the first argument is the destination and the second is the source, so that error is while streaming data from the client to the remote.

So do I have to switch remoteConn and conn? And the result should be like this? I changed like this and it is still not working:

go streamConn("remote to conn", conn, remoteConn)
go streamConn("conn to remote", remoteConn, conn)

result:

Streaming conn to remote 0 readfrom tcp 10.3.4.151:60667: write tcp 10.3.4.151:60667: write: connection timed out
Streaming remote to conn 0 <nil>

from go-tproxy.

Changing the order won't change much. again I think it may be asymmetric routing, are you able to do a packet capture on something further upstream to see if a response is being sent from remote?

from go-tproxy.

That being said, it feels like there may be asymmetric routing happening on your network, so the data is leaving TProxy with the address of your client machine but when it is coming back from the remote host your router is sending it directly to your client rather than sending it to the TProxy server.

Do you possibly have a diagram of the network layout for this test?

I'm not sure what you mean because I'm not very specialized in network.
I have 2 test environments: laptop -> server and laptop -> PC

• I'm use my laptop as client, set default gateway to my server. Both are in the same subnet and IP Range. Laptop is Macbook and Server is Ubuntu 16.04
• I'm use my laptop as client, set default gateway to my PC. Both are in the same subnet and IP Range. Laptop is Macbook and PC is Ubuntu 16.04

from go-tproxy.

@phuongvietvu0306 Have you had progress in getting this to work? I would like to close this issue if it is no longer valid

from go-tproxy.

Closing as stale

from go-tproxy.

can we add a client as another vagrant virtual machine, to the extent of the example code? So there is no setup required for the initial setup?

from go-tproxy.