Question: LXD Bridge Setup about nebula HOT 9 CLOSED

from nebula.

@bmullan My configuration is similar to the nebula example. However, I realised that there are multiple paths from the LXD container to my laptop node:

• container -> LAN -> WAN -> laptop
• container -> host -> LAN -> WAN -> laptop
• container -> host -> host's nebula -> laptop

I believe the feature #52 whitelist/blacklist (once implemented) will solve the issue.

from nebula.

@nfam did you ever figure out how to connect Nebula's TUN interface to LXD's default lxdbr0 bridge?

from nebula.

Hi maintainers, I solved the issue by buying another (better and more expensive) network extender. So I will close this issue.

@bmullan with the new network extender, it just works for

• both lanbr0 and lxdbr0 up or
• one of those up.

from nebula.

@nfam
Ninh, since Nebula's VPN Tunnel End Points (TEP) is a TUN device how/what did you do to get the Nebula TEP IP address (layer 3) connected to your lanbr0 or lxdbr0 layer 2 bridges ?

from nebula.

Now I get what you mean with the term "connect" in the previous question. No, I didn't, and I think it is impossible to connect TUN to a bridge.
They are separated network interfaces, and Routing Table does its job. That's how nebula works, I suppose.

from nebula.

@nfam

Now I get what you mean with the term "connect" in the previous question. No, I didn't, and I think it is impossible to connect TUN to a bridge.

Yes, that's what I meant you can't connect the nebula1 interface to lxdbr0 since the nebula1 interface is a TUN device (L3)

So when you say:

They are separated network interfaces, and Routing Table does its job. That's how nebula works, I suppose.

Did you add anything manually to the Node (re the LXD Host/Server) Routing Table?

I've got 2 Nodes (Node1 and Node2) and 1 Lighthouse setup.

The Lighthouse is on a cloud instance.

Node1 and Node2 are two different VMs on a local Server (all ubuntu 18.04 including lxd containers)

Both Nodes are running SNAP LXD and each Node has 1 container

Node1 has a container CN1
Node2 has a container CN2

I have everything setup and Nebula seems happy.

From Node1 I can ping the Nebula IP address of Node2 and the Lighthouse
From Node2 I can ping the Nebula IP address of Node1 and the Lighthouse

entering into either LXD container ( CN1 on Node1 or CN2 on Node2)

from "inside" CN1 (on host/vm Node1) I can ping the IP of the Nebula TEP on Node2 ok

note: by "inside" I mean $ lxc exec CN1 bash
then at the bash prompt doing the ping

from "inside" CN2 (on host/vm Node2) I can ping the IP of the Nebula TEP on Node1 ok

But what I cannot do is...

from "inside" CN1 (on host/vm Node1) I can not ping the IP of the Node2 lxdbr0 or the Node2 CN2 container.

or vice-versa...

from "inside" CN2 (on host/vm Node2) I can not ping the IP of the Node1 lxdbr0 or the Node2 CN2 container.

So I am puzzled how your LXD container traffic (or LXDBR0 IPs) is reachable across the Nebula VPN (from lxd Host/Node2 to lxd Host/Node2)

from nebula.

@nfam
I think I'm starting to guess what/how you configured your lab setup.
when you configured Nebula you used the 192.168.16.x/24 network
and from one of your comments:

1 LXD Ubuntu 18.04 server (LXD container) inside the above physical server (nebula IP 192.168.16.3)

You are indicating that your LXD container also has a 192.168.16.x IP address (192.168.16.3).

So I am guessing for your LXD configuration you may have used:

$ lxc network edit lxdbr0

and changed LXDBR0 IP address to a 192.168.16.x IP address also?

from nebula.

@bmullan, The problem you encounter is no route between Node1 LXD and Node2 LXD networks. There are 3 ways to solve this.

1. Make only one LXD overlay network across Node1 and Node2. I do not recommend this for homelab due to multiple steps in order to achieve this (hard and sophisticated). However, on big server cluster with hundreds or thousands of LXD containers, this is the way to go.

<------ LXD ------->
+-------+ +--------+
| Node1 | | Node 2 |
+-------+ +--------+

2. Add routes to routing table on Node1 and Node2, so packet can find the way between Node1 LXD and Node2 LXD networks. For homelab, this is the way to go.

LXD <--> Node1 <--> Node2 <--> LXD

3. Use Nebula unsafe_routes . In short, Nebula manages routing table for you in order to route the packet.

Node1 LXD <--> Nebula Network <--> Node 2 LXD

Note: For (2) and (3), Node 1 LXD and Node2 LXD networks must have different subnets.

From the information you provided, I guess (3) is what you intend to do. See #127. To reach Node1 LXD, you can choose either Node1 or CN1 for intermediate nebula hop (via in the config file), but not both. The same is for Node2 LXD.

from nebula.