Not getting service maps about coroot HOT 10 CLOSED

I reproduced the problem on a k3s+cilium cluster. The bug is that node-agent expected all connections to be present in the conntrack table. In the case of Cilium, the destination of a TCP connection has been overwritten at the eBPF level, bypassing conntrack.
This is fixed in coroot-node-agent 1.0.21

from coroot.

Hey @rlex!

Is your Loki under the load? I mean, there must be active connections between components to display them on the service map.
Can you please check the following metrics in your Prometheus?

• rate(container_net_tcp_successful_connects_total{container_id=~".+loki-distributor.+"}[1m])
• container_net_tcp_active_connections{container_id=~".+loki-distributor.+”}

To reduce the number of apps displayed on the overview page, we intentionally hide apps that have no active inbound or outbound connections. Otherwise, many auxiliary services such as sshd, udevd, etc. would be shown there.

from coroot.

Yes, there is constant traffic flow. Some parts are getting picked up, but in a strange way:

Interesting that loki-index-gateway is designated as "external endpoint" while it's part of k8s service-cidr.

As for metrics, they are empty. Only two pods appears under .loki. regex

from coroot.

I deployed it on usual kubernetes and looks like everything is working. But it also have calico as a cni with kube-proxy.

In my setup, it's cilium with ebpf dataplane and DSR enabled. DSR shouldn't affect internal cluster networking, though.

from coroot.

It looks like the issue is related to k3s, not CNI. Is there any way to reproduce your k3s+Loki setup? I'd look into it.

from coroot.

It's fairly typical setup, but i deploy coredns and CNI manually. I also deploy hetzner cloud-controller.

I deploy it with my ansible role

from coroot.

@rlex , it seems like we fixed this issue.
Could you check this by running coroot:0.3.0 + coroot-node-agent:1.0.20 on your k3s cluster?

from coroot.

Sadly no, no luck. k3s is now consided part of control-plane which makes map less cluttered, but there is still no proper links between majority of services, and service-cidr is still considered "external endpoint".
So in my case:

1. pod CIDR is 10.121.0.0/24
2. service CIDR is 10.43.0.0/16

And sometimes both IPs in pod cidr and service cidr is listed as "external endpoint"

But only links i saw in several days are those:

If you have some VM for tests i can try to create almost 1:1 config for debugging.

from coroot.

Can confirm it works now. Thanks alot for great product and great communication!

from coroot.

Thank you for the detailed bug report!

from coroot.