`kube-proxy`'s `--healthz-bind-address` should support IPv4 and IPv6 simultaneously (dual stack) about kubernetes HOT 24 OPEN

The wanted feature here seems to be to accept connects on lo only. A good way is to bind the listen socket to the device using the SO_BINDTODEVICE sockopt. Now, I don't propose this as a solution in this case, but the function may come in handy in some other cases, so I figured out how it's done in go.

// Tcp server that binds to "lo"
package main

import (
	"context"
	"log"
	"net"
	"os"
	"syscall"
)

func main() {
	if len(os.Args) < 2 {
		log.Fatal("No address")
	}
	lconfig := net.ListenConfig{
		Control: listenControl,
	}
	listener, err := lconfig.Listen(context.TODO(), "tcp", os.Args[1])
	if err != nil {
		log.Fatal(err)
	}
	defer listener.Close()
	conn, err := listener.Accept()
	if err != nil {
		log.Fatal(err)
	}
	conn.Write([]byte("Bye...\n"))
	defer conn.Close()
}
func listenControl(network, address string, c syscall.RawConn) error {
	return c.Control(setIf)
}
func setIf(fd uintptr) {
	if err := syscall.BindToDevice(int(fd), "lo"); err != nil {
		log.Fatal(err)
	}
}

./tcp-server :7000
nc -v 127.4.5.6 7000  # works
nc -v ::1 7000        # works
nc -v 172.17.0.1 7000 # Connection refused

As you can see, any address on lo works, but not local address on another interface (172.17.0.1 is on docker0)

from kubernetes.

Um, come to think of it, a device may be better than two addresses here. Unless the addresses are "any" or loopback, they must be different on all nodes. While this can be done, a common configmap makes it hard. An option might be something like:

healthzBindAddress: "[::]:10256"
healthzBindInterface: "lo"

kubernetes/enhancements#4337 (comment)

from kubernetes.

See also:

1. #88458
2. #108737
3. #123559

from kubernetes.

/sig network

from kubernetes.

I think that's doubtful, 0.0.0.0 and :: also bind to both families in kube-apiserver

How does kube-apiserver work for headlthz-bind-address? I think if it defaults to "0.0.0.0:10256", we can access to the service with ipv4 or ipv6 address.

when a server is told to listen on 0.0.0.0 that means "listen on every available network interface".

from kubernetes.

Just FYI, I tested to "scope" ipv6 localhost with [::%lo]:10256. It is accepted by go (but not by kube-proxy), but alas, the scope is simply ignored for non-link-local addresses 😞

from kubernetes.

@sanmai-NL we are planning to add v1alpha2 kube-proxy configuration which will accept a dual-stack CIDR pair of healthz and metrics addresses.

(we are moving towards CIDR to allow users to provide the network instead of just address, healthz and metrics server will be exposed on nodesIPs which belong to that CIDR range)
(ref: https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/784-kube-proxy-component-config#following-fields-will-be-changed)

from kubernetes.

Um, come to think of it, a device may be better than two addresses here. Unless the addresses are "any" or loopback, they must be different on all nodes. While this can be done, a common configmap makes it hard. An option might be something like:

healthzBindAddress: "[::]:10256"
healthzBindInterface: "lo"

from kubernetes.

Thanks for the pointer. Actually I think an interface may be better than a CIDR. Interfaces are more stable. It may be easier to guess the interface than the CIDR some infra-structure provider will use for nodes.

from kubernetes.

Thanks for the pointer. Actually I think an interface may be better than a CIDR. Interfaces are more stable. It may be easier to guess the interface than the CIDR some infra-structure provider will use for nodes.

But people can assign non-loopback addresses to a loopback interface and this would break.

from kubernetes.

A network interface can be named variously depending on node OS config or hot-swapping NICs. They are less stable than, e.g., a pair of loopback CIDR blocks ::1/128,127.0.0.1/8.

from kubernetes.

And CIDRs can change because of DHCP lease, or a new /64 segment for SLAAC. But, I agree with Dan that CIDRs are a more "more-kube-proxy-like approach".

from kubernetes.

And CIDRs can change because of DHCP lease

IPs will change because of the DHCP lease not CIDR, right?
CIDR will be configured with the DHCP server itself, no?

from kubernetes.

@uablrek For the use case I mentioned, CIDR subnets are stable. I think working with devices and supporting zone indexes in IPv6 addresses is interesting and potentially useful, but the requested functionality shouldn't be designed around that.

from kubernetes.

@shaneutt: The label(s) triage/accept cannot be applied, because the repository doesn't have them.

from kubernetes.

/assign @danwinship

from kubernetes.

/triage accepted

from kubernetes.

I feel like we have danced around this area a bunch of times. Can we clarify what the real requirment is and implement it one time that we can use everywhere?

E.g. write a k/utils library that offers "multi-listen" and takes a list of IPs and/or CIDRs, finds all matching local IPs, opens sockets on all of them and presents a facade of a single socket.

from kubernetes.

The documentation suggests that the bind address is singular and that it's either of the IPv4 or IPv6 family. I think that's doubtful, 0.0.0.0 and :: also bind to both families in kube-apiserver.

The documentation is correct, just misleading. If you leave the value unset, then kube-proxy behaves as though you had said either --healthz-bind-address 0.0.0.0:10256 or --healthz-bind-address [::]:10256, depending on whether the cluster is ipv4-primary or ipv6-primary. On Linux, this is sort of pointless because they both have the same effect, but on Windows it actually only binds to one family.

from kubernetes.

I'm not sure anyone really cares a lot about listening on both 127.0.0.1 and ::. You can always listen on just 127.0.0.1, even in a single-stack IPv6 cluster, because you don't need any external IPv4 routing to use 127.0.0.1.

In discussion on the v1alpha2 stuff, we decided the use case for overriding this is "I only want to serve health/metrics on the IP that faces the load balancer / prometheus / whatever". To match the way --nodeport-addresses works, we decided to let the user specify a CIDR, and have kube-proxy pick a local IP from in that CIDR. That way you can specify a single config for all your nodes.

from kubernetes.

we decided to let the user specify a CIDR, and have kube-proxy pick a local IP from in that CIDR

That pattern makes sense to me, but is a single family sufficient? IOW, does --healthz-bind-address=10.9.8.0/24,2601:8675::3099/64 need to work?

from kubernetes.

That pattern makes sense to me, but is a single family sufficient? IOW, does --healthz-bind-address=10.9.8.0/24,2601:8675::3099/64 need to work?

All of the following are valid combinations

--health-bind-address=10.9.8.0/24,2601:8675::3099/64
--health-bind-address=10.9.8.0/24
--health-bind-address=2601:8675::3099/64

but IIRC the plan is to allow users to configure it via v1alpha2 config, not via command line flags.

from kubernetes.

Do we have a socketmux Go package already, or is thst something we'll invent?

from kubernetes.

Do we have a socketmux Go package already, or is thst something we'll invent?

We can do that, we will need to listen on multiple IPs (technically 0.0.0.0/0 is a valid CIDR). That might be handy.

Also would love to have your feedback on #121830 (comment)

from kubernetes.