kubernetes-sigs / kpng Goto Github PK

View Code? Open in Web Editor NEW

244.0 14.0 91.0 2.74 MB

Reworking kube-proxy's architecture

License: Apache License 2.0

Dockerfile 0.12% Go 90.70% Gnuplot 0.03% Shell 8.28% Makefile 0.39% C 0.44% Starlark 0.04%

k8s-sig-network

kpng's People

Contributors

Stargazers

Watchers

Forkers

yuzhiquan attlee-wang yjuns rikatz sfowl jayunit100 knabben hanlins eth0xfeed nordix dougsland kundan2707 rajaskakodkar mcluseau danwinship blasan cybernetics friedrichwilken nehalohia27 zchee aojea yzaccc pshail anusha94 rexagod sriramandev cyclinder bowei avineshtripathi nikhilsharmawe yankay astoycos purneswarprasad pallavijaini8075 yashsingh74 qabpea shaneutt mattfenwick ekmixon papagalu redhat-et algobot76 hvkk8s jbadru73 kerthcet chetak123 gauravkghildiyal isabella232 aahemm nerdalert tangenti marosset vishnoianil yangjunmyfm192085 asim-reza shekhar-rajak tssurya ksankeerth aryan9600 emabota h3llix loop0 heisaman fedepaol jaehnri aroradaman isgasho hongliangl brunopecampos mansikulkarni96 krouser shivaabhishek07 nilesh-chaudhary abhaikollara gitvinit koba1t adeniyistephen shawnh2 clearnicki uablrek bumblebeek sujay2306 loktev-d zfy3000163 mneverov fzhg tanujd11 hirazawaui slightlycreates

kpng's Issues

Testing on Kind: APIServer proxy not working

We have a new development recipe in the PR #5 .... When i run it, everything comes up however

kpng server wants to access the apiserver through 10.96.0.1
apiserver on 10.96.0.1 is a route that normally is written by... kube proxy
since now kube proxy is running (other then kpng), the route falls down dial tcp 10.96.0.1:443: i/o timeout.

to reproduce, you can directly run the hack/local-up-kpng.sh script (uncomment all 3 functions to reproduce from source

E0419 01:36:27.444110       1 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://10.96.0.1:443/api/v1/nodes?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: i/o timeout

It appears that the kube-proxy rules routing to the APIServer arent working.

If possible would like to use that dev recipe as a starting point, merge it, and then fix whatever the routing issue is.

I suppose maybe its a chicken/egg thing , i.e. the KPNG nftables rules , when setup to access the apiserver through the service IP dont work bc KPNG isnt up yet....

Im not sure how kube-proxy normally solves this -

Testing IPVS with Kpng

@rikatz made a prototype of ipvs https://github.com/rikatz/kpng-ipvs-backend to make us happy :)

Ive filed a few NFT issues, but we havent tested wether these issues persist on IPVS as well.

Lets test kpng with IPVS and see which conformance tests pass.

NFT: Add NodePort Support to KPNG

Looks like NodePorts arent working....

is this NFT specific?

@andrewsykim was asking wether or not the nodeport issue was provider specific, i think it is...
because we have nodeports in the spec

gitpod /workspace/kpng $ git grep "NodePort"
jobs/kube2store/service-event-handler.go:                       NodePort: port.NodePort,
pkg/api/localnetv1/services.pb.go:      NodePort       int32    `protobuf:"varint,4,opt,name=NodePort,proto3" json:"NodePort,omitempty"`
pkg/api/localnetv1/services.pb.go:func (x *PortMapping) GetNodePort() int32 {
pkg/api/localnetv1/services.pb.go:              return x.NodePort
pkg/api/localnetv1/services.proto:    int32    NodePort   = 4;

DETAILS

Ive finished running the conformance results for the hack/ scripts:

The major hole at the moment is ~ nodeport support

It appears that most of the service dependent tests fail.

Summarizing 15 Failures:                            
                                                                              
[Fail] [sig-network] Services [It] should be able to change the type from ExternalName to NodePort [Conformance] 
/home/ubuntu/SOURCE/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/network/service.go:1731
                                                                              
[Fail] [sig-network] Services [It] should have session affinity timeout work for NodePort service [LinuxOnly] [Conformance] 
/home/ubuntu/SOURCE/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/network/service.go:3398

[Fail] [sig-network] Services [It] should be able to switch session affinity for NodePort service [LinuxOnly] [Conformance] 
/home/ubuntu/SOURCE/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/network/service.go:3477

[Fail] [sig-network] Services [It] should be able to create a functioning NodePort service [Conformance] 
/home/ubuntu/SOURCE/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/network/service.go:1179

[Fail] [sig-network] Services [It] should be able to switch session affinity for service with type clusterIP [LinuxOnly] [Conformance] 
/home/ubuntu/SOURCE/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/network/service.go:207

[Fail] [sig-network] DNS [It] should provide DNS for ExternalName services [Conformance] 
/home/ubuntu/SOURCE/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/network/dns_common.go:464

[Fail] [sig-network] Services [It] should have session affinity timeout work for service with type clusterIP [LinuxOnly] [Conformance] 
/home/ubuntu/SOURCE/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/network/service.go:202

[Fail] [sig-network] Services [It] should be able to change the type from NodePort to ExternalName [Conformance] 
/home/ubuntu/SOURCE/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/network/service.go:1817

[Fail] [sig-network] DNS [It] should provide DNS for the cluster  [Conformance] 
/home/ubuntu/SOURCE/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/network/dns_common.go:464

[Fail] [sig-network] DNS [It] should provide /etc/hosts entries for the cluster [LinuxOnly] [Conformance] 
/home/ubuntu/SOURCE/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/network/dns_common.go:464

[Fail] [sig-network] Services [It] should have session affinity work for service with type clusterIP [LinuxOnly] [Conformance] 
/home/ubuntu/SOURCE/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/network/service.go:202

[Fail] [sig-network] DNS [It] should provide DNS for services  [Conformance] 
/home/ubuntu/SOURCE/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/network/dns_common.go:464

[Fail] [sig-network] DNS [It] should resolve DNS of partial qualified names for services [LinuxOnly] [Conformance] 
/home/ubuntu/SOURCE/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/network/dns_common.go:464

[Fail] [sig-network] DNS [It] should provide DNS for pods for Subdomain [Conformance] 
/home/ubuntu/SOURCE/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/network/dns_common.go:464

[Fail] [sig-network] Services [It] should have session affinity work for NodePort service [LinuxOnly] [Conformance] 
/home/ubuntu/SOURCE/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/network/service.go:3477

The results are:

Ran 331 of 5703 Specs in 13437.815 seconds
FAIL! -- 316 Passed | 15 Failed | 0 Pending | 5372 Skipped

Fix iptables compilation in kpng-local-up

mostly working but looks like hitting some 1.23 import errors ...

ubuntu@jay-buildbox-6 [18:20:03] [~/SOURCE/kpng/cmd] [fix-iptables_backend *]
-> % go install -trimpath ./...

# k8s.io/client-go/listers/policy/v1
../../../go/pkg/mod/k8s.io/[email protected]/listers/policy/v1/eviction.go:33:41: undefined: "k8s.io/api/policy/v1".Eviction
../../../go/pkg/mod/k8s.io/[email protected]/listers/policy/v1/eviction.go:50:65: undefined: "k8s.io/api/policy/v1".Eviction
../../../go/pkg/mod/k8s.io/[email protected]/listers/policy/v1/eviction.go:52:25: undefined: "k8s.io/api/policy/v1".Eviction
../../../go/pkg/mod/k8s.io/[email protected]/listers/policy/v1/eviction.go:67:41: undefined: "k8s.io/api/policy/v1".Eviction
../../../go/pkg/mod/k8s.io/[email protected]/listers/policy/v1/eviction.go:70:21: undefined: "k8s.io/api/policy/v1".Eviction
../../../go/pkg/mod/k8s.io/[email protected]/listers/policy/v1/eviction.go:82:73: undefined: "k8s.io/api/policy/v1".Eviction
../../../go/pkg/mod/k8s.io/[email protected]/listers/policy/v1/eviction.go:84:25: undefined: "k8s.io/api/policy/v1".Eviction
../../../go/pkg/mod/k8s.io/[email protected]/listers/policy/v1/eviction.go:90:53: undefined: "k8s.io/api/policy/v1".Eviction
../../../go/pkg/mod/k8s.io/[email protected]/listers/policy/v1/eviction.go:98:15: undefined: "k8s.io/api/policy/v1".Eviction
# k8s.io/client-go/applyconfigurations/meta/v1
../../../go/pkg/mod/k8s.io/[email protected]/applyconfigurations/meta/v1/unstructured.go:52:13: undefined: managedfields.GvkParser
../../../go/pkg/mod/k8s.io/[email protected]/applyconfigurations/meta/v1/unstructured.go:58:61: undefined: managedfields.GvkParser
../../../go/pkg/mod/k8s.io/[email protected]/applyconfigurations/meta/v1/unstructured.go:69:9: undefined: managedfields.NewGVKParser
../../../go/pkg/mod/k8s.io/[email protected]/applyconfigurations/meta/v1/unstructured.go:128:33: too many arguments in call to managedfields.ExtractInto
        have (*unstructured.Unstructured, typed.ParseableType, string, *unstructured.Unstructured, string)
        want (runtime.Object, typed.ParseableType, string, interface {})

GH Actions tests (but lets keep mob testing also)

imo testing every friday as a group is a good routine ---
but we should have some bot tests just to make @aojea happy :) also

UPDATE

To get started, go into every file w a go.mod and run

GOOS=linux go build -o kpng ./kpng

in GH Actions
@friedrichwilken

Reach parity with upstream IPTables externalIPAddresses implementation

Looks like in backends/iptables/service.go we dont have any logic for externalIPAddresses implemented?

// filter external ips, source ranges and ingress ips
	// prior to dual stack services, this was considered an error, but with dual stack
	// services, this is actually expected. Hence we downgraded from reporting by events
	// to just log lines with high verbosity

	ipFamilyMap := MapIPsByIPFamily(service.IPs.ExternalIPs)
	info.externalIPs = ipFamilyMap[sct.ipFamily]

rename server->bind

need to rename the --server flag - --bind since the normal
way we test is by binding to a local socket.

make a cool website like kind has

lets steal this !
https://kind.sigs.k8s.io/docs/user/quick-start/

windows userspace proxy porting to kpng

Tracking issue to get the windows userspace proxy working in KP{NG

Compiling KPNG on Windows

$ cd cmd/
$ GOOS=windows go build ./kpng/...

Raises the following error:

go build github.com/google/seesaw/ipvs: build constraints exclude all Go files in /home/knabben/go/pkg/mod/github.com/google/[email protected]/ipvs
# github.com/vishvananda/netns
../../../../pkg/mod/github.com/vishvananda/[email protected]/netns.go:28:13: undefined: unix.Stat_t
../../../../pkg/mod/github.com/vishvananda/[email protected]/netns.go:29:12: undefined: unix.Fstat
../../../../pkg/mod/github.com/vishvananda/[email protected]/netns.go:32:12: undefined: unix.Fstat
../../../../pkg/mod/github.com/vishvananda/[email protected]/netns.go:43:8: undefined: unix.Stat_t
../../../../pkg/mod/github.com/vishvananda/[email protected]/netns.go:44:12: undefined: unix.Fstat
../../../../pkg/mod/github.com/vishvananda/[email protected]/netns.go:56:8: undefined: unix.Stat_t
../../../../pkg/mod/github.com/vishvananda/[email protected]/netns.go:57:12: undefined: unix.Fstat
../../../../pkg/mod/github.com/vishvananda/[email protected]/netns.go:71:12: undefined: unix.Close
# sigs.k8s.io/kpng/server/pkg/server
../server/pkg/server/server.go:42:15: undefined: syscall.Umask
../server/pkg/server/server.go:43:26: undefined: syscall.Umask

We currently have some working going on with comparing old iptables/ipvs/... rules w/ those generated by KPNG.
As part of this, what we need is a way to generate table driven tests for various kube proxy permutations:

Host Port Services
NodePort local enabled / disabled
Loadbalancer services
ClusterIP Services
NodePort services
endpointSlices on / off

ideally if we could have a daemonset which was running, and which then polled every type of service from all nodes, we could generate a profile - which told us which types of services did/didn't work...

                               HostPort Service.        NodePort local          LoadbalancerService.    NodePort Service
pod-on-node1                 .1s
pod-on-node2                 .2s
pod-on-node3                 .2s
pod-on-node4                 .2

I think Doug is going to start prototyping such a test in bash, and then from there, we'll build it out in golang after testing it and generating a non-trivial set of verifications.

alternate testing environment from KIND

We want a alternate way to test , w/o using Kind , because of issues where nft/iptables/ipvs might not be installed properly, and or/ other kernel issues that are unpredictable.

cloud recipe (GCE/EC2/...)
Vagrantfile
other ideas ?

model service affinity

looks like we need to model service Affinity explicitly.
right now its not an attribute in the kpng services.

but generically, we do have docs on this:

doc/service-proxy.md

thanks to @danwinship recent pr...

IPv6 does not work

On a dual-stack K8s cluster with base family IPv4 services using IPv6 does not work, examples;

apiVersion: v1
kind: Service
metadata:
  name: mserver-ipv6
spec:
  ipFamilies:
    - IPv6
  selector:
    app: mserver-deployment
  ports:
  - port: 5001
    name: mconnect
  - port: 5003
    name: ctraffic
---
apiVersion: v1
kind: Service
metadata:
  name: mserver-preferdual
spec:
  ipFamilyPolicy: PreferDualStack
  selector:
    app: mserver-deployment
  ports:
  - port: 5001
    name: mconnect
  - port: 5003
    name: ctraffic

Neither of these works. The mserver-preferdual have dual-stack addresses;

kubectl get svc mserver-preferdual -o json | jq .spec.clusterIPs
[
  "12.0.135.134",
  "fd00:4000::7133"
]

but not even the IPv4 address works.

On a dual-stack K8s cluster with base family IPv6 no services works.

Backend ipvs only works on the local node

When using the ipvs backend there is no masquerading when the call is routed to another node. So only calls to the local node works from the main netns on the node. There must be some rule that uses the KUBE-MARK-MASQ chain;

# iptables -t nat -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N KUBE-KUBELET-CANARY
-N KUBE-MARK-DROP
-N KUBE-MARK-MASQ
-N KUBE-POSTROUTING
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING -s 11.0.0.0/16 ! -d 11.0.0.0/16 -j MASQUERADE
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-POSTROUTING -m mark ! --mark 0x4000/0x4000 -j RETURN
-A KUBE-POSTROUTING -j MARK --set-xmark 0x4000/0x0
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -j MASQUERADE --random-fully

Tracker: conformance testing of kpng: iptables backend

Need an iptables sink backend for kpng

Now we have the nft sink backend which should be the preferred sink backend for Linux platform. However, sometimes nft is not as stable as iptables, and could have byte-level and bit-level endian mismatch (pointed by @mcluseau). Also, in some situations, the system could have iptables-legacy in used, which creates iptables rules that are not backed and recognized by nft.

Given those realities, an iptables backend is still needed. We can use the existing iptables rules as a baseline, and improve that by making the iptables rules implementation moduler, and easier to reason and review.

Once we have a iptables backend, we could add a "meta-backend" namely to-best-fit, which chooses the backend that work best for the system. If the underlying Linux distribution/version supports nft and doesn't use legacy iptables or have conflicting iptables rules, then it will choose nft for users, otherwise it can fallback to this iptables backend.

Diff Store improvements

0ae3c56 added a function to make up for calculating the appended/removed state of nodeport services.

We need to improve the diffstore to support that functionality generically.

a self-awareness "interface" for NFT pod

The NFT tables pod is susceptible to naughty bhaviour when iptables is installed.

To remediate this, and also, to create a precendent for other service proxy backends, lets:

add a "SystsemCheck()" or similar implementatino in the NFT backend
Have it verify that all iptables rules (i.e. see the ones in #12) are empty
if iptables rules non-empty print continous warnings that are extremely idiot-proof , i.e.
warning: ip routing may not work properly due to mutliple kernel firewalls !

This will make it really easy for folks struggling w/ the incompatibilities (iptables, legacy iptables, and nft, and so on) to know why loadbalancing rules written by NFT arent getting honored.

kpng backends should implement session affinity and load balancer specific rules.

This issue is for tracking kpng backends support for session affinity and load balancer specific parameters.

Tracker: conformance testing of kpng: nft backend

Tracker: conformance testing of kpng: ipvs backend

Kpng can not be built with static linking

Used to work, but now fails with;

$ CGO_ENABLED=0 GOOS=linux go build -ldflags "-extldflags '-static' -X main.version=$(date +%F:%T)" ./cmd/kpng
go build github.com/google/seesaw/ipvs: build constraints exclude all Go files in /home/uablrek/go/pkg/mod/github.com/google/[email protected]/ipvs

I suppose the dependency to github.com/google/seesaw/ipvs is the problem. Without digging too deep, can github.com/google/seesaw/ipvs be built to allow static linking for dependent programs?

And BTW, static build is more or less required for a binary release. Otherwise the binary will have dependencies to the libs (and version of libs) on the build host.

upstream NodePort tests fail but ours (service-loadbalancer) tests pass... why?

@hanamantagoudvk

noted that upstream hairpin tests fail, but in general, ours pass:

127 <nil>
observed:

-		x-52621/pod-1	x-52621/pod-2	x-52621/pod-3
x-52621/pod-1	.		.		.	
x-52621/pod-2	.		.		.	
x-52621/pod-3	.		.		.

Lets see what the diff is .... follow on to #30 (if its a red herring related to upstream e2e we can close 30, and close this as well i guess. )

implement ipvs SystsemCheck()

In #13 we mentioned that we should check if NFT is correctly setup and
defined an SystemCheck().

Lets do the same for IPVS.

Impl details:

if this fails:

/ # ipvsadm --help
modprobe: can't load module ip_vs (kernel/net/netfilter/ipvs/ip_vs.ko.xz): Operation not permitted
Can't initialize ipvs: Protocol not available
Are you sure that IP Virtual Server is built in the kernel or as module?

Then we definetly are NOT up and running. maybe check the exit code , to confirm (assume ipvsadm help returns 0 normally).

To install ipvsadm:

yum install -y ipvsadm
modprobe
# i.e. this is equivalent to running 'modprobe ip_vs'

local-up-kpng: large clusters on kind?

Tried an initial take at 20 nodes, on a centos box, and got

 stuck at the boot screen message [OK]Started Update UTMP about System Runlevel Changes.

Not sure why, just a note, we have a shared dev box that uses centos stream

ExternalIPs always empty

The ExternalIPs is always empty even when a service have externalIPs;

$ kubectl get svc
NAME          TYPE           CLUSTER-IP   EXTERNAL-IP        PORT(S)                         AGE
kubernetes    ClusterIP      12.0.0.1     <none>             443/TCP                         72s
mserver       LoadBalancer   12.0.63.60   10.0.0.1,1000::1   5001:31093/TCP,5003:30150/TCP   43s
mserver-udp   LoadBalancer   12.0.207.6   10.0.0.2,1000::2   5001:31084/UDP                  43s
$  print-state 
I0824 09:12:48.195326    2759 client.go:168] connecting to 127.0.0.1:12090
# ------------------------------------------------------------------------
# 2021-08-24 09:12:48.210194493 +0000 UTC m=+0.024634303
#
&{Namespace:"default"  Name:"kubernetes"  Type:"ClusterIP"  IPs:{ClusterIPs:{V4:"12.0.0.1"}  ExternalIPs:{}}  Ports:{Name:"https"  Protocol:TCP  Port:443  TargetPort:6443} [IPs:{V4:"192.168.1.1"}]}
&{Namespace:"default"  Name:"mserver"  Type:"LoadBalancer"  IPs:{ClusterIPs:{V4:"12.0.63.60"}  ExternalIPs:{}}  Ports:{Name:"mconnect"  Protocol:TCP  Port:5001  NodePort:31093  TargetPort:5001}  Ports:{Name:"ctraffic"  Protocol:TCP  Port:5003  NodePort:30150  TargetPort:5003} [IPs:{V4:"11.0.2.2"}  Local:true IPs:{V4:"11.0.3.2"} IPs:{V4:"11.0.1.2"} IPs:{V4:"11.0.4.2"}]}
&{Namespace:"default"  Name:"mserver-udp"  Type:"LoadBalancer"  IPs:{ClusterIPs:{V4:"12.0.207.6"}  ExternalIPs:{}}  Ports:{Name:"mconnect-udp"  Protocol:UDP  Port:5001  NodePort:31084  TargetPort:5001} [IPs:{V4:"11.0.2.2"}  Local:true IPs:{V4:"11.0.3.2"} IPs:{V4:"11.0.1.2"} IPs:{V4:"11.0.4.2"}]}

So, regardless of backend kpng can't support external access.

Makefile, multistage builds?

We'll need to make a makefile that supports:

building
publishing containers to a registry
building at least some subset of binaries
setup modd (use modd under the hood)

per @uablrek lets keep it simple :) NO GRADLE and no https://taskfile.dev/#/

Fixing loopbackIPSet

This is covered in @hanamantagoudvk 's new PR #79

Blog plost / Article: "next generation proxying with KPNG"

This would be a good kpng demo for a newcomer to the project to writeup/publish.

define configs for separate machines (vagrantfile)
Start one kpng frontend
create 2 services w/ names a,b
start one kpng backend to focus on service-proxy-name=a, (nft / ipvs)
start a second kpng backend to focus on service-proxy-name=b, (iptables)
confirm that the backends work properly

Endpoints are missed when creating many services fast

I create many pods and services in one command by apply a directory;

kubectl apply -f /etc/kubernetes/k8s-test/

Most services has 4 endpoints in my setup but when accessing only a few is used;

vm-003 ~ # mconnect -address mserver-deployment.default.svc.xcluster:5001 -nconn 100
Failed connects; 0
Failed reads; 0
mserver-deployment-654bbdc6dc-xj2fk 100
vm-003 ~ # mconnect -address mserver.default.svc.xcluster:5001 -nconn 100
Failed connects; 0
Failed reads; 0
mserver-mmd8f 52
mserver-sjrls 48

Both services has 4 endpoints. I waited for ~30 min to see if sync would fix this but the problem remains.

Output from print-state: print-state.txt

Experiment: swap out kpng kind recipes w/ Vagrantfile and Ubuntu?

that way the nft and other issues related iptables/ipvs kernel versinoing can be isolated from
the core issues we have in other parts of KPNG

Right now, to run Kind, we need to ensure the HOST has the right iptables-legacy configuration, and that
is pretty tricky to do !

remove udp

note this is more of a perf / security issue for large scale clusters so its not super high priority right now

This is probably a decent getting started issue.... The existing kube proxy does some cleanup of Conntrack UDP tables..

	for _, svcIP := range conntrackCleanupServiceIPs.UnsortedList() {
		if err := conntrack.ClearEntriesForIP(proxier.exec, svcIP, v1.ProtocolUDP); err != nil {
			klog.ErrorS(err, "Failed to delete stale service connections", "ip", svcIP)
		}

We should implement this, the same way, in KPNG.

My question here would be - how do we test this ? ... i think upstream there arent good tests for this but im not sure ....

IPVS remotePort and destPort are mixed up !

Thanks to @tibordp for helping us find this today !!!!

Running @rikatz IPVS we find that the CNI pods dont come up, bc routes to the APIServer are not working over IPVS.

the root cause was that

we needed to install ipvs so were going to make a UX improvement there #36
the REAL root cause was that LocalAddress:Port

/ # ipvsadm
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.96.0.1:6443 rr
  -> kpng-proxy-control-plane.kin Masq    1      0          0

ExternalLoadBalancer IPs support

originally some confusion around #60 ...

IMO from todays meeting we should

(most important) We also should have a datamodel

type ServiceStatus struct {
	LoadBalancer LoadBalancerStatus `json:"loadBalancer,omitempty" protobuf:"bytes,1,opt,name=loadBalancer"`
}
type LoadBalancerStatus struct {
	Ingress []LoadBalancerIngress `json:"ingress,omitempty" protobuf:"bytes,1,rep,name=ingress"`
}

(less important) Namespace legacy concepts like ExternalIP in kpng if we use them, i.e. K8sExternalIP

	ExternalIPs []string `json:"externalIPs,omitempty" protobuf:"bytes,5,rep,name=externalIPs"`

(even less important??) need to figure out how to designate / should we support should make sure we have really good support in place for LoadBalancer IPs,

	LoadBalancerIP string `json:"loadBalancerIP,omitempty" protobuf:"bytes,8,opt,name=loadBalancerIP"`

Cut a 1.21 "release" and move on to 1.22 OR make 1.22 backewards compatible....

Newer kube proxy code bases will be using the new endpoints api :

kubernetes/kubernetes@03b7a69

for example this:

                   endpointSliceConfig := config.NewEndpointSliceConfig(informerFactory.Discovery().V1().EndpointSlices(), s.ConfigSyncPeriod)

instead of this:

		endpointSliceConfig := config.NewEndpointSliceConfig(informerFactory.Discovery().V1beta1().EndpointSlices(), s.ConfigSyncPeriod)

That will mean modifying imports like this:

	discovery "k8s.io/api/discovery/v1beta1"
        discoveryinformers "k8s.io/client-go/informers/discovery/v1beta1"

to this

       discovery "k8s.io/api/discovery/v1"

This might i think break the ability to run kpng for 1.22 clusters with older 1.21 backend.... so to deal with this

we can cut a 1.21 < compatible release and move forward with 1.22
we could have 2 watches, one of which fails and one which succeeds, so that its version agnostic
we could also put an if statement, and check on startup <-- this is idea probably?

split up services.proto

services.proto has the local and global data models in it
if we split it into two itll be easeier to understand

Services with label service.kubernetes.io/service-proxy-name excluded

To use a kpng based proxier for a special purpose while the regular kube-proxy is used for "normal" traffic you must use the service.kubernetes.io/service-proxy-name label, for example;

apiVersion: v1
kind: Service
metadata:
  name: sctp-test
  labels:
    service.kubernetes.io/service-proxy-name: nordix-sctp
spec:
  ipFamilyPolicy: PreferDualStack
  selector:
    app: mserver
  ports:
  - port: 6000
    name: sctpt
    protocol: SCTP

This will make the kube-proxy to ignore the service and a specialized proxier can subscribe for services with the desired "service-proxy-name" and handle only those.

But kpng is hardcoded to ignore all services having the service.kubernetes.io/service-proxy-name label;

kpng/jobs/kube2store/kube2store.go

Lines 117 to 122 in 0c0a3c8

 func (j Job) getLabelSelector() labels.Selector { 

 noProxyName, err := labels.NewRequirement(LabelServiceProxyName, selection.DoesNotExist, nil) 

 if err != nil { 

 klog.Exit(err) 

 }

There must be a configuration option for this.

Verify Hairpin proxy scenarios

e2e tests that are failing related to hairpin connectivty
one node cluster tests fail consistently, for some of the e2e service endpoint tests

@hanamantagoudvk is digging into this

NFT container fail if NFT not enabled properly

@hanamantagoud dig into https://developers.redhat.com/blog/2020/08/18/iptables-the-two-variants-and-their-relationship-with-nftables or ricardo katz' blog post, you might be able to find the commands needed for nft to validate that it can run....

Find a couple of system exec calls we can run to validate that nft can work (i.e. check iptables-legacy, check iptables-version)....
Write some golang glue in the nft pod that causes it to log error and fail if it cannot run so that we dont have a 'false positive' proxy that is running

Investigate why DNS related Conformance tests fail

Ive found that DNS tests fail consistently on Kind KPNG recipes. Ive tried a few things to fix it:

using calico as the CNI instead of Kind-net
scaling the # of coredns pods up (in case the error was somehow related to loopback services)

Neither seemed to fix anything. Is there something particularly special about DNS resolution that might be effecting KPNG or nftables?

The DNS Tests which fail conformance are:

DNS resolution
DNS for services
DNS for ExternalName Services

These are all pretty easy to repro, see the first issue where i reported the full suite of 16 failures: #15

AntreaProxy as a backend

With https://github.com/antrea-io/antrea/releases/tag/v1.4.0 now available, antrea can entirely replace the kube proxy.

So, i think we can now implement a KPNG backend in the antrea project as another example. This would be a cool project to collaborate with the antrea community on .

Failing K8s [sig-network] e2e tests

This is a more of a report than an issue. Please feel free to close this issue if there is a better place for reporting test outcomes.

I have discussed on slack that the Conformance suite is not optimal for testing kpng. It takes a long time and tests many non-network things. Many relevant [sig-network] tests are also not in the Conformance suite.

I test with;

kubetest --provider=skeleton --ginkgo-parallel --test --test_args="--num-nodes=4 --ginkgo.focus=\[sig-network\].*ervice --ginkgo.skip=Disruptive|affinity|DNS|GCE|finalizer"

At the time of writing this selects 69 tests of which 63 can be executed with kpng (the remaining are public-cloud and iptables specific). I must use --num-nodes in my env (which is not KinD) but you will probably not need that.

Summary;

Ran 63 of 5768 Specs in 542.795 seconds
FAIL! -- 30 Passed | 33 Failed | 0 Pending | 5705 Skipped
Ginkgo ran 1 suite in 9m11.427761139s
Test Suite Failed

Here is the full output k8s-e2e.txt.

Tests were executed on;

commit 95df644fd8750210c91bd6c938ca74ce768cbac0 (HEAD -> master, origin/master, origin/HEAD)
Author: Mikaël Cluseau <[email protected]>
Date:   Fri May 7 22:54:44 2021 +0200

    api2store: fix tls config build

userspace proxy examples to support removal from core

xref kubernetes/kubernetes#103860

since were removing userspace proxying from core, lets implement an example in KPNG, so that we can
show folks how to use them.

i.e. lets implement the userspace-windows backend :)...

Kpng should have a version command

As development proceeds and test reports and issues are created is becomes crucial to know the kpng version that is tested. For now all kpng users are building their own version and can provide a git commit in the report/issue. But soon users can take a binary version (I assume).

kpng version

Should be added and print a version (git tag) for releases and a commit for "home-built" versions.

Backend standalone dev instructions

Right now a backend engineer can test kpng w the kping-loca-up script but this requires a full cluster (in kind) and isnt super good for real time editing of go backends + testing the dataplane rules.

we can use kpng local --unix... combined with kpng file --to-api, to make a simple, hackable dev env on a linux box that simply writes iptables rules over and over based on file changes, no k8s api required.

Lets make a README_BACKEND_DEV.md file that summarizes how to do this (say for ipvs, or iptables, or nft) , and shows how to use

modd
kpng to file
kpng to api

to do real time backend hacking

More granular backend sync interfaces

In general in KPNG we use a sink.go as the interface between KPNG and underlying backends...

However, to use this, we sometimes have to play w/ data structures on the backends to implement the coarse grained SetEndpoint function. This works pretty well but it means we need to sometimes modify underlying functions in backends that we copy over (i.e. from k8s core)... as an example, we had to change

166 | func (cache *EndpointSliceCache) updatePending(endpointSlice *discovery.EndpointSlice, remove bool) bool {

func (cache *EndpointsCache) updatePending(svcKey types.NamespacedName, key string, endpoint *localnetv1.Endpoint) bool {

when moving iptables impl of upstream Kube-proxy into kpng....

how we currently do it

func (s *Backend) SetEndpoint(namespace, serviceName, key string, endpoint *localnetv1.Endpoint) {

how we should do it

It would be nice if we could have finer grained interfaces like

func (s *Backend) AddNewServiceIPPort(namespace, serviceName, key string, ip string, int port, protocol string ) {
...

Then we could implement a lot more of the logic for the backends that we bring over, generically inside of the sink.go implementations.

make kpng importable using gomodules !: out-of-tree compilation of backends doesnt work

@uablrek mentioned that he couldnt externalize the compilation of his backend:

go: finding module for package sigs.k8s.io/kpng/client
go: found sigs.k8s.io/kpng/client in sigs.k8s.io/kpng v0.0.0-20210813170222-0c0a3c81012c
go: sigs.k8s.io/[email protected] requires

Is this bc we're vendoring the kubernetes core bits ? ...

        k8s.io/[email protected] requires
        k8s.io/[email protected]: reading k8s.io/api/go.mod at revision v0.0.0: unknown revision v0.0.0

$ go build ./main.go 
main.go:24:2: no required module provides package sigs.k8s.io/kpng/client; to add it:
        go get sigs.k8s.io/kpng/client
$ go get sigs.k8s.io/kpng/client
go get: sigs.k8s.io/kpng@none updating to
        sigs.k8s.io/[email protected] requires
        k8s.io/[email protected] requires
        k8s.io/[email protected]: reading k8s.io/api/go.mod at revision v0.0.0: unknown revision v0.0.0

not sure what to do about this - im thinking maybe the issue is that, we vendor the k8s "non-vendorable" bits, and thats whats burning us.

legacy kube-proxy: port over fix for affinity endpoints with incoming fix for iptables proxy

https://github.com/kubernetes/kubernetes/pull/106030/files

also need to do this in the service-lb tests filed here K8sbykeshed/k8s-service-validator#35

hack/kind recipe: Coredns cant resolve 10.96.0.1

Very odd bug we saw w the kind local up recipe:

A normal coredns container cant access the apiserver
A curl image can access it

Through the 10.96.0.1 service endpoint

E0502 13:40:32.731430       1 reflector.go:178] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125: Failed to list *v1.Endpoints: Get "https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: i/o timeout
E0502 13:40:32.731437       1 reflector.go:178] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125: Failed to list *v1.Namespace: Get "https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: i/o timeout
I0502 13:40:32.731627       1 trace.go:116] Trace[1225511528]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125 (started: 2021-05-02 13:40:02.730276654 +0000 UTC m=+0.080721947) (total time: 30.001276262s):
Trace[1225511528]: [30.001276262s] [30.001276262s] END
E0502 13:40:32.731797       1 reflector.go:178] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125: Failed to list *v1.Service: Get "https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: i/o timeout
[INFO] plugin/ready: Still waiting on: "kubernetes"

However a regular curl image on the same endpoint works fine from 10.96.0.1.

hypothesis

the endpoint from 10.96.0.1 -> pod is somehow not written properly to coredns pods ?

clearly, its correctly set, and its accessible through either the Service endpoint OR the pod endpoint--- ...

jayunit100-hyper@jayunit100-hyp:~/SOURCE/kpng/hack$ kubectl get endpoints 
NAME         ENDPOINTS         AGE
kubernetes   172.18.0.4:6443   19m

Ultimately the coredns endpoints dont ever get created:

% kubectl get endpoints -n kube-system
NAME       ENDPOINTS   AGE
kube-dns               7m30s

	func (j Job) getLabelSelector() labels.Selector {
	noProxyName, err := labels.NewRequirement(LabelServiceProxyName, selection.DoesNotExist, nil)
	if err != nil {
	klog.Exit(err)
	}