How do you compile?

% ls                                                                                                                ~/g/src/github.com/miekg/metallb master + deb
AUTHORS       CONTRIBUTING.md  dockerfiles/  fabfile.py  glide.yaml  LICENSE     README.md
bgp-speaker/  controller/      docs/         glide.lock  internal/   manifests/  test-bgp-router/
% cd controller                                                                                                     ~/g/src/github.com/miekg/metallb master + deb
% ls                                                                                                     ~/g/src/github.com/miekg/metallb/controller master + deb
main.go  service.go
% go b                                                                                                   ~/g/src/github.com/miekg/metallb/controller master + deb
main.go:23:2: use of internal package not allowed
main.go:24:2: use of internal package not allowed
main.go:25:2: use of internal package not allowed
%                                                                                                        ~/g/src/github.com/miekg/metallb/controller master + deb

Also to use go.universe.tf don't you need to add a canonical import path?
package pdf // import "rsc.io/pdf"

Test coverage is pretty terrible right now, because this codebase grew out of a prototype.

Need to add more unit tests for individual packages (note: use of heavyweight k8s test frameworks should be limited to the internal/k8s package, to avoid contaminating the rest of the codebase with k8s's icky frameworkiness).

Maybe it's just me but my minikube default config is to use 10.96.0.0/12 as a service cidr so the 10.0.0.x addresses in the tutorial manifests don't work out of the box.

Is this a bug report or a feature request?:
Bug

What happened:
Trying to set up the dev chain using fabric I was unable to push the images.
I found a workaround that requires:

• Adding an entry to the insecure registries in docker for mac preferences
• Change the tags for the docker images (replacing localhost for docker.for.mac.localhost)

See docker/for-mac#1160 (comment) for details.

What you expected to happen:

Being able to push images using fab push out of the box

How to reproduce it (as minimally and precisely as possible):

Clone and run fab start and then fab push on a Mac

Similar to #22, but for Cisco. The hard part with Cisco is that they do not provide free trial versions of the virtual IOS appliance. You have to buy a license for $$$. Alternatively, GNS3 has an IOS emulator that can run normal, non-virtual IOS images inside an emulated router environment. To use that emulator, you just need to own any IOS-using equipement, which allows you to download the IOS firmware images.

Bottom line, Cisco might be much harder/impossible to get working in a way that many people can test with, but we should explore it more completely and document how to do it, or document why it's not possible.

Juniper offers "trial" versions of the vSRX and vMX virtual routers. I think in this case trial version means the router will only run for a limited time before shutting down. That's not a deal-breaker for test environments! And it would be great to have interop testing with Juniper stacks.

One challenge is figuring out how to run the virtual appliances in minikube. They're heavy VM images, not containers, so we may have to wait for Kubevirt to work reliably in Minikube, or have separate documentation on how to set up a hybrid test environment with minikube + other VMs.

Is this a bug report or a feature request?:

Bug

What happened:

Now that the arp and BGP speakers are merge, a ton of documentation needs updating. Notably, the documentation and its manifests are probably all wrong.

Is this a bug report or a feature request?:

bug

What happened:

{"log":"/home/miek/g/src/go.universe.tf/metallb/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:175\n","stream":"stderr","time":"2017-12-14T21:51:19.205839696Z"}
{"log":"/home/miek/g/src/go.universe.tf/metallb/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:143\n","stream":"stderr","time":"2017-12-14T21:51:19.205899539Z"}
{"log":"/home/miek/g/src/go.universe.tf/metallb/arp-speaker/main.go:204\n","stream":"stderr","time":"2017-12-14T21:51:19.205958289Z"}
{"log":"/home/miek/upstream/go/src/runtime/asm_arm.s:971\n","stream":"stderr","time":"2017-12-14T21:51:19.206014955Z"}
{"log":"I1214 21:51:19.204827       1 leader.go:31] Lost leadership\n","stream":"stderr","time":"2017-12-14T21:51:19.206071725Z"}
{"log":"panic: runtime error: invalid memory address or nil pointer dereference [recovered]\n","stream":"stderr","time":"2017-12-14T21:51:19.207697751Z"}
{"log":"\u0009panic: runtime error: invalid memory address or nil pointer dereference\n","stream":"stderr","time":"2017-12-14T21:51:19.207999623Z"}
{"log":"[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x8d48b0]\n","stream":"stderr","time":"2017-12-14T21:51:19.208521597Z"}
{"log":"\n","stream":"stderr","time":"2017-12-14T21:51:19.208590295Z"}
{"log":"goroutine 50 [running]:\n","stream":"stderr","time":"2017-12-14T21:51:19.20911581Z"}
{"log":"go.universe.tf/metallb/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)\n","stream":"stderr","time":"2017-12-14T21:51:19.209617368Z"}
{"log":"\u0009/home/miek/g/src/go.universe.tf/metallb/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0xfc\n","stream":"stderr","time":"2017-12-14T21:51:19.210387882Z"}
{"log":"panic(0x9a51c0, 0x10ec8e8)\n","stream":"stderr","time":"2017-12-14T21:51:19.210857877Z"}
{"log":"\u0009/home/miek/upstream/go/src/runtime/panic.go:491 +0x204\n","stream":"stderr","time":"2017-12-14T21:51:19.211375945Z"}
{"log":"go.universe.tf/metallb/vendor/k8s.io/client-go/tools/leaderelection/resourcelock.(*EndpointsLock).RecordEvent(0x11f78640, 0xa75c1d, 0xd)\n","stream":"stderr","time":"2017-12-14T21:51:19.211865211Z"}
{"log":"\u0009/home/miek/g/src/go.universe.tf/metallb/vendor/k8s.io/client-go/tools/leaderelection/resourcelock/endpointslock.go:92 +0xfc\n","stream":"stderr","time":"2017-12-14T21:51:19.212389113Z"}
{"log":"go.universe.tf/metallb/vendor/k8s.io/client-go/tools/leaderelection.(*LeaderElector).acquire.func1()\n","stream":"stderr","time":"2017-12-14T21:51:19.21270937Z"}
{"log":"\u0009/home/miek/g/src/go.universe.tf/metallb/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:183 +0x98\n","stream":"stderr","time":"2017-12-14T21:51:19.213238271Z"}
{"log":"go.universe.tf/metallb/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0x123b5fb8)\n","stream":"stderr","time":"2017-12-14T21:51:19.213644205Z"}
{"log":"\u0009/home/miek/g/src/go.universe.tf/metallb/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x4c\n","stream":"stderr","time":"2017-12-14T21:51:19.214318938Z"}
{"log":"go.universe.tf/metallb/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x123b5fb8, 0x3b9aca00, 0x0, 0x33333333, 0x3ff33333, 0x12268001, 0x11f48000)\n","stream":"stderr","time":"2017-12-14T21:51:19.215461115Z"}
{"log":"\u0009/home/miek/g/src/go.universe.tf/metallb/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0x90\n","stream":"stderr","time":"2017-12-14T21:51:19.215834601Z"}
{"log":"go.universe.tf/metallb/vendor/k8s.io/client-go/tools/leaderelection.(*LeaderElector).acquire(0x12391000)\n","stream":"stderr","time":"2017-12-14T21:51:19.216152254Z"}
{"log":"\u0009/home/miek/g/src/go.universe.tf/metallb/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:175 +0xac\n","stream":"stderr","time":"2017-12-14T21:51:19.21667777Z"}
{"log":"go.universe.tf/metallb/vendor/k8s.io/client-go/tools/leaderelection.(*LeaderElector).Run(0x12391000)\n","stream":"stderr","time":"2017-12-14T21:51:19.216996829Z"}
{"log":"\u0009/home/miek/g/src/go.universe.tf/metallb/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:143 +0x40\n","stream":"stderr","time":"2017-12-14T21:51:19.217517918Z"}
{"log":"main.main.func1(0x12391000)\n","stream":"stderr","time":"2017-12-14T21:51:19.217822498Z"}
{"log":"\u0009/home/miek/g/src/go.universe.tf/metallb/arp-speaker/main.go:204 +0x1c\n","stream":"stderr","time":"2017-12-14T21:51:19.21833817Z"}
{"log":"created by main.main\n","stream":"stderr","time":"2017-12-14T21:51:19.218640251Z"}
{"log":"\u0009/home/miek/g/src/go.universe.tf/metallb/arp-speaker/main.go:204 +0x388\n","stream":"stderr","time":"2017-12-14T21:51:19.219161913Z"}

I'm not sure if I am holding it wrong or that is just doesn't work on ARM.

The

{"log":"I1214 21:51:19.204827       1 leader.go:31] Lost leadership\n","stream":"stderr","time":"2017-12-14T21:51:19.206071725Z"}

Indicates the callback is called just fine.

TODO list for integrating arp-speaker properly:

• implement master election so that it can be used as a daemonset, see https://godoc.org/k8s.io/client-go/tools/leaderelection
• remove the go-routines from the main loop and just do a for loop
• send unsolicated ARPs on startup and mastership change
• ...
• add/fix manifiest and docker stuff for this new binary

We want a better "test while coding" setup than "compile everything by hand, run it with handcrafted configs, and hope you got everything right." For this, I think the answer is going to be Vagrant and Draft. Roughly:

• Make Vagrant start a 4-machine constellation: 2 machines form a small k8s cluster, 1 runs BIRD with all supported protocols, 1 runs Quagga with all supported protocols.
• Vagrant deploys MetalLB by default, with a configuration that fully peers it with Bird+Quagga, so we can test against 2 implementations.
• Build Draft packs and Helm configs for each MetalLB component, so that you can do fast iteration on a real cluster. e.g. recompile bgp-speaker, Draft automatically pushes the updated binary into the Vagrant k8s cluster.

The vagrant setup needs to be dual stack, for v4+v6 support, and probably needs to enable a bunch of k8s alpha features (at the very least the ipv6 support in k8s 1.9). Ideally, we should be able to repurpose the Vagrant setup as an integration test harness as well.

Kubernetes 1.8/1.9 support admission webhooks. MetalLB should register an admission webhook for its configuration, so that it can early-reject invalid and unsafe configurations.

We could also have the webhook register itself for all service creations/changes, so that we can validate the shape of a requested Service before even writing it to the cluster.

Is this a bug report or a feature request?:

bug

What happened:
CGO_ENABLED=0 GOOS=linux GOARCH=arm go build -o build/linux/arm/arp-speaker

go.universe.tf/metallb/vendor/github.com/mdlayher/raw

../vendor/github.com/mdlayher/raw/timeval.go:17:6: cannot use int64(timeout / time.Second) (type int64) as type int32 in field value
../vendor/github.com/mdlayher/raw/timeval.go:18:7: cannot use int64(timeout % time.Second / time.Microsecond) (type int64) as type int32 in field value
Makefile:22: recipe for target 'arp-speaker' failed

What you expected to happen:

clean compile

How to reproduce it (as minimally and precisely as possible):

See docker branch and execute 'make build'

Anything else we need to know?:

Currently MetalLB suffers from the same load imbalance problems as GCP when using externalTrafficPolicy: Local, because BGP ends up load-balancing by node, regardless of the pod count on each node.

One possible solution is to implement BGP add-path (RFC 7911) in MetalLB's BGP speaker, and make speakers advertise one distinct path for each pod running on a given node. This would effectively push weight information upstream, and compatible routers should therefore weight their traffic distribution.

Annoyingly, the spec doesn't make it clear that routers are expected to translate multiple distinct paths with the same next-hop as a weighted assignment in their FIB. It would make sense naively, but it may not be what people implementing the spec had in mind in terms of use cases. So, we'll have to implement a prototype and see how BIRD and others behave, and if they behave sensibly, we can productionize the change.

During the original prototype, I wrote a more detailed design doc that goes over how MetalLB works and why. I need to pull that out of Google Docs and into the website.

Is this a bug report or a feature request?:

bug

What happened:

 ---> a1b40cd21f4b
Step 2/6 : MAINTAINER Miek Gieben <[email protected]> @miekg
 ---> Running in b5b184feaff6
 ---> d18c593354ba
Removing intermediate container b5b184feaff6
Step 3/6 : ADD arp-speaker /arp-speaker
 ---> 2af6e2dd2774
Step 4/6 : RUN setcap 'cap_net_bind_service=+ep' /arp-speaker
 ---> Running in 7691f2b8cac0
standard_init_linux.go:185: exec user process caused "exec format error"
The command '/bin/sh -c setcap 'cap_net_bind_service=+ep' /arp-speaker' returned a non-zero code: 1
Error response from daemon: No such image: arp-speaker:latest

For this to work we need setcap for each arch; or something else

But do we need setcap at all? Does it make sense in a Dockerfile?

Possibly at the same time that we merge it with bgp-speaker, rather than spend effort writing separate manifests. TBD.

Is this a bug report or a feature request?:

feature request

What happened:

After the speakers got merged in #76 we just use 2 k8s clients, this is quite exspensive; we should just use one. In the words of @danderson:

TODO future cleanup, we can probably have a single k8s client and a single leader elector for all protocols, and do the fanout in this code.

Each k8s client maintains its own cache of all the objects, and a separate connection to the k8s apiserver, so currently this code is consuming 2x the ram it needs, and making 2x the number of necessary connection (scaling concern).

Again, no need to change this right now, I'm just braindumping things we should do later.

arp-speaker needs NET_RAW. Manifest and binary both need the capability added.

Setcap is tricky, because we're building multi-arch binaries and setcap might not work on some of the output binaries. Need to investigate further.

Is this a bug report or a feature request?:

Feature request

What happened:

I wanted to send an email.

What you expected to happen:

I expected an email address I could send email to.

How to reproduce it (as minimally and precisely as possible):

Try to send an email to MetalLB users.

The screenshots are a couple of revisions old. They only show 1 test router, the UI's changed a bit, etc.

Just need a couple of gnome-screenshot -i to update the images.

Is that fabfile.py thing the replacement for make in this project?
i.e. If I send a PR with arp-speaker build + docker foo in a Makefile will that be accepted?

We need to add @miekg and @xnaveira to the AUTHORS, they both have changes submitted to the repository.

Is this a bug report or a feature request?:

Bug

What happened:

I embarrassingly discovered that metallb 0.2.0 is broken. The manifests as configured cannot start either controller or bgp-speaker, because they require runAsNonRoot, and apparently k8s cannot verify that "nobody" is non-root because it's not a numeric UID.

What you expected to happen:

MetalLB 0.2.0 should work :(

How to reproduce it (as minimally and precisely as possible):

Try to apply the MetalLB manifests from the website.

Since we are integrating metallb in our clusters I thought I would share our use case since I think it is a bit special, we're open for inputs and suggestions!

We run k8s in bare metal servers with two network cards, those network cards are configured with L2 ips that are used to establish a BGP session with the switch they are connected to.

Through those two BGP sessions we announce a third ip with /32 mask which effectively is the host ip and gets announced to the rest of the network through the BGP daemon that runs on the switches.

We first thought to peer the bgp-speaker instances with the switches but that added the problem of figuring out the addresses of the switches where the host was connected to. We thought of peering them with the upper layer which have the same address for everyone but that would add complexity to our network, which is used for other than k8s. In the end we came up with the idea of peering the bgp-speaker instances with the bird instance that we already are running in the hosts, this way we can peer with localhost and the services' routes get propagated, and so it is what we are doing now.

We are still figuring out the details and we might need to change how we handle the configs slightly but so far so good.

Is this a bug report or a feature request?: Feature

What happened: I noticed that metallb has gained an ARP speaker, which is great for IPv4, however we'll need a NDP speaker for IPv6 once #7 is implemented.

This is not an issue so much as a lightweight way of gathering information on who is using MetalLB. This is mostly to satisfy our curiosity, but might also help us decide how to evolve the project.

So, if you use MetalLB for something, please chime in here and tell us more!

My poor router only does RIP.

Adding RIP support might be a nice addition.

As I should brush off my Pis, this seems like a nice enhancement to work, esp, for teaching me some internals.

Is this a bug report or a feature request?:

Feature request

What happened:

Right now if you're connecting to a misconfigured BGP peer, metallb retries the connection every 2s.

Most BGP implementations implement error backoff, where if the session errors out, they will blindly refuse any new connection attempts for N seconds, to avoid overloading their flimsy CPUs with broken BGP sessions.

This means that if you try connecting metallb to a misconfigured peer, the logs will show the "real" error once, followed by 1-3 minutes of "reading from peer: EOF" errors as metallb keeps retrying.

What you expected to happen:

The logs should do a better job of highlighting the real error.

Deploying the controller and the speaker on CoreOS servers running kubernetes the dockers never get to "Running" state. Examining the logs:

panic: standard_init_linux.go:178: exec user process caused "operation not permitted" [recovered]
	panic: standard_init_linux.go:178: exec user process caused "operation not permitted"

goroutine 1 [running, locked to thread]:
panic(0x7eb2e0, 0xc820142fc0)
	/usr/lib/go1.6/src/runtime/panic.go:481 +0x3e6
github.com/urfave/cli.HandleAction.func1(0xc8200b92f8)
	/build/amd64-usr/var/tmp/portage/app-emulation/runc-1.0.0_rc2_p9/work/runc-1.0.0_rc2_p9/Godeps/_workspace/src/github.com/urfave/cli/app.go:478 +0x38e
panic(0x7eb2e0, 0xc820142fc0)
	/usr/lib/go1.6/src/runtime/panic.go:443 +0x4e9
github.com/opencontainers/runc/libcontainer.(*LinuxFactory).StartInitialization.func1(0xc8200b8c08, 0xc820022098, 0xc8200b8d18)
	/build/amd64-usr/var/tmp/portage/app-emulation/runc-1.0.0_rc2_p9/work/runc-1.0.0_rc2_p9/Godeps/_workspace/src/github.com/opencontainers/runc/libcontainer/factory_linux.go:259 +0x136
github.com/opencontainers/runc/libcontainer.(*LinuxFactory).StartInitialization(0xc820060870, 0x7ff358a6e488, 0xc820142fc0)
	/build/amd64-usr/var/tmp/portage/app-emulation/runc-1.0.0_rc2_p9/work/runc-1.0.0_rc2_p9/Godeps/_workspace/src/github.com/opencontainers/runc/libcontainer/factory_linux.go:277 +0x5b1
main.glob.func8(0xc820080780, 0x0, 0x0)
	/build/amd64-usr/var/tmp/portage/app-emulation/runc-1.0.0_rc2_p9/work/runc-1.0.0_rc2_p9/main_unix.go:26 +0x68
reflect.Value.call(0x74fac0, 0x9012a0, 0x13, 0x847808, 0x4, 0xc8200b9278, 0x1, 0x1, 0x0, 0x0, ...)
	/usr/lib/go1.6/src/reflect/value.go:435 +0x120d
reflect.Value.Call(0x74fac0, 0x9012a0, 0x13, 0xc8200b9278, 0x1, 0x1, 0x0, 0x0, 0x0)
	/usr/lib/go1.6/src/reflect/value.go:303 +0xb1
github.com/urfave/cli.HandleAction(0x74fac0, 0x9012a0, 0xc820080780, 0x0, 0x0)
	/build/amd64-usr/var/tmp/portage/app-emulation/runc-1.0.0_rc2_p9/work/runc-1.0.0_rc2_p9/Godeps/_workspace/src/github.com/urfave/cli/app.go:487 +0x2ee
github.com/urfave/cli.Command.Run(0x84a6b8, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x8e05e0, 0x51, 0x0, ...)
	/build/amd64-usr/var/tmp/portage/app-emulation/runc-1.0.0_rc2_p9/work/runc-1.0.0_rc2_p9/Godeps/_workspace/src/github.com/urfave/cli/command.go:191 +0xfec
github.com/urfave/cli.(*App).Run(0xc820001980, 0xc82000a100, 0x2, 0x2, 0x0, 0x0)
	/build/amd64-usr/var/tmp/portage/app-emulation/runc-1.0.0_rc2_p9/work/runc-1.0.0_rc2_p9/Godeps/_workspace/src/github.com/urfave/cli/app.go:240 +0xaa4
main.main()
	/build/amd64-usr/var/tmp/portage/app-emulation/runc-1.0.0_rc2_p9/work/runc-1.0.0_rc2_p9/main.go:137 +0xe24

Removing allowPrivilegeEscalation: false from the manifests for both the deployment and the daemonset eliminates the problem.

OS: CoreOS 1520.6.0
Kubernetes: 1.8.4
Docker: Docker version 1.12.6, build a82d35e

Right now, the documentation (linked from the master branch) is diverging from the release code (v0.1 and v0.1.0). Following the instructions right now will install :latest, which is the master build.

I think the solution to this is to have a fabric function to prepare releases, which does an s/master/$VERSION/g in documentation, and adds a "you are viewing the vX.Y.Z documentation" banner. We can then check that into the v0.x branch, tag the doc update as the release, and in the release notes link to the appropriate tagged artefacts.

I should also look at how other projects do this. Maybe look at CoreDNS, Helm, maybe Kubernetes (although they have a way bigger/fancier site than I could possibly need).

Is this a bug report or a feature request?:

feature request

What happened:

I need multi arch images. In a k8s environment this means: amd64 arm arm64 ppc64le s390x
It will also mean using manifest-tool to upload this to docker hub.

See: https://github.com/coredns/coredns/blob/master/Makefile.release#L139

The above link builds a static go binary using the host machine, i.e. not via a layered docker image. It represents the simplest way we could integrate with coredns.

To be fair: I only need arm images.

I will probably hack around quite a bit; but I'm not sure if it will result in a pull request as it will do it in a Makefile, I also don't understand fabfile.py

In theory, Hugo started life as a blogging engine, so it should be easy to create one page per release with change notes, and have Hugo present them in a nice chronological order.

The big question is: am I smart enough to convince Hugo to do this? We'll see...

When updating bgp speakers config if no localpref is explicitly declared in the pool config

advertisements:
      - localpref: 100

the configuration is applied but when a new loadbalanced service is created the advertisement will be empty:
https://github.com/google/metallb/blob/6d7a06f095d0a429e551d10fc610919bf8da2398/bgp-speaker/main.go#L111

which makes bgpspeaker simply ignore that advertisement with the message

<servicename>: announcable, making 0 advertisements

So ip is assigned from the pool but it never gets announced.

There should be a default for localpref or else such a config should trigger an error.

If we're going all multiprotocol with RIP, we should also implement VRRP. Or more specifically, "traffic steering using unsollicited ARP responses", not actually VRRP.

The way VRRP normally works is that the speakers ping each other over the network, and whoever wins takes ownership of the virtual router MAC address. This works well for stateless routers where failover can be transparent, but for endpoints, having a virtual MAC that you need to teach the kernel about is cumbersome and doesn't really help you that much.

What we really want is to just make the LB IPs portable, and we can do that with unsollicited ARP response pinging. The general idea: a new arp-speaker (could be a deployment or a daemonset) runs leader election through kubernetes. The winning leader sends periodic unsollicited ARP responses saying that "service-ip is-at node-mac-addr". It additionally runs an AF_PACKET socket (with appropriate BPF program attached) to listen for ARP who-has for service IPs, and responds to those as well.

The net result is that the local L2 segment will send service IP traffic to the elected cluster node, which will then LB.

This routing mode only really works with the cluster LB policy.

It turns out my primary BGP-speaking piece of hardware, my Unifi security gateway, uses Quagga to speak BGP. Quagga refuses any connection that doesn't advertise multiprotocol support for IPv4 unicast.

So, MetalLB really does need to speak MP-BGP now, at least well enough to be accepted by Quagga. This is also allegedly a compatibility issue with some Cisco hardware, so it's good that we're getting it out of the way now.

Allegedly, IPv6 support is landing in 1.9 as an alpha feature. This is sooner than I expected, so I'll need to get a 1.9 test cluster set up with v6 support and make MetalLB support IPv6.

Among other things, this implies implementing MP-BGP, ugh.

internal/k8s is all the icky glue between Kubernetes and the cleaner interfaces used by the binaries. I need to do research on how k8s usually tests such client logic.

Is this a bug report or a feature request?:

feature request

What happened:

In #50 , @xnaveira describes a cluster architecture where k8s nodes are connected to different BGP-speaking switches. In that setup, MetalLB's peering configuration is not adequate, because the configuration specifies a single peering config for the entire cluster, but in Xavier's cluster, specific nodes can only peer with specific routers.

We should consider supporting, as an advanced configuration option, per-node peering configurations, so that Xavier's use case can Just Work without extra hacks required.

What would be the preferred way to integrate this? Add arp-speaker dir and cut 'n paste bgp-speaker?
Might make sense as a first cut and then we can abstract away the simularaties.

Is this a bug report or a feature request?:

bug

What happened:

When there is no configmap, arp-speaker (which is basically) bgp-speaker, spam the logs every second:

I1215 08:23:05.722742       1 main.go:107] kube-system/kube-scheduler: stopping announcements, service deleted
I1215 08:23:06.809627       1 main.go:107] kube-system/kube-controller-manager: stopping announcements, service deleted
I1215 08:23:07.846047       1 main.go:107] kube-system/kube-scheduler: stopping announcements, service deleted
I1215 08:23:08.849858       1 main.go:107] kube-system/kube-controller-manager: stopping announcements, service deleted
I1215 08:23:09.845065       1 main.go:107] kube-system/kube-scheduler: stopping announcements, service deleted
I1215 08:23:10.895291       1 main.go:107] kube-system/kube-controller-manager: stopping announcements, service deleted
I1215 08:23:11.863993       1 main.go:107] kube-system/kube-scheduler: stopping announcements, service deleted
I1215 08:23:12.941418       1 main.go:107] kube-system/kube-controller-manager: stopping announcements, service deleted
I1215 08:23:13.905972       1 main.go:107] kube-system/kube-scheduler: stopping announcements, service deleted
I1215 08:23:14.974906       1 main.go:107] kube-system/kube-controller-manager: stopping announcements, service deleted
I1215 08:23:15.959814       1 main.go:107] kube-system/kube-scheduler: stopping announcements, service deleted
I1215 08:23:17.022825       1 main.go:107] kube-system/kube-controller-manager: stopping announcements, service deleted
I1215 08:23:18.010477       1 main.go:107] kube-system/kube-scheduler: stopping announcements, service deleted
I1215 08:23:19.059729       1 main.go:107] kube-system/kube-controller-manager: stopping announcements, service deleted
I1215 08:23:20.115877       1 main.go:107] kube-system/kube-scheduler: stopping announcements, service deleted
I1215 08:23:21.104206       1 main.go:107] kube-system/kube-controller-manager: stopping announcements, service deleted
I1215 08:23:22.176145       1 main.go:107] kube-system/kube-scheduler: stopping announcements, service deleted
I1215 08:23:23.138874       1 main.go:107] kube-system/kube-controller-manager: stopping announcements, service deleted
I1215 08:23:24.224595       1 main.go:107] kube-system/kube-scheduler: stopping announcements, service deleted
~~~


**What you expected to happen**:

Nothing is being done, it shouldn't log at all.

Is this a bug report or a feature request?:

Bug

What happened:

There are major changes to metallb's structure and functionality. The documentation needs a full re-read, so that we can correct and improve it.

Is this a bug report or a feature request?:
question

What happened:
Followed the tutorial from https://metallb.universe.tf/tutorial/ and worked fine. when attempting to run quagga on an external server, bgp not able to establish and receiving below error on bgp speaker.

E1215 20:50:38.805259 1 bgp.go:48] read OPEN from "10.xx.xx.xx:179": EOF
E1215 20:50:50.805889 1 bgp.go:48] read OPEN from "10.xx.xx.xx:179": read tcp 192.168.144.225:54296->10.xx.xx.xx:179: i/o timeout
E1215 20:50:52.806877 1 bgp.go:48] read OPEN from "10.xx.xx.xx:179": read tcp 192.168.144.225:54334->10.xx.xx.xx:179: read: connection reset by peer

What you expected to happen:
BGP to establish

How to reproduce it (as minimally and precisely as possible):
Run quagga outside of k8s and attempt to connect.

Anything else we need to know?:

Metallb config map..

kubectl describe configmap config -n metallb-system
Name: config
Namespace: metallb-system
Labels:
Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","data":{"config":"peers:\n- peer-address: 10.xx.xx.xx\n peer-asn: 100\n my-asn: 200\naddress-pools:\n- name: default\n cidr:\n -...

Data

config:

peers:

• peer-address: 10.xx.xx.xx
  peer-asn: 100
  my-asn: 200
  address-pools:
• name: default
  cidr:
  • 10.xx.xx.xx/26
    advertisements:
  • aggregation-length: 32

Environment:

• MetalLB version: master

• Kubernetes version: 1.8.4

• BGP router type/version: quagga-0.99.22.4-4

• OS (e.g. from /etc/os-release):
  NAME="CentOS Linux"
  VERSION="7 (Core)"
  ID="centos"
  ID_LIKE="rhel fedora"
  VERSION_ID="7"
  PRETTY_NAME="CentOS Linux 7 (Core)"
  ANSI_COLOR="0;31"
  CPE_NAME="cpe:/o:centos:centos:7"
  HOME_URL="https://www.centos.org/"
  BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

• Kernel (e.g. uname -a):
  Linux node-7 3.10.0-514.10.2.el7.x86_64 #1 SMP Fri Mar 3 00:04:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Is this a bug report or a feature request?:
Bug

What happened:
After finding a workaround to #39 the fab push script aborts with the following message:

[localhost] local: kubectl set image -n metallb-system ds/bgp-speaker bgp-speaker=10.97.206.15:80/bgp-speaker:1512984515.148616
error: daemonsets/bgp-speaker no kind "DaemonSet" is registered for version "apps/v1beta1"

What you expected to happen:
fab push completes without an error and the new compiled version of the components run on minikube.

How to reproduce it (as minimally and precisely as possible):

Clone, apply #39, fab start, fab push

Versions

kubectl version output:

Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"6e937839ac04a38cac63e6a7a306c5d035fe7b0a", GitTreeState:"clean", BuildDate:"2017-09-28T22:57:57Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"0b9efaeb34a2fc51ff8e4d34ad9bc6375459c4a4", GitTreeState:"clean", BuildDate:"2017-11-29T22:43:34Z", GoVersion:"go1.9.1", Compiler:"gc", Platform:"linux/amd64"}

Currently we have SetBalancer and SetConfig. I think (nothing coded) yet, that DeleteBalancer makes sense as well.

Same as #22 and #23, Arista has a virtual EOS system. We should do interop testing against it.

BGP's unstable hashing causes massive traffic disruptions when the backend set for an IP changes. The canonical way of fixing this would be to have an extra layer of stateful load-balancing with consistent hashing (a-la Maglev), but no such balancer exists in a usable state today.

There is an alternative, based on a hack Google utilized years ago to mitigate the issues of "BGP as a load-balancer". The idea is that service IPs get balanced to a static set of movable "intermediate" VIPs, and each of those intermediate VIPs is dynamically routed to a single real backend node.

For example, let's say service IP 1.2.3.4 should be forwarded to k8s nodes 10.0.0.1 and 10.0.0.2. The obvious set of BGP announcements would be:

node A: 1.2.3.4/32 -> 10.0.0.1
node B: 1.2.3.4/32 -> 10.0.0.2

With this hack, instead, the advertisements would look something like:

node A: 1.2.3.4/32 -> 172.16.0.0, 172.16.0.1, ..., 172.16.0.254, 172.16.0.255
node A: 172.16.0.0/25 -> 10.0.0.1
node B: 1.2.3.4/32 -> 172.16.0.0, 172.16.0.1, ..., 172.16.0.254, 172.16.0.255
node A: 172.16.0.128/25 -> 10.0.0.2

In addition, the BGP speakers on the nodes monitor each others's health, and if one speaker goes down, another "steals" its intermediate IPs.

The net effect is that when a node goes down, only traffic that was going to that node is affected, not all traffic to all nodes.

Implementing this is significantly tricky, because BGP speakers now have to health-check each other and somehow try to guard against split brain issues. There also needs to be a deterministic algorithm for splitting the intermediate range among an arbitrary number of nodes, and another for redistributing a dead speaker's addresses to other nodes, all while trying to keep traffic evenly spread and minimize disruptions.

This may be too complex to justify doing the work. My preferred solution would be that kube-proxy switched to IPVS balancing by default, and someone contributes maglev-style consistent hash + conntrack balancing to IPVS, so that we can just rely on kube-proxy to provide a "stabilization" layer behind BGP. However, that may never happen, so I'm jotting down this alternative here just in case.

BGP add-path is a prerequisite for this change, so it'll be a while even if I do decide to implement it.

Is this a bug report or a feature request?:

feature

What happened:

Logs are still spammy for arp-speaker

What you expected to happen:

I'll remove some of the glogs, we don't care about all those arps flying by.

Is this a bug report or a feature request?:

feature request

What happened:

The protocol selection is now global, it makes more sense to push this down to the address-pools so that you can mix and match.

@danderson:

Can you make this field per-address-pool? In my cluster, I want some address pools advertised over BGP, and others advertised over ARP :)

Another open question: for a single address pool, does it make sense to advertise several protocols? As in, `protocol: ["bgp", "arp"] ? I'm not sure if it makes sense, but I don't know. WDYT?

Not sure about the mutiple protocols, smells a bit like premeture optimization; we can leave that until some asks for it.

In some environments, you'll have some large address pools of "cheap" IPs (e.g. RFC1918), and some smaller pools of "expensive" IPs (e.g. public IPv4 addresses leased on the grey market).

Right now, MetalLB will allocate IPs from any address pool that has free addresses. This might end up burning "expensive" addresses for services that don't need it. Of course, you can override this selection wiith loadBalancerIP and metallb.universe.tf/address-pool, but this is error-prone if the default is still to do something expensive.

Address pool configs should have an optional parameter, auto-assign, defaulting to true. When turned off, MetalLB will not allocate from the given pool unless that pool has been explicitly requested via loadBalancerIP or the address-pool annotation.