Kafka and ZooKeeper are heavyweight backing services about kubeless HOT 15 CLOSED

I've noticed that I have to go out of my way to provision a 5 GB RAM minikube VM to run some simple 5-service stacks that consist of JVM-based heavyweight services that are really designed for dedicated servers. That barely leaves any headroom for my apps, which should be receiving the lions share of RAM.

Since Redis can do pubsub in 8MB of RAM, I always question why people prefer a 700MB equivalent. For what?

MANY of the fat Java services are being redone in C++ and Go to address this issue. The Java services were designed for another era... the old centralized fat server cloud era.

Now that minikube is bringing development back to the desktop, it only makes sense to call these assumptions into question. It just does not take 700 MB to deliver to me a simple message passing piece of plumbing. And I don't have a lot of RAM to spare by wasting it on things like that. Especially if my stack is thick and I'm trying to assemble 15 different things.

from kubeless.

@zerkms you don't need a custom build, you need to implement a controller that deals with your events. See how nats is now supported:
https://medium.com/bitnami-perspectives/introducing-nats-support-in-kubeless-f2dcc49e1f80

from kubeless.

Memory footprint reduction is what you would gain (~ 20x min). For developer box targeted stacks, JVM's are ultra-heavyweight. A developer will run out of memory if their fabric controller takes up just as much memory as the other things that have to run locally.

from kubeless.

it is not that lightweight :)

Apache OpenWhisk uses Kafka, the performance of scale of it are great. Currently we start it easily in a deployment in a single Pod. It is not fit for production but it works out of the box. I am not sure what we would gain by using Redis pub/sub.

There is an "issue" open to try etcd operators to expose a zk interface to kafka. That sounds interesting.

from kubeless.

@drauschenbach actually, did you try to run kubeless ?

Maybe you comment is because you thought you have to install zk and kafka separately ?

We launch a k8s deployment which runs kafka and zk, so you don't have to install them or configure them, it is done by kubeless install

from kubeless.

While the memory consumption is a fair argument, I feel like a simpler solution would be to configure the pods for kafka and zookeeper's JVMs to consume a smaller subset of memory than they do by default.

To support both is more work but k8s objects make it so if the code side can be made to handle either technology then they can both be wrapped with Service objects that share the same name and let users decide which to apply when upping.

In other words kubeless install could default to whichever is most desirable and have a flag to indicate. E.g. kafka/zk deploys + service named 'x' versus redis/etcd deploys + service named 'x', kubeless always tries to talk to service 'x'.

from kubeless.

So, is there any resolution on having Redis as pub/sub engine for kubeless?

I also feel that the best way to go would be to allow the user installing to choose between the avilable options.

from kubeless.

@radu-matei there is no resolution. Right now it is kafka because of enterprise requirements.

But the only interaction between the event system and the kubeless controller is actually in the runtime. To support something like redis or nats, we would just need to:

• add a kubeless controller startup option which specifies which event broker to use which would specify which runtimes to use when a function is created.
• have runtimes that can consume event on the channels created by those brokers.

so this is definitely still open and totally doable, but we are not actively working on a better runtime support.

any PR to address this would be super welcome though :)

from kubeless.

all that said, which solution would you like to see first ? Redis ? Nats ? RabbitMQ ?

keeping in mind that we will need services to be able to publish to these systems to be able to trigger the functions.

from kubeless.

I would vote for http://nats.io/ :)

from kubeless.

@drauschenbach This issue title will only trigger opinions, not facts. I think you should rename it to argue for pluggable backends instead.

What is the practical problem you've seen? In Minikube with the default 2GB memory, a scaled down Kafka cluster leaves lots of room for other services. In Yolean/kubernetes-kafka#35 I've found a 1+1 kafka+zookeeper cluster to run nicely on 700MB memory, or a bit more with jmx monitoring.

I think it's a great idea by this project to trigger functions through kafka. Compared to Fission etc that adds transparency. Backends may become pluggable, but it's also possible that work towards execution semantics will increase the reliance on Kafka.

from kubeless.

Before we start shouting the names of specific alternatives, we should also work out what the requirements are: ie: at-least-once vs at-most-once, reliability vs cost of queue persistence, horizontal scalability targets (in both directions), message size/throughput requirements, etc.

I suspect the various suggestions above are not all targeting the same expected use case (and indeed, there may not be a single solution that suits everyone)...

from kubeless.

see #354

from kubeless.

With 0.5.0, you can now trigger function with any events (as long as you write a controller for it).

Kafka is not bundled by default anymore

from kubeless.

@sebgoa

as long as you write a controller for it

does it mean one needs to have a custom build of kubeless then? Or is it possible to implement a controller aside and plug it somehow?

Thanks

from kubeless.