vmware-archive / kubeless Goto Github PK

View Code? Open in Web Editor NEW

6.9K 169.0 755.0 82.29 MB

Kubernetes Native Serverless Framework

Home Page: https://kubeless.io

License: Apache License 2.0

Go 83.78% Python 0.60% Makefile 0.53% Shell 13.31% Dockerfile 0.37% Jsonnet 1.40%

kubernetes serverless golang serverless-architectures function-as-a-service kubeless faas

kubeless's Introduction

WARNING: Kubeless is no longer actively maintained by VMware.

VMware has made the difficult decision to stop driving this project and therefore we will no longer actively respond to issues or pull requests. If you would like to take over maintaining this project independently from VMware, please let us know so we can add a link to your forked project here.

Thank You.

Overview

kubeless is a Kubernetes-native serverless framework that lets you deploy small bits of code without having to worry about the underlying infrastructure plumbing. It leverages Kubernetes resources to provide auto-scaling, API routing, monitoring, troubleshooting and more.

Kubeless stands out as we use a Custom Resource Definition to be able to create functions as custom kubernetes resources. We then run an in-cluster controller that watches these custom resources and launches runtimes on-demand. The controller dynamically injects the functions code into the runtimes and make them available over HTTP or via a PubSub mechanism.

Kubeless is purely open-source and non-affiliated to any commercial organization. Chime in at anytime, we would love the help and feedback !

Tools

A UI is available. It can run locally or in-cluster.
A serverless framework plugin is available.

Quick start

Check out the instructions for quickly set up Kubeless here.

Building

Consult the developer's guide for a complete set of instruction to build kubeless.

Compatibility Matrix with Kubernetes

Kubeless fully supports Kubernetes versions greater than 1.9 (tested until 1.15). For other versions some of the features in Kubeless may not be available. Our CI run tests against two different platforms: GKE (1.12) and Minikube (1.15). Other platforms are supported but fully compatibiliy cannot be assured.

Roadmap

We would love to get your help, feel free to lend a hand. We are currently looking to implement the following high level features:

Add other runtimes, currently Golang, Python, NodeJS, Ruby, PHP, .NET and Ballerina are supported. We are also providing a way to use custom runtime. Please check this doc for more details.
Investigate other messaging bus (e.g SQS, rabbitMQ)
Optimize for functions startup time
Add distributed tracing (maybe using istio)

Community

Issues: If you find any issues, please file it.

Slack: We're fairly active on slack and you can find us in the #kubeless channel.

kubeless's People

Contributors

Stargazers

Watchers

Forkers

bamedro verchol jojow lucasproclc ngtuna vishnudxb endocode dongsupark milamberspace dluc rosskukulinski cuulee anguslees beuno james-w maxwell92 j-fuentes ipaste dbarranco dmitrykolesnikovich tsenapathy hoangvantu jayarun jcbsmpsn briceburg absoludity 40a wingkaiho maniacs-ops fonghuiyue krischanglaesel bonifaido dmyerscough vdt reghuramrv ddysher imskyer andrewrothstein fouadsemaan arapulido rfeijolo julianpistorius rsmitty rowhit florianludwig andresmgot jjo prydonius sasukeh stackpointcloud x1957 yodata annie2010 artjoman beejhuff ellerbrock cloud-architecture manjufy infradirectory apidirectory tompizmor dgem nicdoye matthewmorgan vtuson huzhengchuan fabriciosanchez aljannuzzi ubaidsayyed54 stethd arjunrao87 kmarquardsen mikalv tylerjl srve4 vijay172 rodcloutier edwardjrp danlishka kinvolk-archives jbianquetti-nami fabioy tolleiv gladiatr72 mike-pun ridruejo-zz jameskhamil erictchiu qpairio sunilkumarmohanty ccll murali-reddy thorfour xiangkongyue123 xyq18 bharu456 raffaello appcoreopc moolen varuntalus

kubeless's Issues

Modify controller to watch third party resources

just echo resources in stdout for now..

Support HTTPS endpoint

how to handle certificates when https endpoint is needed

Figure out how to create topics

we should be able to create kafka topics via kubeless but for that we need to exec into the kaka-controller pod.

add circle-ci

So, I'm gonna change this to circle-ci as bitnami is currently using it.

pass json data to functions

want to be able to pass data to the functions.

How to update functions

Once a function is created we need to figure out how we can update it.

Technically I think we could just kubectl replace the configmap of the function.

We need to check how lambda and gcloud functions handle updates...

Check the need to create a pool of hot runtime

These runtimes are empty, sitting idle (but running) and waiting for a function.

This could shorten the startup time of a function.

But we don't want to reinvent the wheel, and have to write function scheduler on top of k8s.

though that's what fission does.

Improve build

either use glide or a Makefile so we can build easily and handle dependencies

Make a github release

then update README with instructions for download.

Build image for controller

we can make a release of the controller container image publicly.

Make controller create deployment

Setup a handler that triggers a deployment creation.

Just make it create a busybox that sleeps.

Event consumer runtime is blocking

Figure out why the event consumer is blocking on first message.

Interactively, the function does not block, need to check why its blocks in the default setup...

Kubeless function log

I see Google Cloud Functions support viewing history log of function.

$ gcloud alpha functions logs read helloWorld

LEVEL  NAME        EXECUTION_ID    TIME_UTC                 LOG
D      helloWorld  -               2016-03-17 20:35:41.043  User function helloWorld loaded
D      helloWorld  eTe2m2-RVmJU-0  2016-03-17 20:45:18.334  User function triggered, starting execution
I      helloWorld  eTe2m2-RVmJU-0  2016-03-17 20:45:18.335  My Cloud Function: Hello World!
D      helloWorld  eTe2m2-RVmJU-0  2016-03-17 20:45:18.335  Execution took 2 ms, user function completed successfully

Compiled version panics with `redefined: log_dir`

./bundles/kubeless_darwin-amd64/kubeless flag redefined: log_dir
panic: ./bundles/kubeless_darwin-amd64/kubeless flag redefined: log_dir

goroutine 1 [running]:
flag.(*FlagSet).Var(0xc42006e1e0, 0x39765c0, 0xc420308ec0, 0x2ba717b, 0x7, 0x2bd8c0a, 0x2f)
	/usr/local/Cellar/go/1.8/libexec/src/flag/flag.go:793 +0x420
flag.(*FlagSet).StringVar(0xc42006e1e0, 0xc420308ec0, 0x2ba717b, 0x7, 0x0, 0x0, 0x2bd8c0a, 0x2f)
	/usr/local/Cellar/go/1.8/libexec/src/flag/flag.go:696 +0x8b
flag.(*FlagSet).String(0xc42006e1e0, 0x2ba717b, 0x7, 0x0, 0x0, 0x2bd8c0a, 0x2f, 0xc420308eb0)
	/usr/local/Cellar/go/1.8/libexec/src/flag/flag.go:709 +0x90
flag.String(0x2ba717b, 0x7, 0x0, 0x0, 0x2bd8c0a, 0x2f, 0x39c9340)
	/usr/local/Cellar/go/1.8/libexec/src/flag/flag.go:716 +0x69
github.com/skippbox/kubeless/vendor/k8s.io/client-go/1.4/vendor/github.com/golang/glog.init()
	/Users/rossk/dev/gocode/src/github.com/skippbox/kubeless/vendor/k8s.io/client-go/1.4/vendor/github.com/golang/glog/glog_file.go:41 +0x14a
github.com/skippbox/kubeless/vendor/k8s.io/client-go/1.4/pkg/labels.init()
	/Users/rossk/dev/gocode/src/github.com/skippbox/kubeless/vendor/k8s.io/client-go/1.4/pkg/labels/selector.go:823 +0x64
github.com/skippbox/kubeless/vendor/k8s.io/client-go/1.4/pkg/api/unversioned.init()
	/Users/rossk/dev/gocode/src/github.com/skippbox/kubeless/vendor/k8s.io/client-go/1.4/pkg/api/unversioned/zz_generated.deepcopy.go:391 +0x8c
github.com/skippbox/kubeless/vendor/k8s.io/client-go/1.4/pkg/api.init()
	/Users/rossk/dev/gocode/src/github.com/skippbox/kubeless/vendor/k8s.io/client-go/1.4/pkg/api/zz_generated.deepcopy.go:3750 +0x7b
github.com/skippbox/kubeless/vendor/k8s.io/client-go/1.4/rest.init()
	/Users/rossk/dev/gocode/src/github.com/skippbox/kubeless/vendor/k8s.io/client-go/1.4/rest/versions.go:89 +0x82
github.com/skippbox/kubeless/vendor/k8s.io/kubernetes/plugin/pkg/client/auth/gcp.init()
	/Users/rossk/dev/gocode/src/github.com/skippbox/kubeless/vendor/k8s.io/kubernetes/plugin/pkg/client/auth/gcp/gcp.go:120 +0x62
github.com/skippbox/kubeless/vendor/k8s.io/kubernetes/plugin/pkg/client/auth.init()
	/Users/rossk/dev/gocode/src/github.com/skippbox/kubeless/vendor/k8s.io/kubernetes/plugin/pkg/client/auth/plugins.go:24 +0x44
github.com/skippbox/kubeless/vendor/k8s.io/kubernetes/pkg/client/unversioned.init()
	/Users/rossk/dev/gocode/src/github.com/skippbox/kubeless/vendor/k8s.io/kubernetes/pkg/client/unversioned/util.go:80 +0xf0
github.com/skippbox/kubeless/pkg/utils.init()
	/Users/rossk/dev/gocode/src/github.com/skippbox/kubeless/pkg/utils/k8sutil.go:550 +0x89
github.com/skippbox/kubeless/cmd.init()
	/Users/rossk/dev/gocode/src/github.com/skippbox/kubeless/cmd/version.go:37 +0x7f
main.init()
	/Users/rossk/dev/gocode/src/github.com/rosskukulinski/kubeless/main.go:24 +0x44

@sebgoa I'm still learning Golang - is there something I've missed or screwed up?

Kafka and ZooKeeper are heavyweight backing services

If this project is in the spirit of lightweight or server-less k8s, then these heavyweight dependencies should be replaced with their lightweight alternatives like Redis for pub/sub and etcd or Consul for KV.

maintain separate binaries

The need of separating server & cli binaries is increasing

refactor kubeless controller

I am seeing some changes upstream in writing a controller for getting better performance. Kubeless controller need to be updated to that point too.

Ref: https://github.com/kubernetes/community/blob/master/contributors/devel/controllers.md#guidelines

Import Kubewatch controller skeleton

this might be a good start.

Just remove the SLACK handler.

kubeless uninstall

remove all running functions. Probably!
remove thirdpartyresource
remote kubeless-controller deployment

kafka version is hard coded when doing `kubeless topic` commands

but I think we are pulling :latest tag and it gets updated :)

Transition to Bitnami Org

Since Skippbox got acquired, we need to move kubeless to Bitnami Org.

Add topics variable to event runtime

Once #27 is solved, we can add a env var to the runtime manifest, load that env in the Python runtime and use it to listen to events only in that topic.

move the topic publish to function publish

seems counter intuitive to publish via the kubeless function publish rather than kubeless topic publish

we probably need to switch these calls.

Add `kubeless call`

We should be able to invoke the function from the command line.

This requires exposing the function deployment as a service, or using a hacky port-froward connection.

Add `kubeless function ls`

Which basically will list the custom resource...

Add Java runtime

Handle namespaces

Two things:

the controller may need to run in a dedicated namespace
the functions may be started in a different namespace than "default"

Add javascript runtime

Create Ingress rules

If we were deployed in a cluster with a DNS zone etc...

We could automatically create an ingress rule for the function...so that we can invoke it as <function_name>.<DNS_name>

Kind is missing in lambdas

I created a function and it seems kind is missing from the lambda resource:

$ ./kubeless function create hello --runtime python --handler hello.handler --from-file ./examples/hello.py
sebimac:kubeless (master)$ ./kubeless function ls
hello
sebimac:kubeless (master)$ kubectl get lambdas
NAME      LABELS    DATA
hello     <none>    {"metadata":{"creationTimestamp":"2016-11-28T10:28...
sebimac:kubeless (master)$ kubectl get lambdas hello -o yaml
error: Object 'Kind' is missing in '{"metadata":{"name":"hello","namespace":"default","selfLink":"/apis/k8s.io/v1/namespaces/default/lambdas/hello","uid":"60896b8f-b555-11e6-9987-3e7edc3e1d78","resourceVersion":"136959","creationTimestamp":"2016-11-28T10:28:17Z"},"spec":{"handler":"hello.handler","lambda":"import json\ndef handler():\n    return \"hello world\"\n","runtime":"python","version":"0"}}
'

Bash completion

persistence: try backing kafka with the etcd operator and zetcd

The CoreOS team has built zetcd which proxies Zookeeper API to an etcd backend. That, along with the etcd Operator, should make a nice Kubernetes native Zookeeper setup.

cc @heyitsanthony @xiang90

Does not compile because of missing context

~/import/kubeless16:11 >make
CGO_ENABLED=1 ./script/make.sh binary
---> Making bundle: binary (in .)
../gopath/src/k8s.io/kubernetes/vendor/k8s.io/client-go/rest/request.go:21:2: cannot find package "context" in any of:
/home/thomas/import/gopath/src/k8s.io/kubernetes/vendor/context (vendor tree)
/home/thomas/import/go/src/context (from $GOROOT)
/home/thomas/import/gopath/src/context (from $GOPATH)
Makefile:15: recipe for target 'binary' failed
make: *** [binary] Error 1

Differentiate between http and events via flags when creating a function

When we create a function we should be able to pass a flag that makes the difference between a http endpoint or an event endpoint.

http endpoint are what we have now, you curl the function, the runtime injects the function and wraps it into http CRUD.

kubeless function create hello --trigger-http

event endpoints: a consumer gets started and listens to a Kafka managed topic. when we publish a message to that topic, the consumer gets the message and executes the method.
https://cloud.google.com/functions/docs/tutorials/pubsub

kubeless function create hello --trigger-topic <topic_name>

For now, we should be able to handle things without a specified topic name. We will use kubeless as a topic name. that way:

kubeless function create hello --trigger-topic

should work. Just read the value and if none, set it to kubeless.

Get rid of the need for a local proxy

we still need a local proxy, we should get rid of it.

a better k8s client or split server and client in two.

messges received by the kafka consumer are not json objects

somehow the messages received by the kafka consumer are not decoded as json.

maybe the call:

kubeless function publish --topic s3 --data '{"foo":"bar"}'

does not send proper strings...

Make the CLI more consistent with gcloud

Currently we are mixing lambda and gcloud CLI syntax.

Let's make it more consistent with gcloud for now.

Improving stdout messages

Make not working

just tried make:

$ make
CGO_ENABLED=1 ./script/make.sh binary
---> Making bundle: binary (in .)
package main
	imports github.com/skippbox/kubeless/cmd
	imports github.com/skippbox/kubeless/pkg/controller
	imports github.com/skippbox/kubeless/pkg/function
	imports github.com/skippbox/kubeless/pkg/utils
	imports k8s.io/kubernetes/pkg/kubectl/cmd/util
	imports k8s.io/kubernetes/pkg/registry/thirdpartyresourcedata
	imports k8s.io/kubernetes/pkg/registry/generic
	imports k8s.io/kubernetes/pkg/storage/storagebackend/factory
	imports k8s.io/kubernetes/pkg/storage/etcd
	imports k8s.io/kubernetes/pkg/storage/etcd/metrics
	imports github.com/prometheus/client_golang/prometheus
	imports github.com/prometheus/common/expfmt
	imports github.com/prometheus/common/internal/bitbucket.org/ww/goautoneg: use of internal package not allowed
make: *** [binary] Error 1

So we may have to write our own HPA with a custom prometheus metric.

$ Kubeless version

Showing version info

Let controller start zookeeper and kafka

to handle event based triggers, we need to add an event stream processor.

We can use kafka (which needs zookeeper).

kubeless install will launch the controller and the zk and kafka.

At first we can use a single topic but we will need to be able to add topics on the fly.

Then the runtime need to be adjusted to be consumers of events....

Events:
  FirstSeen	LastSeen	Count	From			SubobjectPath	Type		Reason		Message
  ---------	--------	-----	----			-------------	--------	------		-------
  16h		3m		201	{kubelet minikubevm}			Warning		FailedSync	Error syncing pod, skipping: failed to "StartContainer" for "kubeless" with RunContainerError: "runContainer: Error response from daemon: Container command '/kubeless' not found or does not exist."

  16h	3m	200	{kubelet minikubevm}	spec.containers{kubeless}	Normal	Pulled		Container image "skippbox/kubeless-controller:0.0.2" already present on machine

maybe we need to push a new controller image.

kubeless install --controller-version

That makes more sense to have an option to figure out the specific version of controller being installed. Will take the latest if the flag is not used. Like this:

$ kubeless install --controller-version 0.0.7

function ls should return more info

$ ./kubeless function create hello --runtime python --handler hello.handler --from-file ./examples/hello.py
sebimac:kubeless (master)$ ./kubeless function ls
hello

We should return something like

$ ./kubeless function ls
NAME   HANDLER   RUNTIME
hello     handler        python