Minimal working example using Beam, Go, Apache Flink, Bazel, and Kubernetes.
This example is tested with minikube, but it should work for other Kubernetes
distributions as well.
See hello-minikube for a tutorial.
Start minikube:
minikube startTo build container images from this repository and run them in the minikube, set up a local registry:
minikube addons enable registryTo get a local registry working, I typically follow minikube's instructions for MacOS, even though I run minikube on Ubuntu Linux:
docker run --rm -it --network=host alpine ash -c "apk add socat && socat TCP-LISTEN:5000,reuseaddr,fork TCP:$(minikube ip):5000"bazel run //:example-pipeline.applyInspect the logs in your terminal:
kubectl logs --follow job/example-pipelineYou should see output like this at the bottom:
2022/05/21 05:24:35 got output:
Appear: 1
Bedlam: 3
British: 5
Could: 2
Cunning: 1
Detested: 1
Did: 4
Doth: 3
Draw: 4
Drum: 1
During: 1
Edgar: 10
Exeunt: 33
Fear: 1
Filial: 1
...
TODO: NOT WORKING YET
bazel run //:beam-flink-job-server.apply
bazel run //:beam-flink-job-server-service.applyRune kubefwd in a terminal:
sudo -E <PATH_TO_KUBEFWD> services \
--kubeconfig <PATH_TO_HOME>/.kube/config \
--namespace kubernetes-dashboard \
--namespace default \
--namespace kube-systemfails:
bazel run :beam-go-k8s -- \
--runner flink \
--endpoint beam-flink-job-server.default:8099 \
--output /tmp/output-flink.txt \
--environment_type LOOPBACKTODO: NOT WORKING YET
I would like to figure out how to use github.com/apache/flink-kubernetes-operator to run jobs.
bazel run //:cert-manager.applyInstall the operator:
helm repo add flink-operator-repo https://downloads.apache.org/flink/flink-kubernetes-operator-0.1.0/
helm install flink-kubernetes-operator flink-operator-repo/flink-kubernetes-operator(TODO: Install this with bazel commands to avoid needing to install helm locally.)
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent