ipeluffo / airflow-on-kubernetes Goto Github PK
View Code? Open in Web Editor NEWSource code for guide to run Apache Airflow on Kubernetes
License: Apache License 2.0
Source code for guide to run Apache Airflow on Kubernetes
License: Apache License 2.0
Hi Peluffo,
I try to use your setup script to run my minikube.
The deployment seemed to go successfully, however, checking the log from airflow scheduler pod tells me that KubernetesJobWatcher fails due to HTTP connection timeout.
Traceback (most recent call last):
File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/usr/local/lib/python3.7/site-packages/airflow/contrib/executors/kubernetes_executor.py", line 340, in run
self.worker_uuid, self.kube_config)
File "/usr/local/lib/python3.7/site-packages/airflow/contrib/executors/kubernetes_executor.py", line 364, in _run
**kwargs):
File "/usr/local/airflow/.local/lib/python3.7/site-packages/kubernetes/watch/watch.py", line 144, in stream
for line in iter_resp_lines(resp):
File "/usr/local/airflow/.local/lib/python3.7/site-packages/kubernetes/watch/watch.py", line 46, in iter_resp_lines
for seg in resp.read_chunked(decode_content=False):
File "/usr/local/lib/python3.7/site-packages/urllib3/response.py", line 781, in read_chunked
self._original_response.close()
File "/usr/local/lib/python3.7/contextlib.py", line 130, in __exit__
self.gen.throw(type, value, traceback)
File "/usr/local/lib/python3.7/site-packages/urllib3/response.py", line 430, in _error_catcher
raise ReadTimeoutError(self._pool, None, "Read timed out.")
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='10.96.0.1', port=443): Read timed out.
[2020-08-25 07:31:44,677] {{kubernetes_executor.py:447}} ERROR - Error while health checking kube watcher process. Process died for unknown reasons
[2020-08-25 07:31:44,690] {{kubernetes_executor.py:351}} INFO - Event: and now my watch begins starting at resource_version: 0
What I did is:
$ git clone https://github.com/ipeluffo/airflow-on-kubernetes.git
$ cd airflow-on-kubernetes
$ sh script-apply.sh
$ kubectl get pod
NAME READY STATUS RESTARTS AGE
airflow-scheduler-9f47d7f67-2xfs4 1/1 Running 0 76s
airflow-webserver-77446fcfd-rw8hp 1/1 Running 0 76s
postgres-6d9f4b87c4-bwxkn 1/1 Running 0 77s
Airflow-webserver can be accessed via web browser. But when I try to execute some DAGs, the job will not be executed because of the errors in airflow-scheduler pod.
I found same issue on stackoverflow, but your script and configuration files seem to be correct...
My minkube setups are the following:
minikube: version: v1.10.1, commit: 63ab801ac27e5742ae442ce36dff7877dcccb278
kubectl:
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GitCommit:"2e7996e3e2712684bc73f0dec0200d64eec7fe40", GitTreeState:"clean", BuildDate:"2020-05-20T12:52:00Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T11:48:36Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
OS: Ubuntu 18.04
Do you have any idea to solve this problem?
Looks like puckel/docker-airflow is no longer avilable. Had to pull the image from https://hub.docker.com/r/apache/airflow
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.