Comments (4)
Another thing that I have already tried:
Attached the same service account airbyte-admin
with another pod and tried to created kubernetes resources from there and I am able to do it.
from airbyte.
@talha-naeem1 can you share your values.yaml file?
from airbyte.
@marcosmarxm yes
here is my values.yaml file:
### TEST FOR RELEASE WORKFLOW
## @section Global Parameters
## global -- Global params that are overwritten with umbrella chart
global:
## serviceAccountName -- Service Account name override
serviceAccountName: &service-account-name "airbyte-admin"
## deploymentMode -- Deployment mode, whether or not render the default env vars and volumes in deployment spec
deploymentMode: "oss"
edition: "community"
env_vars: {}
## database [object] -- Object used to overrite database configuration(to use external DB)
## database.secretName -- secret name where DB creds are stored
## database.secretValue -- secret value for database password
## database.host -- Database host override
## database.port -- Database port override
database:
secretName: ""
secretValue: ""
host: ""
port: ""
state:
## state.storage.type Determines which state storage will be utilized. One of "MINIO", "S3" or "GCS"
storage:
type: "S3"
logs:
## logs.accessKey.password Logs Access Key
## logs.accessKey.existingSecret
## logs.accessKey.existingSecretKey
accessKey:
password: ""
existingSecret: ""
existingSecretKey: ""
## logs.secretKey.password Logs Secret Key
## logs.secretKey.existingSecret
## logs.secretKey.existingSecretKey
secretKey:
password: ""
existingSecret: ""
existingSecretKey: ""
## logs.storage.type Determines which log storage will be utilized. One of "MINIO", "S3" or "GCS"
## Used in conjunction with logs.minio.*, logs.s3.* or logs.gcs.*
storage:
type: "S3"
## logs.minio.enabled Switch to enable or disable the Minio helm chart
## logs.minio.nodeSelector [object] Node labels for pod assignment
## logs.minio.tolerations [array] Tolerations for webapp pod assignment.
## logs.minio.affinity [object] Affinity and anti-affinity for webapp pod assignment.
minio:
enabled: false
nodeSelector: {}
tolerations: []
affinity: {}
## logs.externalMinio.enabled Switch to enable or disable an external Minio instance
## logs.externalMinio.host External Minio Host
## logs.externalMinio.port External Minio Port
## logs.externalMinio.endpoint Fully qualified hostname for s3-compatible storage
externalMinio:
enabled: false
host: localhost
port: 9000
## logs.s3.enabled Switch to enable or disable custom S3 Log location
## logs.s3.bucket Bucket name where logs should be stored
## logs.s3.bucketRegion Region of the bucket (must be empty if using minio)
s3:
enabled: true
bucket: cdpaas-airbyte-stage-logs
bucketRegion: "us-east-2"
## Google Cloud Storage (GCS) Log Location Configuration
## logs.gcs.bucket GCS bucket name
## logs.gcs.credentials The path the GCS creds are written to
## logs.gcs.credentialsJson Base64 encoded json GCP credentials file contents
gcs:
bucket: ""
# If you are mounting an existing secret to extraVolumes on scheduler, server and worker
# deployments, then set credentials to the path of the mounted JSON file
credentials: ""
# If credentialsJson is set then credentials auto resolves (to /secrets/gcs-log-creds/gcp.json)
credentialsJson: ""
metrics:
## These parameters are used in the airbyte-env ConfigMap, which is then mounted in deployments in the airbyte-worker and airbyte-metrics Charts.
## global.metrics.metricClient The metric client to configure globally. Supports "otel"
metricClient: "otel"
## global.metrics.otelCollectorEndpoint The open-telemetry-collector endpoint that metrics will be sent to.
otelCollectorEndpoint: "http://otel-collector.dev.svc.cluster.local:4317"
jobs:
## Jobs resource requests and limits
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
## We usually recommend not to specify default resources and to leave this as a conscious
## choice for the user. This also increases chances charts run on environments with little
## resources, such as Minikube. If you do want to specify resources, uncomment the following
## lines, adjust them as necessary, and remove the curly braces after 'resources:'.
## jobs.resources.limits [object] The resources limits for jobs
## jobs.resources.requests [object] The requested resources for jobs
resources:
## Example:
## requests:
## memory: 256Mi
## cpu: 250m
requests:
cpu: 100m
memory: 25Mi
## Example:
## limits:
## cpu: 200m
## memory: 1Gi
limits:
cpu: '200m'
memory: 50Mi
kube:
## JOB_KUBE_ANNOTATIONS
## pod annotations of the sync job and the default pod annotations fallback for others jobs
## jobs.kube.annotations [object] key/value annotations applied to kube jobs
annotations: {}
## JOB_KUBE_LABELS
## pod labels of the sync job and the default pod labels fallback for others jobs
## jobs.kube.labels [object] key/value labels applied to kube jobs
labels: {}
## JOB_KUBE_NODE_SELECTORS
## pod node selector of the sync job and the default pod node selector fallback for others jobs
## jobs.kube.nodeSelector [object] key/value node selector applied to kube jobs
# nodeSelector: {}
nodeSelector:
dedicated: sync_workers
## JOB_KUBE_TOLERATIONS
## jobs.kube.tolerations [array] Tolerations for jobs.kube pod assignment.
## ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
## any boolean values should be quoted to ensure the value is passed through as a string, eg:
## - key: airbyte-server
## operator: Equal
## value: "true"
## effect: NoSchedule
# tolerations: []
tolerations:
- key: "dedicated"
operator: "Equal"
value: "sync_workers"
effect: "NoSchedule"
## JOB_KUBE_MAIN_CONTAINER_IMAGE_PULL_SECRET
## image pull secret to use for job pod
## jobs.kube.main_container_image_pull_secret [string] image pull secret to use for job pod
main_container_image_pull_secret: ""
images:
## JOB_KUBE_BUSYBOX_IMAGE
## busybox image used by the job pod
## jobs.kube.images.busybox [string] busybox image used by the job pod
busybox: ""
## JOB_KUBE_SOCAT_IMAGE
## socat image used by the job pod
## jobs.kube.images.socat [string] socat image used by the job pod
socat: ""
## JOB_KUBE_CURL_IMAGE
## curl image used by the job pod
## jobs.kube.images.curl [string] curl image used by the job pod
curl: ""
## @section Common Parameters
## nameOverride -- String to partially override airbyte.fullname template with a string (will prepend the release name)
##
nameOverride: ""
## fullnameOverride -- String to fully override airbyte.fullname template with a string
##
fullnameOverride: ""
## Pods Service Account
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/
## serviceAccount.annotations -- [object] Annotations for service account. Evaluated as a template. Only used if `create` is `true`.
## serviceAccount.create -- Specifies whether a ServiceAccount should be created
## serviceAccount.name -- Name of the service account to use. If not set and create is true, a name is generated using the fullname template.
##
serviceAccount:
create: true
annotations: {
eks.amazonaws.com/role-arn: arn:aws:iam::440570343202:role/airbyte-stage-irsa
}
name: *service-account-name
# version Sets the AIRBYTE_VERSION environment variable. Defaults to Chart.AppVersion.
# If changing the image tags below, you should probably also update this.
version: ""
## @section Webapp Parameters
webapp:
enabled: true
## webapp.replicaCount Number of webapp replicas
replicaCount: 1
## webapp.image.repository The repository to use for the airbyte webapp image.
## webapp.image.pullPolicy the pull policy to use for the airbyte webapp image
## webapp.image.tag The airbyte webapp image tag. Defaults to the chart's AppVersion
image:
repository: airbyte/webapp
pullPolicy: IfNotPresent
## webapp.podAnnotations [object] Add extra annotations to the webapp pod(s)
##
podAnnotations: {}
## webapp.podLabels [object] Add extra labels to the webapp pod(s)
##
podLabels: {}
## webapp.containerSecurityContext Security context for the container
## Examples:
## containerSecurityContext:
## runAsNonRoot: true
## runAsUser: 1000
## readOnlyRootFilesystem: true
containerSecurityContext: {}
## Configure extra options for the webapp containers' liveness and readiness probes
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#configure-probes
## webapp.livenessProbe.enabled Enable livenessProbe on the webapp
## webapp.livenessProbe.initialDelaySeconds Initial delay seconds for livenessProbe
## webapp.livenessProbe.periodSeconds Period seconds for livenessProbe
## webapp.livenessProbe.timeoutSeconds Timeout seconds for livenessProbe
## webapp.livenessProbe.failureThreshold Failure threshold for livenessProbe
## webapp.livenessProbe.successThreshold Success threshold for livenessProbe
##
livenessProbe:
enabled: false
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 1
failureThreshold: 3
successThreshold: 1
## webapp.readinessProbe.enabled Enable readinessProbe on the webapp
## webapp.readinessProbe.initialDelaySeconds Initial delay seconds for readinessProbe
## webapp.readinessProbe.periodSeconds Period seconds for readinessProbe
## webapp.readinessProbe.timeoutSeconds Timeout seconds for readinessProbe
## webapp.readinessProbe.failureThreshold Failure threshold for readinessProbe
## webapp.readinessProbe.successThreshold Success threshold for readinessProbe
##
readinessProbe:
enabled: false
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 1
failureThreshold: 3
successThreshold: 1
## webapp.service.type The service type to use for the webapp service
## webapp.service.port The service port to expose the webapp on
## webapp.service.annotations Annotations for the webapp service resource
service:
type: ClusterIP
port: 80
annotations: {}
## Web app resource requests and limits
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
## We usually recommend not to specify default resources and to leave this as a conscious
## choice for the user. This also increases chances charts run on environments with little
## resources, such as Minikube. If you do want to specify resources, uncomment the following
## lines, adjust them as necessary, and remove the curly braces after 'resources:'.
## webapp.resources.limits [object] The resources limits for the Web container
## webapp.resources.requests [object] The requested resources for the Web container
resources:
## Example:
## limits:
## cpu: 200m
## memory: 1Gi
limits: {}
## Examples:
## requests:
## memory: 256Mi
## cpu: 250m
requests: {}
## webapp.nodeSelector [object] Node labels for pod assignment
## Ref: https://kubernetes.io/docs/user-guide/node-selection/
##
nodeSelector: {}
## webapp.tolerations [array] Tolerations for webapp pod assignment.
## ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
##
# tolerations: []
tolerations:
- key: "dedicated"
operator: "Equal"
value: "airbyte_core"
effect: "NoSchedule"
## webapp.affinity [object] Affinity and anti-affinity for webapp pod assignment.
## ref: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity
##
# affinity: {}
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: dedicated
operator: In
values:
- airbyte_core
## Configure the ingress resource that allows you to access the Airbyte installation.
## ref: http://kubernetes.io/docs/user-guide/ingress/
## webapp.ingress.enabled Set to true to enable ingress record generation
## webapp.ingress.className Specifies ingressClassName for clusters >= 1.18+
## webapp.ingress.annotations [object] Ingress annotations done as key:value pairs
## webapp.ingress.hosts The list of hostnames to be covered with this ingress record.
## webapp.ingress.tls [array] Custom ingress TLS configuration
ingress:
enabled: false
className: ""
annotations: {}
# kubernetes.io/ingress.class: nginx
# kubernetes.io/tls-acme: "true"
hosts: []
# - host: chart-example.local
# paths:
# - path: /
# pathType: ImplementationSpecific
tls: []
# - secretName: chart-example-tls
# hosts:
# - chart-example.local
## webapp.api.url The webapp API url.
api:
url: /api/v1/
connector-builder-server:
url: /connector-builder-api
## webapp.fullstory.enabled Whether or not to enable fullstory
fullstory:
enabled: false
## webapp.extraVolumeMounts [array] Additional volumeMounts for webapp container(s).
## Examples (when using `webapp.containerSecurityContext.readOnlyRootFilesystem=true`):
## extraVolumeMounts:
## - name: var-run
## mountPath: /var/run/
## - name: var-cache-nginx
## mountPath: /var/cache/nginx
## - mountPath: /etc/nginx/conf.d
## name: nginx-conf-d
##
extraVolumeMounts: []
## webapp.extraVolumes [array] Additional volumes for webapp pod(s).
## Examples (when using `webapp.containerSecurityContext.readOnlyRootFilesystem=true`):
## extraVolumes:
## - name: var-run
## emptyDir: {}
## - name: var-cache-nginx
## emptyDir: {}
## - name: nginx-conf-d
## emptyDir: {}
##
extraVolumes: []
## extraContainer [array] Additional container for server pod(s)
## Example:
# extraContainers:
# - name: otel_collector
# image: somerepo/someimage:sometag
# args: [
# "--important-args"
# ]
# ports:
# - containerPort: 443
# volumeMounts:
# - name: volumeMountCool
# mountPath: /some/path
# readOnly: true
extraContainers: []
## extraInitContainers [array] Additional init containers for server pod(s)
## Example:
# extraInitContainers:
# - name: sleepy
# image: alpine
# command: ['sleep', '60']
extraInitContainers: []
## extraEnv [array] Supply extra env variables to main container using full notation
## Example: (With default env vars and values taken from generated config map)
# extraEnv:
# - name: POSTGRES_USER
# valueFrom:
# secretKeyRef:
# name: airbyte-secrets
# key: DATABASE_USER
# - name: POSTGRES_PWD
# valueFrom:
# secretKeyRef:
# name: airbyte-secrets
# key: DATABASE_PASSWORD
# - name: DYNAMIC_CONFIG_FILE_PATH
# value: "config/dynamicconfig/development.yaml"
# - name: DB
# value: "postgresql"
# - name: DB_PORT
# valueFrom:
# configMapKeyRef:
# name: airbyte-env
# key: DATABASE_PORT
# - name: POSTGRES_SEEDS
# valueFrom:
# configMapKeyRef:
# name: airbyte-env
# key: DATABASE_HOST
##
extraEnv: []
## secrets [object] Supply additional secrets to container
## Example:
## secrets:
## DATABASE_PASSWORD: strong-password
## DATABASE_USER: my-db-user
secrets: {}
## env_vars [object] Supply extra env variables to main container using simplified notation
## Example:
## env_vars:
## DATABASE_HOST: airbyte-db
## DATABASE_PORT: 5432
env_vars: {}
## @section Pod Sweeper parameters
pod-sweeper:
enabled: true
## podSweeper.image.repository The image repository to use for the pod sweeper
## podSweeper.image.pullPolicy The pull policy for the pod sweeper image
## podSweeper.image.tag The pod sweeper image tag to use
image:
repository: bitnami/kubectl
pullPolicy: IfNotPresent
tag: latest
## podSweeper.podAnnotations [object] Add extra annotations to the podSweeper pod
##
podAnnotations: {}
## podSweeper.podLabels [object] Add extra labels to the podSweeper pod
##
podLabels: {}
## podSweeper.containerSecurityContext Security context for the container
## Examples:
## containerSecurityContext:
## runAsNonRoot: true
## runAsUser: 1000
## readOnlyRootFilesystem: true
containerSecurityContext: {}
## Configure extra options for the podSweeper containers' liveness and readiness probes
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#configure-probes
## podSweeper.livenessProbe.enabled Enable livenessProbe on the podSweeper
## podSweeper.livenessProbe.initialDelaySeconds Initial delay seconds for livenessProbe
## podSweeper.livenessProbe.periodSeconds Period seconds for livenessProbe
## podSweeper.livenessProbe.timeoutSeconds Timeout seconds for livenessProbe
## podSweeper.livenessProbe.failureThreshold Failure threshold for livenessProbe
## podSweeper.livenessProbe.successThreshold Success threshold for livenessProbe
##
livenessProbe:
enabled: true
initialDelaySeconds: 5
periodSeconds: 30
timeoutSeconds: 1
failureThreshold: 3
successThreshold: 1
## podSweeper.readinessProbe.enabled Enable readinessProbe on the podSweeper
## podSweeper.readinessProbe.initialDelaySeconds Initial delay seconds for readinessProbe
## podSweeper.readinessProbe.periodSeconds Period seconds for readinessProbe
## podSweeper.readinessProbe.timeoutSeconds Timeout seconds for readinessProbe
## podSweeper.readinessProbe.failureThreshold Failure threshold for readinessProbe
## podSweeper.readinessProbe.successThreshold Success threshold for readinessProbe
##
readinessProbe:
enabled: true
initialDelaySeconds: 5
periodSeconds: 30
timeoutSeconds: 1
failureThreshold: 3
successThreshold: 1
## Pod Sweeper app resource requests and limits
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
## We usually recommend not to specify default resources and to leave this as a conscious
## choice for the user. This also increases chances charts run on environments with little
## resources, such as Minikube. If you do want to specify resources, uncomment the following
## lines, adjust them as necessary, and remove the curly braces after 'resources:'.
## podSweeper.resources.limits [object] The resources limits for the podSweeper container
## podSweeper.resources.requests [object] The requested resources for the podSweeper container
resources:
## Example:
## limits:
## cpu: 200m
## memory: 1Gi
limits: {}
## Examples:
## requests:
## memory: 256Mi
## cpu: 250m
requests: {}
## podSweeper.nodeSelector [object] Node labels for pod assignment
## Ref: https://kubernetes.io/docs/user-guide/node-selection/
##
nodeSelector: {}
## podSweeper.tolerations [array] Tolerations for podSweeper pod assignment.
## ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
##
# tolerations: []
tolerations:
- key: "dedicated"
operator: "Equal"
value: "airbyte_core"
effect: "NoSchedule"
# affinity: {}
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: dedicated
operator: In
values:
- airbyte_core
## podSweeper.affinity [object] Affinity and anti-affinity for podSweeper pod assignment.
## ref: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity
##
# affinity: {}
## podSweeper.extraVolumeMounts [array] Additional volumeMounts for podSweeper container(s).
## Examples:
## extraVolumeMounts:
## - name: tmpdir
## mountPath: /tmp
##
extraVolumeMounts: []
## podSweeper.extraVolumes [array] Additional volumes for podSweeper pod(s).
## Examples:
## extraVolumes:
## - name: tmpdir
## emptyDir: {}
##
extraVolumes: []
## @section Server parameters
server:
enabled: true
## server.replicaCount Number of server replicas
replicaCount: 1
## server.image.repository The repository to use for the airbyte server image.
## server.image.pullPolicy the pull policy to use for the airbyte server image
## server.image.tag The airbyte server image tag. Defaults to the chart's AppVersion
image:
repository: airbyte/server
pullPolicy: IfNotPresent
## server.podAnnotations [object] Add extra annotations to the server pod(s)
##
podAnnotations: {}
## server.podLabels [object] Add extra labels to the server pod(s)
##
podLabels: {}
## server.containerSecurityContext Security context for the container
## Examples:
## containerSecurityContext:
## runAsNonRoot: true
## runAsUser: 1000
## readOnlyRootFilesystem: true
containerSecurityContext: {}
## Configure extra options for the server containers' liveness and readiness probes
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#configure-probes
## server.livenessProbe.enabled Enable livenessProbe on the server
## server.livenessProbe.initialDelaySeconds Initial delay seconds for livenessProbe
## server.livenessProbe.periodSeconds Period seconds for livenessProbe
## server.livenessProbe.timeoutSeconds Timeout seconds for livenessProbe
## server.livenessProbe.failureThreshold Failure threshold for livenessProbe
## server.livenessProbe.successThreshold Success threshold for livenessProbe
##
livenessProbe:
enabled: false
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 10
failureThreshold: 3
successThreshold: 1
## server.readinessProbe.enabled Enable readinessProbe on the server
## server.readinessProbe.initialDelaySeconds Initial delay seconds for readinessProbe
## server.readinessProbe.periodSeconds Period seconds for readinessProbe
## server.readinessProbe.timeoutSeconds Timeout seconds for readinessProbe
## server.readinessProbe.failureThreshold Failure threshold for readinessProbe
## server.readinessProbe.successThreshold Success threshold for readinessProbe
##
readinessProbe:
enabled: false
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 10
failureThreshold: 3
successThreshold: 1
## server resource requests and limits
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
## We usually recommend not to specify default resources and to leave this as a conscious
## choice for the user. This also increases chances charts run on environments with little
## resources, such as Minikube. If you do want to specify resources, uncomment the following
## lines, adjust them as necessary, and remove the curly braces after 'resources:'.
## server.resources.limits [object] The resources limits for the server container
## server.resources.requests [object] The requested resources for the server container
resources:
## Example:
## limits:
## cpu: 200m
## memory: 1Gi
limits: {}
## Examples:
## requests:
## memory: 256Mi
## cpu: 250m
requests: {}
## server.nodeSelector [object] Node labels for pod assignment
## Ref: https://kubernetes.io/docs/user-guide/node-selection/
##
nodeSelector: {}
## server.tolerations [array] Tolerations for server pod assignment.
## ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
##
# tolerations: []
tolerations:
- key: "dedicated"
operator: "Equal"
value: "airbyte_core"
effect: "NoSchedule"
## server.affinity [object] Affinity and anti-affinity for server pod assignment.
## ref: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity
##
# affinity: {}
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: dedicated
operator: In
values:
- airbyte_core
## server.log.level The log level to log at.
log:
level: "INFO"
## server.extraVolumeMounts [array] Additional volumeMounts for server container(s).
## Examples (when using `server.containerSecurityContext.readOnlyRootFilesystem=true`):
## extraVolumeMounts:
## - name: tmpdir
## mountPath: /tmp
##
extraVolumeMounts: []
## server.extraVolumes [array] Additional volumes for server pod(s).
## Examples (when using `server.containerSecurityContext.readOnlyRootFilesystem=true`):
## extraVolumes:
## - name: tmpdir
## emptyDir: {}
##
extraVolumes: []
## extraContainer [array] Additional container for server pod(s)
## Example:
# extraContainers:
# - name: otel_collector
# image: somerepo/someimage:sometag
# args: [
# "--important-args"
# ]
# ports:
# - containerPort: 443
# volumeMounts:
# - name: volumeMountCool
# mountPath: /some/path
# readOnly: true
extraContainers: []
## extraInitContainers [array] Additional init containers for server pod(s)
## Example:
# extraInitContainers:
# - name: sleepy
# image: alpine
# command: ['sleep', '60']
extraInitContainers: []
## extraEnv [array] Supply extra env variables to main container using full notation
## Example: (With default env vars and values taken from generated config map)
# extraEnv:
# - name: AIRBYTE_VERSION
# valueFrom:
# configMapKeyRef:
# name: airbyte-env
# key: AIRBYTE_VERSION
# - name: API_URL
# valueFrom:
# configMapKeyRef:
# name: airbyte-env
# key: API_URL
# - name: WEBAPP_URL
# valueFrom:
# configMapKeyRef:
# name: airbyte-env
# key: WEBAPP_URL
# - name: TRACKING_STRATEGY
# valueFrom:
# configMapKeyRef:
# name: airbyte-env
# key: TRACKING_STRATEGY
# - name: FULLSTORY
# valueFrom:
# configMapKeyRef:
# name: airbyte-env
# key: FULLSTORY
# - name: INTERNAL_API_HOST
# valueFrom:
# configMapKeyRef:
# name: airbyte-env
# key: INTERNAL_API_HOST
##
extraEnv: []
## secrets [object] Supply additional secrets to container
## Example:
## secrets:
## DATABASE_PASSWORD: strong-password
## DATABASE_USER: my-db-user
secrets: {}
## env_vars [object] Supply extra env variables to main container using simplified notation
## Example:
## env_vars:
## DATABASE_HOST: airbyte-db
## DATABASE_PORT: 5432
env_vars: {}
## @section Worker Parameters
worker:
enabled: true
## worker.replicaCount Number of worker replicas
replicaCount: 1
## worker.image.repository The repository to use for the airbyte worker image.
## worker.image.pullPolicy the pull policy to use for the airbyte worker image
## worker.image.tag The airbyte worker image tag. Defaults to the chart's AppVersion
image:
repository: airbyte/worker
pullPolicy: IfNotPresent
## worker.podAnnotations [object] Add extra annotations to the worker pod(s)
##
podAnnotations: {}
## worker.podLabels [object] Add extra labels to the worker pod(s)
##
podLabels: {}
## worker.containerSecurityContext Security context for the container
## Examples:
## containerSecurityContext:
## runAsNonRoot: true
## runAsUser: 1000
## readOnlyRootFilesystem: true
containerSecurityContext: {}
## Configure extra options for the worker containers' liveness and readiness probes
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#configure-probes
## worker.livenessProbe.enabled Enable livenessProbe on the worker
## worker.livenessProbe.initialDelaySeconds Initial delay seconds for livenessProbe
## worker.livenessProbe.periodSeconds Period seconds for livenessProbe
## worker.livenessProbe.timeoutSeconds Timeout seconds for livenessProbe
## worker.livenessProbe.failureThreshold Failure threshold for livenessProbe
## worker.livenessProbe.successThreshold Success threshold for livenessProbe
##
livenessProbe:
enabled: false
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 1
failureThreshold: 3
successThreshold: 1
## worker.readinessProbe.enabled Enable readinessProbe on the worker
## worker.readinessProbe.initialDelaySeconds Initial delay seconds for readinessProbe
## worker.readinessProbe.periodSeconds Period seconds for readinessProbe
## worker.readinessProbe.timeoutSeconds Timeout seconds for readinessProbe
## worker.readinessProbe.failureThreshold Failure threshold for readinessProbe
## worker.readinessProbe.successThreshold Success threshold for readinessProbe
##
readinessProbe:
enabled: false
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 1
failureThreshold: 3
successThreshold: 1
## worker resource requests and limits
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
## We usually recommend not to specify default resources and to leave this as a conscious
## choice for the user. This also increases chances charts run on environments with little
## resources, such as Minikube. If you do want to specify resources, uncomment the following
## lines, adjust them as necessary, and remove the curly braces after 'resources:'.
## worker.resources.limits [object] The resources limits for the worker container
## worker.resources.requests [object] The requested resources for the worker container
resources:
## Example:
## limits:
## cpu: 200m
## memory: 1Gi
limits: {}
## Examples:
## requests:
## memory: 256Mi
## cpu: 250m
requests: {}
## worker.nodeSelector [object] Node labels for pod assignment
## Ref: https://kubernetes.io/docs/user-guide/node-selection/
##
nodeSelector: {}
## worker.tolerations [array] Tolerations for worker pod assignment.
## ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
##
# tolerations: []
tolerations:
- key: "dedicated"
operator: "Equal"
value: "airbyte_core"
effect: "NoSchedule"
## worker.affinity [object] Affinity and anti-affinity for worker pod assignment.
## ref: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity
##
# affinity: {}
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: dedicated
operator: In
values:
- airbyte_core
## worker.log.level The log level to log at.
log:
level: "INFO"
## worker.extraEnv [array] Additional env vars for worker pod(s).
## Example:
##
extraEnv:
- name: JOB_KUBE_TOLERATIONS
value: "key=dedicated,operator=Equal,effect=NoSchedule,value=sync_workers"
- name: JOB_KUBE_NODE_SELECTORS
value: "dedicated=sync_workers"
- name: DISCOVER_JOB_KUBE_NODE_SELECTORS
value: "dedicated=sync_workers"
- name: CHECK_JOB_KUBE_NODE_SELECTORS
value: "dedicated=sync_workers"
- name: SPEC_JOB_KUBE_NODE_SELECTORS
value: "dedicated=sync_workers"
- name: MAX_SYNC_WORKERS
value: "50"
- name: MAX_CHECK_WORKERS
value: "50"
- name: MAX_DISCOVERY_WORKERS
value: "50"
- name: MAX_SPEC_WORKERS
value: "50"
# - name: USE_STREAM_CAPABLE_STATE
# value: "true"
# - name: SUBMITTER_NUM_THREADS
# value: "200"
# - name: MAX_SYNC_WORKERS
# value: "100"
# extraEnv: []
## worker.extraVolumeMounts [array] Additional volumeMounts for worker container(s).
## Examples (when using `worker.containerSecurityContext.readOnlyRootFilesystem=true`):
## extraVolumeMounts:
## - name: tmpdir
## mountPath: /tmp
##
extraVolumeMounts: []
## worker.extraVolumes [array] Additional volumes for worker pod(s).
## Examples (when using `worker.containerSecurityContext.readOnlyRootFilesystem=true`):
## extraVolumes:
## - name: tmpdir
## emptyDir: {}
##
extraVolumes: []
extraContainers: []
hpa:
enabled: false
debug:
enabled: false
## Container Orchestrator
## enabled -- Enable or disable Orchestrator
## image -- Orchestrator image
containerOrchestrator:
enabled: false
image: ""
## current no exist documentations
activityMaxAttempt: ""
activityInitialDelayBetweenAttemptsSeconds: ""
activityMaxDelayBetweenAttemptsSeconds: ""
maxNotifyWorkers: 5
## @section Metrics parameters
metrics:
enabled: true
## metrics.replicaCount Number of metrics-reporter replicas
replicaCount: 1
## metrics.image.repository The repository to use for the airbyte metrics-reporter image.
## metrics.image.pullPolicy the pull policy to use for the airbyte metrics-reporter image
## metrics.image.tag The airbyte metrics-reporter image tag. Defaults to the chart's AppVersion
image:
repository: airbyte/metrics-reporter
pullPolicy: IfNotPresent
## metrics.podAnnotations [object] Add extra annotations to the metrics-reporter pod
##
podAnnotations: {}
## metrics.podLabels [object] Add extra labels to the metrics-reporter pod
##
podLabels: {}
## metrics.containerSecurityContext Security context for the container
## Examples:
## containerSecurityContext:
## runAsNonRoot: true
## runAsUser: 1000
## readOnlyRootFilesystem: true
containerSecurityContext: {}
## metrics-reporter app resource requests and limits
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
## We usually recommend not to specify default resources and to leave this as a conscious
## choice for the user. This also increases chances charts run on environments with little
## resources, such as Minikube. If you do want to specify resources, uncomment the following
## lines, adjust them as necessary, and remove the curly braces after 'resources:'.
## metrics.resources.limits [object] The resources limits for the metrics-reporter container
## metrics.resources.requests [object] The requested resources for the metrics-reporter container
resources:
## Example:
## limits:
## cpu: 200m
## memory: 1Gi
limits: {}
## Examples:
## requests:
## memory: 256Mi
## cpu: 250m
requests: {}
## metrics.nodeSelector [object] Node labels for pod assignment
## Ref: https://kubernetes.io/docs/user-guide/node-selection/
##
nodeSelector: {}
## metrics.tolerations [array] Tolerations for metrics-reporter pod assignment.
## ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
##
# tolerations: []
tolerations:
- key: "dedicated"
operator: "Equal"
value: "airbyte_core"
effect: "NoSchedule"
## metrics.affinity [object] Affinity and anti-affinity for metrics-reporter pod assignment.
## ref: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity
##
# affinity: {}
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: dedicated
operator: In
values:
- airbyte_core
## metrics.extraEnv [array] Additional env vars for metrics-reporter pod(s).
## Example:
##
## extraEnv:
## - name: SAMPLE_ENV_VAR
## value: "key=sample-value"
extraEnv:
- name: PUBLISH_METRICS
value: "true"
## metrics.extraVolumeMounts [array] Additional volumeMounts for metrics-reporter container(s).
## Examples (when using `metrics.containerSecurityContext.readOnlyRootFilesystem=true`):
## extraVolumeMounts:
## - name: tmpdir
## mountPath: /tmp
##
extraVolumeMounts: []
## metrics.extraVolumes [array] Additional volumes for metrics-reporter pod(s).
## Examples (when using `metrics.containerSecurityContext.readOnlyRootFilesystem=true`):
## extraVolumes:
## - name: tmpdir
## emptyDir: {}
##
extraVolumes: []
extraContainers: []
secrets: {}
env_vars: {}
## @section Bootloader Parameters
airbyte-bootloader:
enabled: true
## image.repository The repository to use for the airbyte bootloader image.
## image.pullPolicy the pull policy to use for the airbyte bootloader image
## image.tag The airbyte bootloader image tag. Defaults to the chart's AppVersion
image:
repository: airbyte/bootloader
pullPolicy: IfNotPresent
## podAnnotations [object] Add extra annotations to the bootloader pod
##
podAnnotations: {}
## podLabels [object] Add extra labels to the bootloader pod
##
podLabels: {}
## nodeSelector [object] Node labels for pod assignment
## Ref: https://kubernetes.io/docs/user-guide/node-selection/
##
nodeSelector: {}
## tolerations [array] Tolerations for worker pod assignment.
## ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
##
# tolerations: []
tolerations:
- key: "dedicated"
operator: "Equal"
value: "airbyte_core"
effect: "NoSchedule"
## Bootloader resource requests and limits
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
## We usually recommend not to specify default resources and to leave this as a conscious
## choice for the user. This also increases chances charts run on environments with little
## resources, such as Minikube. If you do want to specify resources, uncomment the following
## lines, adjust them as necessary, and remove the curly braces after 'resources:'.
## bootloader.resources.limits [object] The resources limits for the airbyte bootloader image
## bootloader.resources.requests [object] The requested resources for the airbyte bootloader image
resources:
## Example:
## requests:
## memory: 256Mi
## cpu: 250m
requests: {}
## Example:
## limits:
## cpu: 200m
## memory: 1Gi
limits: {}
## affinity [object] Affinity and anti-affinity for bootloader pod assignment.
## ref: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity
##
# affinity: {}
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: dedicated
operator: In
values:
- airbyte_core
## extraEnv [array] Supply extra env variables to main container using full notation
## Example: (With default env vars and values taken from generated config map)
## extraEnv:
## - name: AIRBYTE_VERSION
## valueFrom:
## configMapKeyRef:
## name: airbyte-env
## key: AIRBYTE_VERSION
## - name: DATABASE_HOST
## valueFrom:
## configMapKeyRef:
## name: airbyte-env
## key: DATABASE_HOST
## - name: DATABASE_PORT
## valueFrom:
## configMapKeyRef:
## name: airbyte-env
## key: DATABASE_PORT
## - name: DATABASE_PASSWORD
## valueFrom:
## secretKeyRef:
## name: airbyte-secrets
## key: DATABASE_PASSWORD
## - name: DATABASE_URL
## valueFrom:
## configMapKeyRef:
## name: airbyte-env
## key: DATABASE_URL
## - name: DATABASE_USER
## valueFrom:
## secretKeyRef:
## name: airbyte-secrets
## key: DATABASE_USER
##
extraEnv: []
## secrets [object] Supply additional secrets to container
## Example:
## secrets:
## DATABASE_PASSWORD: strong-password
## DATABASE_USER: my-db-user
secrets: {}
## env_vars [object] Supply extra env variables to main container using simplified notation
## Example:
## env_vars:
## DATABASE_HOST: airbyte-db
## DATABASE_PORT: 5432
env_vars: {}
## extraContainer [array] Additional container for server pod(s)
## Example:
# extraContainers:
# - name: otel_collector
# image: somerepo/someimage:sometag
# args: [
# "--important-args"
# ]
# ports:
# - containerPort: 443
# volumeMounts:
# - name: volumeMountCool
# mountPath: /some/path
# readOnly: true
extraContainers: []
## extraInitContainers [array] Additional init containers for server pod(s)
## Example:
# extraInitContainers:
# - name: sleepy
# image: alpine
# command: ['sleep', '60']
extraInitContainers: []
## extraVolumeMounts [array] Additional volumeMounts for server container(s).
## Examples (when using `containerSecurityContext.readOnlyRootFilesystem=true`):
## extraVolumeMounts:
## - name: tmpdir
## mountPath: /tmp
##
extraVolumeMounts: []
## extraVolumes [array] Additional volumes for server pod(s).
## Examples (when using `containerSecurityContext.readOnlyRootFilesystem=true`):
## extraVolumes:
## - name: tmpdir
## emptyDir: {}
##
extraVolumes: []
## @section Temporal parameters
## TODO: Move to consuming temporal from a dedicated helm chart
temporal:
enabled: true
## temporal.replicaCount The number of temporal replicas to deploy
replicaCount: 1
## temporal.image.repository The temporal image repository to use
## temporal.image.pullPolicy The pull policy for the temporal image
## temporal.image.tag The temporal image tag to use
image:
repository: temporalio/auto-setup
pullPolicy: IfNotPresent
tag: "1.20.1"
## temporal.service.type The Kubernetes Service Type
## temporal.service.port The temporal port and exposed kubernetes port
service:
type: ClusterIP
port: 7233
## temporal.podAnnotations [object] Add extra annotations to the temporal pod
##
podAnnotations: {}
## temporal.podLabels [object] Add extra labels to the temporal pod
##
podLabels: {}
## temporal.containerSecurityContext Security context for the container
## Examples:
## containerSecurityContext:
## runAsNonRoot: true
## runAsUser: 1000
## readOnlyRootFilesystem: true
containerSecurityContext: {}
## temporal.extraInitContainers Additional InitContainers to initialize the pod
## Examples (when using `temporal.containerSecurityContext.readOnlyRootFilesystem=true`):
## extraInitContainers:
## - name: config-loader
## image: temporalio/auto-setup:1.20.1
## command:
## - /bin/sh
## - -c
## - >-
## find /etc/temporal/config/ -maxdepth 1 -mindepth 1 -exec cp -ar {} /config/ \;
## volumeMounts:
## - name: config
## mountPath: /config
extraInitContainers: []
## Configure extra options for the temporal containers' liveness and readiness probes
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#configure-probes
## temporal.livenessProbe.enabled Enable livenessProbe on the temporal
## temporal.livenessProbe.initialDelaySeconds Initial delay seconds for livenessProbe
## temporal.livenessProbe.periodSeconds Period seconds for livenessProbe
## temporal.livenessProbe.timeoutSeconds Timeout seconds for livenessProbe
## temporal.livenessProbe.failureThreshold Failure threshold for livenessProbe
## temporal.livenessProbe.successThreshold Success threshold for livenessProbe
##
livenessProbe:
enabled: false
## temporal.readinessProbe.enabled Enable readinessProbe on the temporal
## temporal.readinessProbe.initialDelaySeconds Initial delay seconds for readinessProbe
## temporal.readinessProbe.periodSeconds Period seconds for readinessProbe
## temporal.readinessProbe.timeoutSeconds Timeout seconds for readinessProbe
## temporal.readinessProbe.failureThreshold Failure threshold for readinessProbe
## temporal.readinessProbe.successThreshold Success threshold for readinessProbe
##
readinessProbe:
enabled: false
## temporal.nodeSelector [object] Node labels for temporal pod assignment
## Ref: https://kubernetes.io/docs/user-guide/node-selection/
##
nodeSelector: {}
## temporal.tolerations [array] Tolerations for temporal pod assignment.
## ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
##
# tolerations: []
tolerations:
- key: "dedicated"
operator: "Equal"
value: "airbyte_core"
effect: "NoSchedule"
## temporal.affinity [object] Affinity and anti-affinity for temporal pod assignment.
## ref: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity
##
# affinity: {}
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: dedicated
operator: In
values:
- airbyte_core
## temporal.extraEnv [array] Additional env vars for temporal pod(s).
## Example:
##
## extraEnv:
## - name: SAMPLE_ENV_VAR
## value: "key=sample-value"
extraEnv: []
## temporal.extraVolumeMounts [array] Additional volumeMounts for temporal container(s).
## Examples (when using `temporal.containerSecurityContext.readOnlyRootFilesystem=true`):
## extraVolumeMounts:
## - name: tmpdir
## mountPath: /tmp
## - name: config
## mountPath: /etc/temporal/config
##
extraVolumeMounts: []
## temporal.extraVolumes [array] Additional volumes for temporal pod(s).
## Examples (when using `temporal.containerSecurityContext.readOnlyRootFilesystem=true`):
## extraVolumes:
## - name: tmpdir
## emptyDir: {}
## - name: config
## emptyDir: {}
##
extraVolumes: []
## Temporal resource requests and limits
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
## We usually recommend not to specify default resources and to leave this as a conscious
## choice for the user. This also increases chances charts run on environments with little
## resources, such as Minikube. If you do want to specify resources, uncomment the following
## lines, adjust them as necessary, and remove the curly braces after 'resources:'.
## temporal.resources.limits [object] The resources limits for temporal pod(s)
## temporal.resources.requests [object] The requested resources for temporal pod(s)
resources:
## Example:
## requests:
## memory: 256Mi
## cpu: 250m
requests: {}
## Example:
## limits:
## cpu: 200m
## memory: 1Gi
limits: {}
extraContainers: []
## @section Airbyte Database parameters
## PostgreSQL chart configuration
## ref: https://github.com/bitnami/charts/blob/master/bitnami/postgresql/values.yaml
## postgresql.enabled Switch to enable or disable the PostgreSQL helm chart
## postgresql.postgresqlUsername Airbyte Postgresql username
## postgresql.postgresqlPassword Airbyte Postgresql password
## postgresql.postgresqlDatabase Airbyte Postgresql database
## postgresql.existingSecret Name of an existing secret containing the PostgreSQL password ('postgresql-password' key)
## postgresql.containerSecurityContext.runAsNonRoot Ensures the container will run with a non-root user
## postgresql.commonAnnotations.helm.sh/hook It will determine when the hook should be rendered
## postgresql.commonAnnotations.helm.sh/hook-weight The order in which the hooks are executed. If weight is lower, it has higher priority
##
postgresql:
enabled: false
## image.repository Repository for airbyte-db statefulset
image:
repository: airbyte/db
postgresqlUsername: airbyte
postgresqlPassword: airbyte
postgresqlDatabase: db-airbyte
# fullnameOverride: *db-hostname
## This secret is used in case of postgresql.enabled=true and we would like to specify password for newly created postgresql instance
##
existingSecret: ""
containerSecurityContext:
runAsNonRoot: true
commonAnnotations:
helm.sh/hook: pre-install,pre-upgrade
helm.sh/hook-weight: "-1"
## External PostgreSQL configuration
## All of these values are only used when postgresql.enabled is set to false
## externalDatabase.host Database host
## externalDatabase.user non-root Username for Airbyte Database
## externalDatabase.password Database password
## externalDatabase.existingSecret Name of an existing secret resource containing the DB password
## externalDatabase.existingSecretPasswordKey Name of an existing secret key containing the DB password
## externalDatabase.database Database name
## externalDatabase.port Database port number
## externalDatabase.jdbcUrl Database full JDBL URL (ex: jdbc:postgresql://host:port/db?parameters)
##
externalDatabase:
host: "psql.us-east-2.rds.amazonaws.com"
user: "postgres_user"
password: "pswd"
existingSecret: ""
existingSecretPasswordKey: ""
database: airbyte
port: 5432
jdbcUrl: "jdbc:postgresql://psqludakk.us-east-2.rds.amazonaws.com:5432/airbyte"
## @section Logs parameters
## @section Minio chart overwrites
## minio.accessKey.password Minio Access Key
## minio.secretKey.password Minio Secret Key
minio:
enabled: false
## image.repository Minio image used by Minio helm chart
## image.tag Minio tag image
image:
repository: minio/minio
tag: latest
auth:
rootUser: minio
rootPassword: minio123
storage:
volumeClaimValue: 500Mi
## @section cron parameters
cron:
enabled: false
## cron.replicaCount Number of cron replicas
replicaCount: 1
## cron.image.repository The repository to use for the airbyte cron image.
## cron.image.pullPolicy the pull policy to use for the airbyte cron image
## cron.image.tag The airbyte cron image tag. Defaults to the chart's AppVersion
image:
repository: airbyte/cron
pullPolicy: IfNotPresent
## cron.podAnnotations [object] Add extra annotations to the cron pod(s)
##
podAnnotations: {}
## cron.podLabels [object] Add extra labels to the cron pod(s)
##
podLabels: {}
## cron.containerSecurityContext Security context for the container
## Examples:
## containerSecurityContext:
## runAsNonRoot: true
## runAsUser: 1000
## readOnlyRootFilesystem: true
containerSecurityContext: {}
## Configure extra options for the cron containers' liveness and readiness probes
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#configure-probes
## cron.livenessProbe.enabled Enable livenessProbe on the cron
## cron.livenessProbe.initialDelaySeconds Initial delay seconds for livenessProbe
## cron.livenessProbe.periodSeconds Period seconds for livenessProbe
## cron.livenessProbe.timeoutSeconds Timeout seconds for livenessProbe
## cron.livenessProbe.failureThreshold Failure threshold for livenessProbe
## cron.livenessProbe.successThreshold Success threshold for livenessProbe
##
livenessProbe:
enabled: true
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 1
failureThreshold: 3
successThreshold: 1
## cron.readinessProbe.enabled Enable readinessProbe on the cron
## cron.readinessProbe.initialDelaySeconds Initial delay seconds for readinessProbe
## cron.readinessProbe.periodSeconds Period seconds for readinessProbe
## cron.readinessProbe.timeoutSeconds Timeout seconds for readinessProbe
## cron.readinessProbe.failureThreshold Failure threshold for readinessProbe
## cron.readinessProbe.successThreshold Success threshold for readinessProbe
##
readinessProbe:
enabled: true
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 1
failureThreshold: 3
successThreshold: 1
## cron resource requests and limits
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
## We usually recommend not to specify default resources and to leave this as a conscious
## choice for the user. This also increases chances charts run on environments with little
## resources, such as Minikube. If you do want to specify resources, uncomment the following
## lines, adjust them as necessary, and remove the curly braces after 'resources:'.
## cron.resources.limits [object] The resources limits for the cron container
## cron.resources.requests [object] The requested resources for the cron container
resources:
## Example:
## limits:
## cpu: 200m
## memory: 1Gi
limits: {}
## Examples:
## requests:
## memory: 256Mi
## cpu: 250m
requests: {}
## cron.nodeSelector [object] Node labels for pod assignment
## Ref: https://kubernetes.io/docs/user-guide/node-selection/
##
nodeSelector: {}
## cron.tolerations [array] Tolerations for cron pod assignment.
## ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
##
# tolerations: []
tolerations:
- key: "dedicated"
operator: "Equal"
value: "airbyte_core"
effect: "NoSchedule"
## cron.affinity [object] Affinity and anti-affinity for cron pod assignment.
## ref: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity
##
# affinity: {}
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: dedicated
operator: In
values:
- airbyte_core
## cron.log.level The log level to log at.
log:
level: "INFO"
## cron.extraVolumeMounts [array] Additional volumeMounts for cron container(s).
## Examples (when using `cron.containerSecurityContext.readOnlyRootFilesystem=true`):
## extraVolumeMounts:
## - name: tmpdir
## mountPath: /tmp
##
extraVolumeMounts: []
## cron.extraVolumes [array] Additional volumes for cron pod(s).
## Examples (when using `cron.containerSecurityContext.readOnlyRootFilesystem=true`):
## extraVolumes:
## - name: tmpdir
## emptyDir: {}
##
extraVolumes: []
## extraContainer [array] Additional container for cron pod(s)
## Example:
# extraContainers:
# - name: otel_collector
# image: somerepo/someimage:sometag
# args: [
# "--important-args"
# ]
# ports:
# - containerPort: 443
# volumeMounts:
# - name: volumeMountCool
# mountPath: /some/path
# readOnly: true
extraContainers: []
## extraInitContainers [array] Additional init containers for cron pod(s)
## Example:
# extraInitContainers:
# - name: sleepy
# image: alpine
# command: ['sleep', '60']
extraInitContainers: []
## extraEnv [array] Supply extra env variables to main container using full notation
## Example: (With default env vars and values taken from generated config map)
#extraEnv:
# - name: AIRBYTE_VERSION
# valueFrom:
# configMapKeyRef:
# name: airbyte-env
# key: AIRBYTE_VERSION
# - name: API_URL
# valueFrom:
# configMapKeyRef:
# name: airbyte-env
# key: API_URL
# - name: WEBAPP_URL
# valueFrom:
# configMapKeyRef:
# name: airbyte-env
# key: WEBAPP_URL
# - name: TRACKING_STRATEGY
# valueFrom:
# configMapKeyRef:
# name: airbyte-env
# key: TRACKING_STRATEGY
# - name: FULLSTORY
# valueFrom:
# configMapKeyRef:
# name: airbyte-env
# key: FULLSTORY
# - name: INTERNAL_API_HOST
# valueFrom:
# configMapKeyRef:
# name: airbyte-env
# key: INTERNAL_API_HOST
extraEnv: []
## secrets [object] Supply additional secrets to container
## Example:
## secrets:
## DATABASE_PASSWORD: strong-password
## DATABASE_USER: my-db-user
secrets: {}
## env_vars [object] Supply extra env variables to main container using simplified notation
## Example:
## env_vars:
## DATABASE_HOST: airbyte-db
## DATABASE_PORT: 5432
env_vars: {}
connector-builder-server:
enabled: true
## connector-builder-server.replicaCount Number of connector-builder-server replicas
replicaCount: 1
## connector-builder-server.image.repository The repository to use for the airbyte connector-builder-server image.
## connector-builder-server.image.pullPolicy the pull policy to use for the airbyte connector-builder-server image
## connector-builder-server.image.tag The airbyte connector-builder-server image tag. Defaults to the chart's AppVersion
image:
repository: airbyte/connector-builder-server
pullPolicy: IfNotPresent
## Configure extra options for the server containers' liveness and readiness probes
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#configure-probes
## server.livenessProbe.enabled Enable livenessProbe on the server
## server.livenessProbe.initialDelaySeconds Initial delay seconds for livenessProbe
## server.livenessProbe.periodSeconds Period seconds for livenessProbe
## server.livenessProbe.timeoutSeconds Timeout seconds for livenessProbe
## server.livenessProbe.failureThreshold Failure threshold for livenessProbe
## server.livenessProbe.successThreshold Success threshold for livenessProbe
##
livenessProbe:
enabled: false
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 10
failureThreshold: 3
successThreshold: 1
## server.readinessProbe.enabled Enable readinessProbe on the server
## server.readinessProbe.initialDelaySeconds Initial delay seconds for readinessProbe
## server.readinessProbe.periodSeconds Period seconds for readinessProbe
## server.readinessProbe.timeoutSeconds Timeout seconds for readinessProbe
## server.readinessProbe.failureThreshold Failure threshold for readinessProbe
## server.readinessProbe.successThreshold Success threshold for readinessProbe
##
readinessProbe:
enabled: false
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 10
failureThreshold: 3
successThreshold: 1
#Toleration and affinity
tolerations:
- key: "dedicated"
operator: "Equal"
value: "airbyte_core"
effect: "NoSchedule"
## webapp.affinity [object] Affinity and anti-affinity for webapp pod assignment.
## ref: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity
##
# affinity: {}
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: dedicated
operator: In
values:
- airbyte_core
## connector-builder-server resource requests and limits
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
## We usually recommend not to specify default resources and to leave this as a conscious
## choice for the user. This also increases chances charts run on environments with little
## resources, such as Minikube. If you do want to specify resources, uncomment the following
## lines, adjust them as necessary, and remove the curly braces after 'resources:'.
## connector-builder-server.resources.limits [object] The resources limits for the connector-builder-server container
## connector-builder-server.resources.requests [object] The requested resources for the connector-builder-server container
resources:
## Example:
## limits:
## cpu: 200m
## memory: 1Gi
limits: {}
## Examples:
## requests:
## memory: 256Mi
## cpu: 250m
requests: {}
## connector-builder-server.log.level The log level to log at.
log:
level: "INFO"
env_vars: {}
service:
port: 80
airbyte-api-server:
enabled: true
## airbyte-api-server.replicaCount Number of airbyte-api-server replicas
replicaCount: 1
## airbyte-api-server.image.repository The repository to use for the airbyte airbyte-api-server image.
## airbyte-api-server.image.pullPolicy the pull policy to use for the airbyte airbyte-api-server image
## airbyte-api-server.image.tag The airbyte airbyte-api-server image tag. Defaults to the chart's AppVersion
image:
repository: airbyte/airbyte-api-server
pullPolicy: IfNotPresent
## Configure extra options for the server containers' liveness and readiness probes
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#configure-probes
## server.livenessProbe.enabled Enable livenessProbe on the server
## server.livenessProbe.initialDelaySeconds Initial delay seconds for livenessProbe
## server.livenessProbe.periodSeconds Period seconds for livenessProbe
## server.livenessProbe.timeoutSeconds Timeout seconds for livenessProbe
## server.livenessProbe.failureThreshold Failure threshold for livenessProbe
## server.livenessProbe.successThreshold Success threshold for livenessProbe
##
livenessProbe:
enabled: false
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 10
failureThreshold: 3
successThreshold: 1
## server.readinessProbe.enabled Enable readinessProbe on the server
## server.readinessProbe.initialDelaySeconds Initial delay seconds for readinessProbe
## server.readinessProbe.periodSeconds Period seconds for readinessProbe
## server.readinessProbe.timeoutSeconds Timeout seconds for readinessProbe
## server.readinessProbe.failureThreshold Failure threshold for readinessProbe
## server.readinessProbe.successThreshold Success threshold for readinessProbe
##
readinessProbe:
enabled: false
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 10
failureThreshold: 3
successThreshold: 1
## airbyte-api-server resource requests and limits
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
## We usually recommend not to specify default resources and to leave this as a conscious
## choice for the user. This also increases chances charts run on environments with little
## resources, such as Minikube. If you do want to specify resources, uncomment the following
## lines, adjust them as necessary, and remove the curly braces after 'resources:'.
## airbyte-api-server.resources.limits [object] The resources limits for the airbyte-api-server container
## airbyte-api-server.resources.requests [object] The requested resources for the airbyte-api-server container
resources:
## Example:
## limits:
## cpu: 200m
## memory: 1Gi
limits: {}
## Examples:
## requests:
## memory: 256Mi
## cpu: 250m
requests: {}
# Toleration and affinity
tolerations:
- key: "dedicated"
operator: "Equal"
value: "airbyte_core"
effect: "NoSchedule"
## webapp.affinity [object] Affinity and anti-affinity for webapp pod assignment.
## ref: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity
##
# affinity: {}
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: dedicated
operator: In
values:
- airbyte_core
## airbyte-api-server.log.level The log level to log at.
log:
level: "INFO"
env_vars: {}
service:
port: 80
## Configure the ingress resource that allows you to access the Airbyte API.
## ref: http://kubernetes.io/docs/user-guide/ingress/
## webapp.ingress.enabled Set to true to enable ingress record generation
## webapp.ingress.className Specifies ingressClassName for clusters >= 1.18+
## webapp.ingress.annotations [object] Ingress annotations done as key:value pairs
## webapp.ingress.hosts The list of hostnames to be covered with this ingress record.
## webapp.ingress.tls [array] Custom ingress TLS configuration
ingress:
enabled: false
className: ""
annotations: {}
# kubernetes.io/ingress.class: nginx
# kubernetes.io/tls-acme: "true"
hosts: []
# - host: chart-example.local
# paths:
# - path: /
# pathType: ImplementationSpecific
tls: []
# - secretName: chart-example-tls
# hosts:
# - chart-example.local
keycloak:
enabled: false
auth:
adminUsername: airbyteAdmin
adminPassword: keycloak123
keycloak-setup:
enabled: false
otel:
name: otel-collector
promLabel: kube-prometheus-stack
replicas: 1
config:
receivers:
otlp:
protocols:
grpc:
endpoint: "0.0.0.0:4317"
processors:
batch:
memory_limiter:
limit_mib: 1500
spike_limit_mib: 512
check_interval: 5s
exporters:
prometheus:
endpoint: "0.0.0.0:8889"
namespace: "default"
service:
pipelines:
metrics:
receivers: [otlp]
processors: [memory_limiter]
exporters: [prometheus]
dashboard:
labelValue: 1
grafanaFolder: AIRBYTE
namespace: monitoring
from airbyte.
In my case the reason was memory limits on sync jobs. We have identified the issue, Actually sync pods were failing with exit code 137, which means they had memory pressure. (although pods were not in OOMkilled state -- nvm)
Initially we had applied this resource quota on airbyte jobs:
global:
jobs:
resources:
requests:
cpu: 100m
memory: 25Mi
limits:
cpu: 200m
memory: 50Mi
We updated the resource quota to the following and our sync pods started working fine.
global:
jobs:
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: '200m'
memory: 1Gi
[Resolved]
from airbyte.
Related Issues (20)
- [placeholder] spike for implementing RFR in python cdk
- replace all instances of AirbyteLogger
- Storybook site (components.airbyte.dev) down, no DNS records HOT 1
- airbyte-ci tool using broken dependency HOT 3
- Bake in latest CDK versions at build time (in addition to on boot + cron)
- [source-facebook-marketing] Ad Insights schema missing video_thruplay_watched_actions field HOT 1
- [source-klarna] incorrect config.region values: change 'us' --> 'na' HOT 1
- Temporal v1.23.0 fails to start with error: Unable to create SQL database, but v1.20.1 works fine. HOT 3
- [source-zendesk-support] Please switch to Cursor Pagination at the earliest HOT 1
- [helm] Minio create bucket : permission denied HOT 4
- [source-stripe] 100% of CPU and fails after a few hours HOT 1
- Revisit pagerduty alerts
- Getting error "Could not resolve placeholder ${AIRBYTE_API_HOST}" when trying to call the API HOT 1
- Weird issue where files are not pushed to s3 HOT 2
- Datadog metric for trace errors on DeclarativeSourcesUpdater
- Replace LocalManifestDefinitionImageVersionsProvider: default to build version?
- [destination-bigquery] java.lang.RuntimeException: Unable to deserialize PartialAirbyteMessage. HOT 2
- Snowflake destination automatic clustering turned on HOT 1
- [source-MySQL] Can't select incremental sync in CDC-based connection HOT 6
- [source-shopify] `product` stream field `tags` Data mismatch only the first tag is returned HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from airbyte.