keitaroinc / ckan-helm Goto Github PK

Using the helm chart, in the shared directory /var/lib/ckan/default, I have my src folder with my own custom plugins.
Each custom plugin contains a structure like the following:

drwxr-x---    2 ckan     ckan             0 Mar 20 07:35 .
drwxr-x---    2 ckan     ckan             0 Mar 20 07:35 ..
-rw-r-----    1 ckan     ckan            67 Mar 20 07:35 .coveragerc
drwxr-x---    2 ckan     ckan             0 Mar 20 07:35 .git
drwxr-x---    2 ckan     ckan             0 Mar 20 07:35 .github
-rw-r-----    1 ckan     ckan           607 Mar 20 07:35 .gitignore
-rw-r-----    1 ckan     ckan         34500 Mar 20 07:35 LICENSE
-rw-r-----    1 ckan     ckan           232 Mar 20 07:35 MANIFEST.in
-rw-r-----    1 ckan     ckan          3413 Mar 20 07:35 README.md
drwxr-x---    2 ckan     ckan             0 Mar 20 07:35 build
drwxr-x---    2 ckan     ckan             0 Mar 20 07:35 ckanext
drwxr-x---    2 ckan     ckan             0 Mar 20 07:35 ckanext_mypersonal_theme.egg-info
-rw-r-----    1 ckan     ckan            12 Mar 20 07:35 dev-requirements.txt
drwxr-x---    2 ckan     ckan             0 Mar 20 07:35 dist
-rw-r-----    1 ckan     ckan             0 Mar 20 07:35 requirements.txt
-rw-r-----    1 ckan     ckan          1803 Mar 20 07:35 setup.cfg
-rw-r-----    1 ckan     ckan           662 Mar 20 07:35 setup.py
-rw-r-----    1 ckan     ckan           797 Mar 20 07:35 test.ini
-rw-r-----    1 ckan     ckan           799 Mar 20 07:35 test.ini.save

Following the README of the container, I have executed the following script to be able to install it (AFTER HELM CHART IS DEPLOYED):

pip install --no-index --find-links=ckanext-mypersonal_theme mypersonal_theme && ckan config-tool ${APP_DIR}/production.ini "ckan.plugins = envvars activity datastore scheming_datasets scheming_groups scheming_organizations mypersonal_theme datapusher"

However, I am getting the following error:

Defaulting to user installation because normal site-packages is not writeable
Looking in links: ckanext-mypersonal_theme/
WARNING: Location 'ckanext-mypersonal_theme/' is ignored: it is either a non-existing path or lacks a specific scheme.
ERROR: Could not find a version that satisfies the requirement mypersonal_theme (from versions: none)
ERROR: No matching distribution found for mypersonal_theme

Should I do any step before this one? Or what I am doing wrong?

We use ingressClassName in our cluster. I added the following lines to the ingress template under spec:
{{- if .Values.ingress.className }}
ingressClassName: {{ .Values.ingress.className | quote }}
{{- end }}

According to the official CKAN 2.9 documentation, the CKAN configuration file is named ckan.ini from version 2.9 onwards.

The previous names (production.ini or development.ini etc.) will be phased out.

The psql-init/psql-init.py script still uses the production.ini file:

This might fail (as it did for me) depending on the Image and configuration you use.

I propose the implementation of one of following alternatives:

• at least check whether the file exists or catch the exception thrown by subprocess.check_output(...)
• first try the ckan.ini file, then the others if it fails
• make the filename settable via an environment variable

What do you think? Would be happy to implement the change..

I have installed ckan in OKE of OCI but getting the psql init job failing.
The logs give:

File "/srv/app/src/ckan/ckan/cli/cli.py", line 135, in _add_ctx_object\n ctx.obj = CtxObject(path)\n File "/srv/app/src/ckan/ckan/cli/cli.py", line 57, in __init__\n self.app = make_app(self.config)\n File "/srv/app/src/ckan/ckan/config/middleware/__init__.py", line 58, in make_app\n flask_app = make_flask_stack(conf)\n File "/srv/app/src/ckan/ckan/config/middleware/flask_app.py", line 168, in make_flask_stack\n raise RuntimeError(u\'You must provide a value for the secret key\'\nRuntimeError: You must provide a value for the secret key with the SECRET_KEY config option\n'

helm install ckan-p keitaro-charts/ckan

solr-0 fails to start.

kubectl logs solr-0
solr 18:29:37.39
solr 18:29:37.39 Welcome to the Bitnami solr container
solr 18:29:37.39 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-solr
solr 18:29:37.39 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-solr/issues
solr 18:29:37.39
solr 18:29:37.39 INFO ==> ** Starting solr setup **
solr 18:29:37.40 INFO ==> Validating settings in SOLR_* env vars...
solr 18:29:37.40 ERROR ==> Not enough nodes for the replicas and shards indicated

is there any workaround?

The README references several charts as dependencies

Looking at the chart.yaml file those dependencies are:
dependencies:

• name: redis
  version: 16.5.5
  repository: "https://keitaro-charts.storage.googleapis.com"
  condition: redis.enabled
• name: solr
  version: 4.1.2
  repository: "https://keitaro-charts.storage.googleapis.com"
  condition: solr.enabled
• name: postgresql
  version: 10.16.2
  repository: "https://keitaro-charts.storage.googleapis.com"
  condition: postgresql.enabled
• name: datapusher
  version: 1.0.0
  repository: "https://keitaro-charts.storage.googleapis.com"
  condition: datapusher.enabled

My assumption is that the "https://keitaro-charts.storage.googleapis.com" somehow mirrors the referenced dependencies, but as a user there is no way to verify that. I recommend updating the README to explicitly state the intent for the referenced dependency charts.

Hi! It is normal for the CKAN to have more than one replica?

Working with the datapusher+ branch I get some errors creating the container related acces to the image repository:

t3-docker.pkg.dev/amplus-data/common/ckan-datapusher-plus:v0.10.1": failed to authorize: failed to fetch anonymous token: unexpected status: 403 Forbidden
failed to resolve reference "europe-west3-docker.pkg.dev/amplus-data/common/ckan-solar:v0.10.1": failed to authorize: failed to fetch anonymous token: unexpected status: 403 Forbidden

Status:

• Infra: OKE

• ckan-helm version : master, v3.3.0 (tried both of them)

• Updates on the values:

replicCount of SolR must be greater than 1 (set to 2) because when we run only one replica we got this error: ERROR ==> Not enough nodes for the replicas and shards indicated

And with multiple replicas, I guess we face another problem even when we force the name of the collection we get that there is issue with CKAN schema.

kubectl get po output:

ckan-9f877fb85-srmv7                      0/1     Running            2 (32s ago)   3m27s
ckan-email-notifications-28194225-7w8f2   0/1     CrashLoopBackOff   4 (19s ago)   2m
ckan4-zookeeper-0                         1/1     Running            0             3m27s
datapusher-78bcd4689d-7nlfn               1/1     Running            0             3m27s
postgres-0                                1/1     Running            0             3m27s
psql-init-nqrcg                           0/1     Completed          0             3m27s
redis-master-0                            1/1     Running            0             3m27s
solr-0                                    1/1     Running            0             3m16s
solr-1                                    1/1     Running            0             3m16s
solr-init-2xb9z                           0/1     Completed          0             3m27s

ko logs ckan po --previous output:

Defaulted container "ckan" out of: ckan, set-volume-ownsership (init)
Not all environment variables are set. Generating sessions...
beaker.session.secret = 
Setting secrets in ini file
Edited option beaker.session.secret = ""->"h6v2uFPOX9Ix77P_MDNuOlTZ5yASd5jsbi0E2ey5fg0" (section "app:main")
Edited option api_token.jwt.encode.secret = "string:N1altO06sRSvpNDuDBTKSJRX9"->"string:yETZ_6kfVJtrkKHFcDxZzbZPnvB4KbKhKrDntzVXae8" (section "app:main")
Edited option api_token.jwt.decode.secret = "string:N1altO06sRSvpNDuDBTKSJRX9"->"string:PcvANbwTSCTdIvgoOnbDBvCReb3Kbk8-kMPcxcSg8nY" (section "app:main")
[prerun] Start check_db_connection...
[prerun] Start check_solr_connection...
[prerun] Unable to connect to solr...try again in a while.
[prerun] Start check_solr_connection...
[prerun] Unable to connect to solr...try again in a while.
[prerun] Start check_solr_connection...
[prerun] Unable to connect to solr...try again in a while.
[prerun] Start check_solr_connection...
[prerun] Unable to connect to solr...try again in a while.
[prerun] Start check_solr_connection...
[prerun] Unable to connect to solr...try again in a while.
[prerun] Start check_solr_connection...
[prerun] Giving up after 5 tries...
[CKAN prerun] FAILED. Exiting...

Most of the variables in the production.ini (config file) are defined to pass to the docker container in the values.yaml (plugins, title, siteurl, etc.). However, I would like to add more variables based on my plugins added.

Is there any functionality or way in the helm chart for this case?

Some environment (like ours) do not allow for pulling remote images, and thus the install tends to break on the busybox:latest container used in both the init-container from CKAN and the test-connection container from the Helm template.
It'd be great if both occurrences of the busybox image can be configured.

Proposal:

Add some fields to the values.yaml which allows for configuring the busybox registries:

...
image:
  repository: keitaro/ckan
  tag: 2.9.4
  pullPolicy: IfNotPresent
  testConnection:
    repository: harbor.containers.wurnet.nl/library/busybox
    tag: latest
    pullPolicy: IfNotPresent
  initContainer:
    repository: harbor.containers.wurnet.nl/library/busybox
    tag: latest
    pullPolicy: IfNotPresent  
...

I'm currently trying to do a rollout with Flux.CD, this renders the template before applying. When rendering the helm chart there are duplicate keys in the liveness and readiness probes of SOLR.

This is cause by the following lines in the solr chart:
templates/statefulset.yaml

          livenessProbe: {{- include "common.tplvalues.render" (dict "value" (omit .Values.livenessProbe "enabled") "context" $) | nindent 12 }}
            initialDelaySeconds: {{ .Values.livenessProbe.initialDelaySeconds }}
            timeoutSeconds: {{ .Values.livenessProbe.timeoutSeconds }}
            failureThreshold: {{ .Values.livenessProbe.failureThreshold }}
            periodSeconds: {{ .Values.livenessProbe.periodSeconds }}
            successThreshold: {{ .Values.livenessProbe.successThreshold }}

          readinessProbe: {{- include "common.tplvalues.render" (dict "value" (omit .Values.readinessProbe "enabled") "context" $) | nindent 12 }}
            initialDelaySeconds: {{ .Values.readinessProbe.initialDelaySeconds }}
            timeoutSeconds: {{ .Values.readinessProbe.timeoutSeconds }}
            failureThreshold: {{ .Values.readinessProbe.failureThreshold }}
            periodSeconds: {{ .Values.readinessProbe.periodSeconds }}
            successThreshold: {{ .Values.readinessProbe.successThreshold }}

you can reproduce this using the following command: helm template keitaro-charts/solr | less or with helm template keitaro-charts/ckan | less

My sugestion would be to remove both static bindings for the liveness and readiness probes.

            initialDelaySeconds: {{ .Values.livenessProbe.initialDelaySeconds }}
            timeoutSeconds: {{ .Values.livenessProbe.timeoutSeconds }}
            failureThreshold: {{ .Values.livenessProbe.failureThreshold }}
            periodSeconds: {{ .Values.livenessProbe.periodSeconds }}
            successThreshold: {{ .Values.livenessProbe.successThreshold }}

            initialDelaySeconds: {{ .Values.readinessProbe.initialDelaySeconds }}
            timeoutSeconds: {{ .Values.readinessProbe.timeoutSeconds }}
            failureThreshold: {{ .Values.readinessProbe.failureThreshold }}
            periodSeconds: {{ .Values.readinessProbe.periodSeconds }}
            successThreshold: {{ .Values.readinessProbe.successThreshold }}

Another suggestion would be to use the bitnami chart directly

I am deploying this services into sub directory instead of the root directory like
https://sample.com/data/

I checked the ckan document and found the root_path should work.
But how to set the environmental variables using the helm chart?

Thanks.

Hi, I want to set ssl https for my deployed ckan in AKS azure. Is it something related setting the ingress in values.yaml?. Thanks

We would like to use ckan 2.9 and encounter issues with the helm chart. I.e paster is now ckan. It is planned to use the helm chart for ckan 2.9?

Currently, in the readme, it says that activityStreamsEmailNotifications is a string, and it is also like that in the default values.yaml. However, the {{- if .Values.ckan.activityStreamsEmailNotifications -}} will also return true if you change its value to "false", so maybe it should be a boolean (or the check should be changed)?

Hi How to change port 5000 to 80 in values.yaml ?

The deployment manifest, by default (at least in the master branch) is using the following volumeMounts code:

          volumeMounts:
          - name: "ckan"
            mountPath: {{ .Values.ckan.storagePath }}
            readOnly: false

Where, by default, storagePath value is "/var/lib/ckan/default"

However, once I deploy the container, when I shell into it, that folder does not exists:

By default, when I join to the container, the default path with the config files is: /srv/app

Where can I found the paths related uploads, resources, plugins and base installation? Are in the same folder /srv/app ?

In a VM installation the folders are:

• /var/lib/ckan = Uploads, resources and plugins
• /usr/lib/ckan = Base installation

However, in the container, those paths don't exist.

Is there a recommendation on how best to enable the ckanext-saml2auth plugin when using this Helm chart?

Hi, How to we customize / change the ckan's theme?. thanks

Hi I got the following error when deploying in azure kubernates (AKS): kubectl logs psql-init-wgwp5

Master DB: postgres Master User: postgres
Checking whether database is up...
Creating user ckan_default
ERROR DB:  role "ckan_default" already exists

Creating user datastorerw
ERROR DB:  role "datastorerw" already exists

Creating user datastorero
ERROR DB:  role "datastorero" already exists

Creating database ckan_default with owner ckan_default
ERROR DB:  database "ckan_default" already exists

Creating database datastore_default with owner datastorerw
ERROR DB:  database "datastore_default" already exists

Traceback (most recent call last):
  File "/srv/psql-init/psql-init.py", line 176, in <module>
    sql = subprocess.check_output(["/usr/bin/ckan",
  File "/usr/lib/python3.8/subprocess.py", line 411, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.8/subprocess.py", line 489, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/lib/python3.8/subprocess.py", line 854, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.8/subprocess.py", line 1702, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/usr/bin/ckan'

Hi I got the following error:

Check_solr_connection...
HTTPConnectionPool(host='solr-headless', port=8983): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f2964c964f0>: Failed to establish a new connection: [Errno 111] Connection refused'))
Unable to connect to solr...retrying.

Thanks

Some folks when they reach my ckan app get 502 consistently, while others get a working page consistently. Interestingly, if the 502 folks open up an incognito browser and try the url it works. Both get an instant response.

I noticed if I watch the ckan container log when one of the 502 folks refresh the page I get:

invalid request block size: 4313 (max 4096)...skip
[uwsgi-http key: ckan.its-cdo-non.k.idaho.gov client_addr: 192.168.1.3 client_port: 13549] hr_instance_read(): Connection reset by peer [plugins/http/http.c line 647]
invalid request block size: 4313 (max 4096)...skip
[uwsgi-http key: ckan.its-cdo-non.k.idaho.gov client_addr: 192.168.1.3 client_port: 14061] hr_instance_read(): Connection reset by peer [plugins/http/http.c line 647]

kubernetes v1.20.12
nginx-ingress-controller 1.2.1 (the kubernetes.github.io version)

The current Helm Chart Version released is ckan 1.0.1. => https://keitaro-charts.storage.googleapis.com/
Can someone publish a new version ?
Thx

I want to share in a PVC the folder /srv/app to be able to migrate plugins, resources, datsources from another CKAN.

What I added in the CKAN deployment.yaml template:

      volumes:
        - name: "ckan"
          persistentVolumeClaim:
            claimName: "ckan"
        - name: "ckan2"
          persistentVolumeClaim:
            claimName: "ckan2"

Related CKAN container:

          volumeMounts:
          - name: "ckan"
            mountPath: {{ .Values.ckan.storagePath }}
            readOnly: false
          - name: "ckan2"
            mountPath: "/srv/app"
            readOnly: false

However, starting up the pod, gets a CrashLoopBackOff:

As far as I understood, it follow these steps:

1. Run the init container (busy box) to set 92:92 (CKAN user) owners to the /var/lib/ckan folder.
2. Run the CKAN container (keitaro/ckan) which in the Dockerfile, copy all folders from /srv/app to the container with the same path:

So I don't understand why it says the file in the folder /srv/app doesn't not exists... Is there any way to sahre the /srv/app folder to be able to migrate the resources (plugins), datastore and maybe the config.ini?

helm install ckan-p keitaro-charts/ckan
Error: INSTALLATION FAILED: YAML parse error on ckan/templates/deployment.yaml: error converting YAML to JSON: yaml: line 10: did not find expected key

any workaround?

i need to download ckan chart 3.3.0 repo, but the last provided chart version is 3.2.0 on github

Error: INSTALLATION FAILED: failed to fetch https://keitaro-charts.storage.googleapis.com/charts/ckan-v2.0.2.tgz : 404 Not Found

thanks for sharing this helm chart.
how can one add harvester extension using your helm chart?

when i install ckan using : helm repo add keitaro-charts https://keitaro-charts.storage.googleapis.com i get : version: ckan- v3.3.0

But if i install using the github repo tag 3.3.0 i get : Version : ckan-2.0.1

Using CKAN version 2.9.9 fails every time. Switching back to 2.9.8 works successfully.

Hi Team,

I am trying to deploy CKAN application in Google cloud platform using helm chart version 0.1.5. But instead of creating a DB pod i am using an external DB(google cloud sql) details to save the user details. But when i tried to deploy the same i am getting error in psql-init job pod where i am able to create DB users and database with the existing psql-init.py script present in /srv/psql-init folder. But while executing permission line to the datastore in production.ini it is providing me error.

Steps to re-create the issue:

1. Need to provide cloudsql details in the db section of values.yaml
2. Next deploy the pods using helm chart.

% kubectl logs solr-0
solr 13:42:11.86
solr 13:42:11.86 Welcome to the Bitnami solr container
solr 13:42:11.86 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-solr
solr 13:42:11.86 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-solr/issues
solr 13:42:11.86
solr 13:42:11.87 INFO ==> ** Starting solr setup **
solr 13:42:11.88 INFO ==> Validating settings in SOLR_* env vars...
solr 13:42:11.88 ERROR ==> Not enough nodes for the replicas and shards indicated

I've noticed that it's governed under a custom license. I have a few questions that I hope can be clarified:

• Could you provide a detailed explanation of the terms and conditions of this license?
• How does this custom license compare to more common open-source licenses like MIT, GPL, or Apache 2.0?
• How does this license affect potential contributors?

Thank you for your attention to these questions.

Hello,
our solr engine restarts intermittently.

is there any way to find the root cause?

adding some info below. ===================>

Events:
Type Reason Age From Message

Normal Killing 34m (x43 over 6d4h) kubelet Container solr failed liveness probe, will be restarted
Warning Unhealthy 9m55s (x324 over 6d13h) kubelet Liveness probe failed: Get "http://10.141.0.125:8983/solr/admin/info/system": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Warning Unhealthy 9m50s (x891 over 6d13h) kubelet Readiness probe failed: Get "http://10.141.0.125:8983/solr/admin/info/system": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
[root@mae1deuj1 ~]# helm list
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
ckan-r ckan-r 1 2022-10-17 09:40:13.2771763 -0400 -0400 deployed ckan-1.0.2 2.9.2
[root@mae1deuj1 ~]# k get pods
NAME READY STATUS RESTARTS AGE
ckan-678bd8db47-9hhpw 1/1 Running 5 (7d4h ago) 7d4h
ckan-email-notifications-28155975-vfvkv 0/1 Completed 0 36m
ckan-email-notifications-28155990-hk7rt 0/1 Completed 0 21m
ckan-email-notifications-28156005-p9rrm 0/1 Completed 0 6m20s
ckan-r-zookeeper-0 1/1 Running 0 7d4h
datapusher-6b8d458cbd-xl2bk 1/1 Running 0 7d4h
postgres-0 1/1 Running 0 7d4h
redis-master-0 1/1 Running 0 7d4h
solr-0 1/1 Running 43 (35m ago) 7d4h

Was looking for a ckan helm chart and seems yours is the most up to date, tnx!
I noticed the psql-init job does not complete successfully; it crashes when setting the datastore permissions (/usr/bin/ckan -c /srv/app/production.ini datastore set-permissions).

After looking into it, I noticed errors like:

ERROR [ckan.lib.search.common] Failed to connect to server at 'http://127.0.0.1:8983/solr/select/?q=%2A%3A%2A&rows=1&wt=json'
,are you sure that URL is correct?
Checking it in a browser might help: HTTPConnectionPool(host='127.0.0.1', port=8983):
Max retries exceeded with url: /solr/select/?q=%2A%3A%2A&rows=1&wt=json
(Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fad2b589a30>:
Failed to establish a new connection: [Errno 111] Connection refused'))

then a bit further:

During handling of the above exception, another exception occurred:
...
  File "/srv/app/src/ckan/ckanext/datastore/backend/__init__.py", line 90, in set_active_backend
    schema = config.get(u'ckan.datastore.write_url').split(u':')[0]
AttributeError: 'NoneType' object has no attribute 'split'

I could prevent the crash by adding CKAN_SOLR_URL and CKAN_DATASTORE_WRITE_URL to templates\psql-init-job.yaml. (The solr error does not seem to cause the crash; it works without the solr url too, the datastore one is though).
It is a bit confusing to then have CKAN__DATASTORE__WRITE_URL (which is used elsewhere in the psql init apparently) and CKAN_DATASTORE_WRITE_URL (fewer underscores, used by the ckan binary) but worked.

After this, I still got an error for re.sub("\\\connect.*", "", sql), since sql contains bytes, so I first did a .decode('utf-8') on that.
Then it was successful.

helm install with set does not seem to work.

helm install ckan keitaro-charts/ckan -f alues.yaml --set ckan.max_resource_size=1000

any idea? thanks

Map a new variable (.Values.pvc.storageClassName) in the PVC of ckan to be able to define the storageClassName
PR : #29

This helm chart does not work on Kubernetes 1.25 (on Azure).
Investigation shows that this is because PodDisruptionBudgets policy v1beta1 has been removed and now only v1 is supported.

https://kubernetes.io/docs/reference/using-api/deprecation-guide/#poddisruptionbudget-v125

This seems to cause problems in the SOLR dependency and possibly elsewhere.
We note there is also a pull request which may partially fix this problem, however we don't think this fixes the solr dependency.

#62

With docker implementing rate limits I have selected to use a local repository. I was able to specify this in a couple places but didn't see a way to specify the repo for zookeeper. Ended up editing the statefulset after it was deployed to make the change. Is it possible to do this via the helm chart?

An idea:
I've seen several charts with this up top:

global:
  imageRegistry: ""

and when used, it lets you specify your local repo, which all images get pulled from