datopian / ckan-cloud-docker Goto Github PK

View Code? Open in Web Editor NEW

21.0 21.0 23.0 991 KB

Docker images and compose environment for local development and testing of ckan-cloud

License: MIT License

Dockerfile 10.72% Shell 77.75% Python 6.01% VCL 4.04% Makefile 1.48%

ckan-cloud-docker's People

Contributors

Stargazers

Watchers

ckan-cloud-docker's Issues

cca-operator: update instance should not use kubectl rollout status

better to just watch the relevant pod and wait for it to be running

kubectl rollout status has some weird behavior a deployment starts while waiting for rollout

ckan should support oauth login providers

cca-operator create-instance command should support import of data

child of ViderumGlobal/PM#5

Add an optional attribute to CKAN instance configuration, e.g.:

importFrom:
  DB_URL: postgresql://.....
  SOLR_URL: http://foo.bar:8983
  .
  .

When create-instance command encounters this value:

get the relevant DB / SOLR connection strings / minio storage URL
either connect directly to the importFrom values, or schedule new infra. and import the data there
deploy the helm chart - ensuring it uses the imported data

pull request build fails from Travis

https://travis-ci.org/ViderumGlobal/ckan-cloud-docker/builds/537694540

install_standard_ckan_extension_github should check which files exist in the repo and install accordingly

it should detect requirements.txt file - in requirements.txt / pip-requirements.txt / no requirements file and ckan-pip install

SOLR should get it's schema from the CKAN container

Suggested implementation: add initContainer to SOLR which runs the CKAN docker image with entrypoint that copies the SOLR schema to relevant SOLR core directory

error in Step 30/48 - ubuntu20 locallly instance

Step 30/48 : RUN . /post_install_functions.sh && eval "${POST_INSTALL}"
 ---> Running in 3d0d726bf532
--2022-06-20 20:29:49--  https://raw.githubusercontent.com/ckan/ckanext-spatial/master/requirements.txt
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.109.133, 185.199.110.133, 185.199.111.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.109.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 149 [text/plain]
Saving to: '/tmp/tmp.mrdZacbLku'

     0K                                                       100% 4.67M=0s

2022-06-20 20:29:49 (4.67 MB/s) - '/tmp/tmp.mrdZacbLku' saved [149/149]

DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
Looking in indexes: https://pypi.org/simple/
Collecting ckantoolkit
  Downloading ckantoolkit-0.0.7.tar.gz (2.8 kB)
Collecting GeoAlchemy>=0.6
  Downloading GeoAlchemy-0.7.2.tar.gz (57 kB)
Collecting GeoAlchemy2==0.5.0
  Downloading GeoAlchemy2-0.5.0-py2.py3-none-any.whl (16 kB)
Collecting Shapely>=1.2.13
  Downloading Shapely-1.7.1-cp27-cp27mu-manylinux1_x86_64.whl (1.0 MB)
ERROR: Could not find a version that satisfies the requirement pyproj==2.6.1 (from -r /tmp/tmp.mrdZacbLku (line 5)) (from versions: 1.8.0, 1.8.1a0, 1.8.1, 1.9.3, 1.9.4, 1.9.5, 1.9.5.1, 1.9.6, 2.0.0, 2.0.1, 2.0.2, 2.1.0, 2.1.1, 2.1.2, 2.1.3, 2.2.0, 2.2.1, 2.2.2)
ERROR: No matching distribution found for pyproj==2.6.1 (from -r /tmp/tmp.mrdZacbLku (line 5))
--2022-06-20 20:29:56--  https://raw.githubusercontent.com/ckan/ckanext-spatial/master/pip-requirements.txt
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.109.133, 185.199.110.133, 185.199.111.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.109.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 16 [text/plain]
Saving to: '/tmp/tmp.mrdZacbLku'

     0K                                                       100%  607K=0s

2022-06-20 20:29:56 (607 KB/s) - '/tmp/tmp.mrdZacbLku' saved [16/16]

DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
Looking in indexes: https://pypi.org/simple/
ERROR: Could not find a version that satisfies the requirement requirements.txt (from -r /tmp/tmp.mrdZacbLku (line 1)) (from versions: none)
ERROR: No matching distribution found for requirements.txt (from -r /tmp/tmp.mrdZacbLku (line 1))
ERROR: Service 'ckan' failed to build: The command '/bin/sh -c . /post_install_functions.sh && eval "${POST_INSTALL}"' returned a non-zero code: 1

python
Python 3.8.10 (default, Mar 15 2022, 12:22:08)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

cca-operator server: use kubernetes configmaps/secrets to store state

currently, state is stored in filesystem, it should instead be stored in kubernetes configmaps/secrets

cca-operator server: create/update instance should run asynchroneously

these actions should return an action_id which you could then query for the status/progress

it could then run some of the create/update steps in parallel

Create self-hosted installation guide

We want to install CKAN in a self-hosted environment.
Usually this means clients owning a machine (or machines), but not k8s cluster.

We need to create an internal installation guide for this scenario, using what we have (i.e. not plain vanilla ckan) - e.g. using our docker compose setup.

Write a document containing:

/cc @OriHoch to review

Solr create collection command needs -force

The documented command fails:

» kubectl --context minikube -n ckan-cloud exec $SOLRCLOUD_POD_NAME -- \
>     bin/solr create_collection -c ${CKAN_NAMESPACE} -d ckan_default -n ckan_default
WARNING: Creating cores as the root user can cause Solr to fail and is not advisable. Exiting.
         If you started Solr as root (not advisable either), force core creation by adding argument -force
command terminated with exit code 1

When running it with force, however it proceeds normally:

kubectl --context minikube -n ckan-cloud exec $SOLRCLOUD_POD_NAME -- bin/solr create_collection -c ${CKAN_NAMESPACE} -d ckan_default -n ckan_default -force

Hebrew "ת" cannot be used in tags

When adding a dataset and filling "Required Common Core Metadata", I tried to enter Hebrew letters in some metadata fields.
I found that the Keywords (Tags) field does not accept the Hebrew letter "ת". It is on the same key as the English "," which is used to separate tags in this field.

My browser is:
Google Chrome
Version 74.0.3729.108 (Official Build) (64-bit)
Running on Debian Stretch (9.9).

change instance databases to use centralized Amazon RDS DB

for production deployments only. Minikube should keep the self-hosted DB.

cca-operator: add 18f/dns to the supported DNS providers

should support modifying a repository such as https://github.com/18f/dns

cca-operator only needs to modify and push changes to github, registration is then handled by the repo's CI/CD

create-instance should detect un-recoverable errors

When cca-operator create-instance / update-instance fails to start an instance (e.g. bad configuration causing ckan or its dependencies to crashloop), it should exit with a proper exit code and not wait indefinitely.

Error loading datagov-tjheme

Runnig the datagov-theme configuration (without the NG harvester yaml file)

docker-compose -f docker-compose.yaml \
    -f .docker-compose-db.yaml   \
    -f .docker-compose.datagov-theme.yaml \
    up

fails to load Nginx

2019/12/31 17:07:22 [emerg] 1#1: host not found in upstream "harvester" in /etc/nginx/conf.d/default.conf:9
nginx_1                | nginx: [emerg] host not found in upstream "harvester" in /etc/nginx/conf.d/default.conf:9

ckan:5000 works fine

Task:

Allow Datagov to work without NG Harvester

Analysis:
Best solution is to extend the nginx config in NG Harvster yaml file. If this is not possible we need top ensure nginx to ignore this error.

cca-operator should keep a detailed log of actions and details of all previous configurations / values for each instance

setup jenkins and cca-operator in the docker-compose environment for testing

bugs on CKAN instance created via provisioning ui

instance created via the provisioning UI: https://cloud-ori-test-1-5c974f4f.ckan.io/

See JS error, seems to be related to subagencies, or some kind of resource which is missing:

Seems like this JS error causes other unrelated problems, e.g. add dataset page is broken.

cca-operator for cluster management should run on Kubernetes using Amazon EFS shared volume

Add get-instance-values command to cca_operator which will return a CKAN instance values.yaml file
Move the values files from /etc/ckan-cloud on the cloud management server to Amazon EFS shared volume.
Write a Kubernetes job which will allow to update the values and run the cca-operator.
Write a shell script which will schedule this job with relevant arguments.

No db when running locally

I am setting up a local development environment according to the README.

Everything installs and starts successfully.

But, when I run docker-compose up -d nginx ckan logs the following error: psql: could not translate host name "db" to address: Name or service not known

provisioning api throws:
provisioning-api_1 | KeyError: 'PRIVATE_SSH_KEY'

I guess I need a an SSH key to provision the database.

docker-compose dev environment - support okfn docker stack, improve dev flows

There are a few standard CKAN docker setups used for development in different projects, we would like to converge on a common base CKAN docker setup which will allow to get updates and patches developed faster and deployed more reliably.

Following are the main setups:

Some example of questions the dev docs should address:

How do I quickly install CKAN and have a running instance locally with docker? (I think already solved)
How do I create an extension template in my local machine using paster commands knowing that CKAN is in the dockerized env?
How do I separate that extension in a new git repo so I can have independence to install it in other instances?
How do I quickly refresh the changes in my extension into the dockerized env so I can have quick feedback of my changes?
How do I run the tests for my extension in the dockerized env so I can have a quick test-development cycle?
How do I debug my methods in the dockerized env so I can have a better understanding of whats going on with my logic?

related: datopian/ckan.io#90

add support for postgis to the centralized DB

see #20

Failure when deleting and recreating an instance with the same config

ckan container remains on crashloop, with the error:

sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) FATAL:  password authentication failed for user "ckan"

Full stack trace:

  File "/usr/local/bin/ckan-paster", line 11, in <module>
    sys.exit(run())
  File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/paste/script/command.py", line 102, in run
    invoke(command, command_name, options, args[1:])
  File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/paste/script/command.py", line 141, in invoke
    exit_code = runner.run(args)
  File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/paste/script/command.py", line 236, in run
    result = self.command()
  File "/usr/lib/ckan/venv/src/ckan/ckan/lib/cli.py", line 357, in command
    self._load_config(cmd!='upgrade')
  File "/usr/lib/ckan/venv/src/ckan/ckan/lib/cli.py", line 330, in _load_config
    self.site_user = load_config(self.options.config, load_site_user)
  File "/usr/lib/ckan/venv/src/ckan/ckan/lib/cli.py", line 237, in load_config
    load_environment(conf.global_conf, conf.local_conf)
  File "/usr/lib/ckan/venv/src/ckan/ckan/config/environment.py", line 112, in load_environment
    p.load_all()
  File "/usr/lib/ckan/venv/src/ckan/ckan/plugins/core.py", line 130, in load_all
    unload_all()
  File "/usr/lib/ckan/venv/src/ckan/ckan/plugins/core.py", line 183, in unload_all
    unload(*reversed(_PLUGINS))
  File "/usr/lib/ckan/venv/src/ckan/ckan/plugins/core.py", line 211, in unload
    plugins_update()
  File "/usr/lib/ckan/venv/src/ckan/ckan/plugins/core.py", line 122, in plugins_update
    environment.update_config()
  File "/usr/lib/ckan/venv/src/ckan/ckan/config/environment.py", line 285, in update_config
    model.init_model(engine)
  File "/usr/lib/ckan/venv/src/ckan/ckan/model/__init__.py", line 157, in init_model
    version_table = Table('migrate_version', meta.metadata, autoload=True)
  File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 439, in __new__
    metadata._remove_table(name, schema)
  File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 434, in __new__
    table._init(name, metadata, *args, **kw)
  File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 514, in _init
    include_columns, _extend_on=_extend_on)
  File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 540, in _autoload
    _extend_on=_extend_on
  File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2044, in run_callable
    with self.contextual_connect() as conn:
  File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2112, in contextual_connect
    self._wrap_pool_connect(self.pool.connect, None),
  File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2151, in _wrap_pool_connect
    e, dialect, self)
  File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1465, in _handle_dbapi_exception_noconnection
    exc_info
  File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2147, in _wrap_pool_connect
    return fn()
  File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 387, in connect
    return _ConnectionFairy._checkout(self)
  File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 766, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 516, in checkout
    rec = pool._do_get()
  File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 1138, in _do_get
    self._dec_overflow()
  File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 1135, in _do_get
    return self._create_connection()
  File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 333, in _create_connection
    return _ConnectionRecord(self)
  File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 461, in __init__
    self.__connect(first_connect_check=True)
  File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 651, in __connect
    connection = pool._invoke_creator(self)
  File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect
    return dialect.connect(*cargs, **cparams)
  File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 393, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/psycopg2/__init__.py", line 130, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) FATAL:  password authentication failed for user "ckan"

cca-operator create-instance command should be modular and more robust

split create-instance script to distinct sub-scripts (e.g. update-load-balancer, deploy-ckan-instance etc..)
merge the create-instance script with update-instance script - the script needs to verify each part is deployed and fix if necessary.
add health / readiness checks for the different components and use those checks in the script

delete-instance should create a backup and then delete all related infrastructure

create a backup before deleting (using create-backup command from #23)

deleting all related infra, including but not limited to:

centralized DBS (db / datastore-db / jobs-db)
solrcloud collection
EFS storage
load balancer configuration
DNS names

create_instance tries to add collection on evicted solr pod

When trying to add new instance, there were a few evicted solrcloud pods in the cluster.
create_instance doesn't check for the pod's status, and tries to create the collection using a command on an evicted pod and not the actually running one.

Make ckan_init.sh output more helpful

This will wait indefinitely if any of the prereqs fails - you can't tell which without exec'ing in and running each command in turn. It would be helpful for diagnostics to have some log messages in between indicating the check and the exit code returned.

minikube-values.yaml has non existing ckanHelmChartVersion init

Trying to create values for new instance according to REAMDE, following commands should be run

INSTANCE_ID=test2
curl https://raw.githubusercontent.com/ViderumGlobal/ckan-cloud-helm/master/minikube-values.yaml \
    | tee /etc/ckan-cloud/${INSTANCE_ID}_values.yaml

However ckanHelmChartVersion defined there is set to v0.0.0 that does not exist in ckan helm chart repo (anymore?).

Setting it to v0.0.0-fix-plugins-indent, v0.0.4, v0.0.5, or v0.0.6 manually solves the problem

datopian / ckan-cloud-docker Goto Github PK

ckan-cloud-docker's People

Contributors

Stargazers

Watchers

Forkers

ckan-cloud-docker's Issues

Recommend Projects

Recommend Topics

Recommend Org