datopian / ckan-cloud-docker Goto Github PK
View Code? Open in Web Editor NEWDocker images and compose environment for local development and testing of ckan-cloud
License: MIT License
Docker images and compose environment for local development and testing of ckan-cloud
License: MIT License
better to just watch the relevant pod and wait for it to be running
kubectl rollout status has some weird behavior a deployment starts while waiting for rollout
child of ViderumGlobal/PM#5
Add an optional attribute to CKAN instance configuration, e.g.:
importFrom:
DB_URL: postgresql://.....
SOLR_URL: http://foo.bar:8983
.
.
When create-instance command encounters this value:
it should detect requirements.txt file - in requirements.txt / pip-requirements.txt / no requirements file and ckan-pip install
Suggested implementation: add initContainer to SOLR which runs the CKAN docker image with entrypoint that copies the SOLR schema to relevant SOLR core directory
Step 30/48 : RUN . /post_install_functions.sh && eval "${POST_INSTALL}"
---> Running in 3d0d726bf532
--2022-06-20 20:29:49-- https://raw.githubusercontent.com/ckan/ckanext-spatial/master/requirements.txt
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.109.133, 185.199.110.133, 185.199.111.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.109.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 149 [text/plain]
Saving to: '/tmp/tmp.mrdZacbLku'
0K 100% 4.67M=0s
2022-06-20 20:29:49 (4.67 MB/s) - '/tmp/tmp.mrdZacbLku' saved [149/149]
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
Looking in indexes: https://pypi.org/simple/
Collecting ckantoolkit
Downloading ckantoolkit-0.0.7.tar.gz (2.8 kB)
Collecting GeoAlchemy>=0.6
Downloading GeoAlchemy-0.7.2.tar.gz (57 kB)
Collecting GeoAlchemy2==0.5.0
Downloading GeoAlchemy2-0.5.0-py2.py3-none-any.whl (16 kB)
Collecting Shapely>=1.2.13
Downloading Shapely-1.7.1-cp27-cp27mu-manylinux1_x86_64.whl (1.0 MB)
ERROR: Could not find a version that satisfies the requirement pyproj==2.6.1 (from -r /tmp/tmp.mrdZacbLku (line 5)) (from versions: 1.8.0, 1.8.1a0, 1.8.1, 1.9.3, 1.9.4, 1.9.5, 1.9.5.1, 1.9.6, 2.0.0, 2.0.1, 2.0.2, 2.1.0, 2.1.1, 2.1.2, 2.1.3, 2.2.0, 2.2.1, 2.2.2)
ERROR: No matching distribution found for pyproj==2.6.1 (from -r /tmp/tmp.mrdZacbLku (line 5))
--2022-06-20 20:29:56-- https://raw.githubusercontent.com/ckan/ckanext-spatial/master/pip-requirements.txt
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.109.133, 185.199.110.133, 185.199.111.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.109.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 16 [text/plain]
Saving to: '/tmp/tmp.mrdZacbLku'
0K 100% 607K=0s
2022-06-20 20:29:56 (607 KB/s) - '/tmp/tmp.mrdZacbLku' saved [16/16]
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
Looking in indexes: https://pypi.org/simple/
ERROR: Could not find a version that satisfies the requirement requirements.txt (from -r /tmp/tmp.mrdZacbLku (line 1)) (from versions: none)
ERROR: No matching distribution found for requirements.txt (from -r /tmp/tmp.mrdZacbLku (line 1))
ERROR: Service 'ckan' failed to build: The command '/bin/sh -c . /post_install_functions.sh && eval "${POST_INSTALL}"' returned a non-zero code: 1
python
Python 3.8.10 (default, Mar 15 2022, 12:22:08)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
currently, state is stored in filesystem, it should instead be stored in kubernetes configmaps/secrets
these actions should return an action_id which you could then query for the status/progress
it could then run some of the create/update steps in parallel
We want to install CKAN in a self-hosted environment.
Usually this means clients owning a machine (or machines), but not k8s cluster.
We need to create an internal installation guide for this scenario, using what we have (i.e. not plain vanilla ckan) - e.g. using our docker compose setup.
Write a document containing:
/cc @OriHoch to review
The documented command fails:
» kubectl --context minikube -n ckan-cloud exec $SOLRCLOUD_POD_NAME -- \
> bin/solr create_collection -c ${CKAN_NAMESPACE} -d ckan_default -n ckan_default
WARNING: Creating cores as the root user can cause Solr to fail and is not advisable. Exiting.
If you started Solr as root (not advisable either), force core creation by adding argument -force
command terminated with exit code 1
When running it with force, however it proceeds normally:
kubectl --context minikube -n ckan-cloud exec $SOLRCLOUD_POD_NAME -- bin/solr create_collection -c ${CKAN_NAMESPACE} -d ckan_default -n ckan_default -force
When adding a dataset and filling "Required Common Core Metadata", I tried to enter Hebrew letters in some metadata fields.
I found that the Keywords (Tags) field does not accept the Hebrew letter "ת". It is on the same key as the English "," which is used to separate tags in this field.
My browser is:
Google Chrome
Version 74.0.3729.108 (Official Build) (64-bit)
Running on Debian Stretch (9.9).
for production deployments only. Minikube should keep the self-hosted DB.
should support modifying a repository such as https://github.com/18f/dns
cca-operator only needs to modify and push changes to github, registration is then handled by the repo's CI/CD
When cca-operator create-instance / update-instance fails to start an instance (e.g. bad configuration causing ckan or its dependencies to crashloop), it should exit with a proper exit code and not wait indefinitely.
Runnig the datagov-theme configuration (without the NG harvester yaml file)
docker-compose -f docker-compose.yaml \
-f .docker-compose-db.yaml \
-f .docker-compose.datagov-theme.yaml \
up
fails to load Nginx
2019/12/31 17:07:22 [emerg] 1#1: host not found in upstream "harvester" in /etc/nginx/conf.d/default.conf:9
nginx_1 | nginx: [emerg] host not found in upstream "harvester" in /etc/nginx/conf.d/default.conf:9
ckan:5000 works fine
Task:
Analysis:
Best solution is to extend the nginx config in NG Harvster yaml file. If this is not possible we need top ensure nginx to ignore this error.
instance created via the provisioning UI: https://cloud-ori-test-1-5c974f4f.ckan.io/
See JS error, seems to be related to subagencies, or some kind of resource which is missing:
Seems like this JS error causes other unrelated problems, e.g. add dataset page is broken.
I am setting up a local development environment according to the README.
Everything installs and starts successfully.
But, when I run docker-compose up -d nginx
ckan logs the following error: psql: could not translate host name "db" to address: Name or service not known
provisioning api throws:
provisioning-api_1 | KeyError: 'PRIVATE_SSH_KEY'
I guess I need a an SSH key to provision the database.
There are a few standard CKAN docker setups used for development in different projects, we would like to converge on a common base CKAN docker setup which will allow to get updates and patches developed faster and deployed more reliably.
Following are the main setups:
Some example of questions the dev docs should address:
related: datopian/ckan.io#90
see #20
ckan container remains on crashloop, with the error:
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) FATAL: password authentication failed for user "ckan"
Full stack trace:
File "/usr/local/bin/ckan-paster", line 11, in <module>
sys.exit(run())
File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/paste/script/command.py", line 102, in run
invoke(command, command_name, options, args[1:])
File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/paste/script/command.py", line 141, in invoke
exit_code = runner.run(args)
File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/paste/script/command.py", line 236, in run
result = self.command()
File "/usr/lib/ckan/venv/src/ckan/ckan/lib/cli.py", line 357, in command
self._load_config(cmd!='upgrade')
File "/usr/lib/ckan/venv/src/ckan/ckan/lib/cli.py", line 330, in _load_config
self.site_user = load_config(self.options.config, load_site_user)
File "/usr/lib/ckan/venv/src/ckan/ckan/lib/cli.py", line 237, in load_config
load_environment(conf.global_conf, conf.local_conf)
File "/usr/lib/ckan/venv/src/ckan/ckan/config/environment.py", line 112, in load_environment
p.load_all()
File "/usr/lib/ckan/venv/src/ckan/ckan/plugins/core.py", line 130, in load_all
unload_all()
File "/usr/lib/ckan/venv/src/ckan/ckan/plugins/core.py", line 183, in unload_all
unload(*reversed(_PLUGINS))
File "/usr/lib/ckan/venv/src/ckan/ckan/plugins/core.py", line 211, in unload
plugins_update()
File "/usr/lib/ckan/venv/src/ckan/ckan/plugins/core.py", line 122, in plugins_update
environment.update_config()
File "/usr/lib/ckan/venv/src/ckan/ckan/config/environment.py", line 285, in update_config
model.init_model(engine)
File "/usr/lib/ckan/venv/src/ckan/ckan/model/__init__.py", line 157, in init_model
version_table = Table('migrate_version', meta.metadata, autoload=True)
File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 439, in __new__
metadata._remove_table(name, schema)
File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 434, in __new__
table._init(name, metadata, *args, **kw)
File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 514, in _init
include_columns, _extend_on=_extend_on)
File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 540, in _autoload
_extend_on=_extend_on
File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2044, in run_callable
with self.contextual_connect() as conn:
File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2112, in contextual_connect
self._wrap_pool_connect(self.pool.connect, None),
File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2151, in _wrap_pool_connect
e, dialect, self)
File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1465, in _handle_dbapi_exception_noconnection
exc_info
File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2147, in _wrap_pool_connect
return fn()
File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 387, in connect
return _ConnectionFairy._checkout(self)
File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 766, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 516, in checkout
rec = pool._do_get()
File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 1138, in _do_get
self._dec_overflow()
File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 1135, in _do_get
return self._create_connection()
File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 333, in _create_connection
return _ConnectionRecord(self)
File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 461, in __init__
self.__connect(first_connect_check=True)
File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 651, in __connect
connection = pool._invoke_creator(self)
File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect
return dialect.connect(*cargs, **cparams)
File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 393, in connect
return self.dbapi.connect(*cargs, **cparams)
File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/psycopg2/__init__.py", line 130, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) FATAL: password authentication failed for user "ckan"
create a backup before deleting (using create-backup command from #23)
deleting all related infra, including but not limited to:
When trying to add new instance, there were a few evicted solrcloud pods in the cluster.
create_instance doesn't check for the pod's status, and tries to create the collection using a command on an evicted pod and not the actually running one.
This will wait indefinitely if any of the prereqs fails - you can't tell which without exec'ing in and running each command in turn. It would be helpful for diagnostics to have some log messages in between indicating the check and the exit code returned.
Trying to create values for new instance according to REAMDE, following commands should be run
INSTANCE_ID=test2
curl https://raw.githubusercontent.com/ViderumGlobal/ckan-cloud-helm/master/minikube-values.yaml \
| tee /etc/ckan-cloud/${INSTANCE_ID}_values.yaml
However ckanHelmChartVersion
defined there is set to v0.0.0 that does not exist in ckan helm chart repo (anymore?).
Setting it to v0.0.0-fix-plugins-indent
, v0.0.4
, v0.0.5
, or v0.0.6
manually solves the problem
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.