Migrate Metadata is a tool to migrate all datasets from a CKAN instance to a metastore-lib backend such as GitHub or a local filesystem.
Currently migrate-metadata
has support for Python 2.7 only.
First, clone the repo via git:
$ git clone https://github.com/datopian/migrate-metadata.git
Move to directory:
$ cd migrate-metadata
Install all requirements and the package (it is recommended that this is done in a virtual environment):
$ pip install -r requirements.txt .
$ python setup.py develop
To import all CKAN datasets into a metastore backend, run:
$ metastore-import-ckan -c $CKAN_API_URL -k $CKAN_API_KEY \
-m $METASTORE_TYPE -o $METASTORE_OPTIONS
Replace all environment variables above with relevant values.
$METASTORE_TYPE
should be a metastore-lib backend type$METASTORE_OPTIONS
should be a JSON-serialized object with the configucation options expected by the specific metastore-lib backend you are using.
See the metastore-lib documentation for a list of supported backends and their respective configuration options.
Run metastore-import-ckan --help
to get the full list of
command line options.
Import all CKAN datasets from http://ckan:500 to a local filesystem metastore in ./metastore:
$ metastore-import-ckan -c http://ckan:5000 -k 123-abc-321-xyz \
-m filesystem -o '{"uri":"./metastore"}'
Import all CKAN datasets from http://ckan:500 to a private GitHub repository:
$ GITHUB_OPTS='{
"github_options": {"login_or_token": "averylongtokenthatwasgeneratedespeciallyforthis"},
"private":true,
"default_owner":"myorganization"}'
$ metastore-import-ckan -c http://ckan:5000 -k 123-abc-321-xyz \
-m github -o "$GITHUB_OPTS"
The jq
command line tool can be useful to debug the output of the migration process
while working locally. For example, to list all datasets migrated you can execute:
$ cat $(ls metastore/p/*/* | grep -v '\.') | jq '.title'
To run tests:
make test
This project is licensed under the MIT License - see the LICENSE file for details