This repo holds scripts and files to build slim docker images for Arvados components and run them using docker-compose.
Arvados seems to fail if you're using cgroup v2, so make sure you're using The rest of this documentation needs update, that will be my next step. In the meantime, you can get arvie up and running with these steps:
- download the repo and cd to it
- create an environment
- populate the gems/pips/npm caches (you'll need this only the 1st time, or when rebuilding them), it will take some time
- build arvie images locally. If you don't want to build them locally, you can download them from dockerhub (just skip this step, jump to next step, and images will be downloaded. Note that dockerhub is imposing limits, so this might take a while.
- run arvie
- Add an entry in
/etc/hosts
to get DNS entries for your cluster:
Add the CA certificate to your browser: this will prevent security errors when accessing Arvados services with your web browser.
- Go to the certificate manager in your browser.
In Chrome, this can be found under “Settings → Advanced → Manage Certificates” or by entering chrome://settings/certificates in the URL bar. In Firefox, this can be found under “Preferences → Privacy & Security” or entering about:preferences#privacy in the URL bar and then choosing “View Certificates…”. Select the “Authorities” tab, then press the “Import” button. Choose arvie.arv.local-CA.crt
The certificate will be added under “__ARVIE__”.
Enter the URL `https://workbench.arvie.arv.local:8443`_ in your browser.
Log in to your cluster (initial user/pass: alice/alice)
If you want to run an arvados' shell run
- Stop Arvie with
./arvie down
Data will be persisted under the ./local/arvie
subdir so, if you start arvie again, your work will be
available again.
This is a Work in Progress... Can break, can fail, can even not work at all. If that's not suitable for you, sorry. If you can deal with that, feel free to contribute 😄
This is build using upstream's docker binaries (>=19.03), from the Docker's repos. The build process uses the "new" features from BuildKit, like caching layers, so this might fail to build if using older/custom docker versions. These features have been added to docker in version 18.03, so it's quite possible you already have them in place.
Afaik, docker-compose-alike and buildkit behaviours have been incorporated in the latest docker binaries (ie, plugin buildx can replace BuildKit), but haven't tested them yet. To be done 😄
Mostly because I wanted to play with the tools involved, but also, because I think it can be useful in some ways:
- For development: although already exists a docker image for Arvados testing/demo arbox, everything runs in a single image. Arvie runs every component in its own instance, so you can rebuild/recreate each component individually, or even test them without having to spin up or rebuild the whole cluster again.
- For Production (some day): As I'm writing this to play with docker-compose and buildkit, it can probably be used to be deployed in the cloud (ie, using Kelda. (TODO)
- Speed: building ALL the images from scratch takes less that 10 minutes in my laptop (i7, 16GB ram). After the initial build, rebuilding any component takes somewhere between 5 seconds to a minute (Rails images are the ones that take most time).
- Networked infrastructure: arvie creates a docker network where each Arvados' component runs in its own named container, making it easier to spot components' relationships.
- Docker images: you can create your own Arvados' docker images for each component (see below). I'm uploading images built from Arvados' master branch to a `dockerhub repo https://hub.docker.com/u/nmarvie`_ so you can use this without needed to build anything but the caches
- Kubernetes: the slim images and config already available in Arvie can probably be migrated for a k8s setup. (TODO)
- Other usages? Ie., learn new tools/things? 😄
To make it easier to use arvie or modify/customize it, things are organized in different subdirs:
- arvados: contains a copy of Arvados' repository. It is downloaded and populated when you run
arvie prepare
. It's a git-submodule of Arvie, so you can either manage it as such or just change to it and manage independently. This directory is usually mounted in the running instances under/usr/src/arvados
. - cache: holds gems, pips, npms and go packages that are used in the Rails apps or in various build stages, to speed things.
- configs: the configuration files for Arvados, Nginx, Postgresql are stored here. They're mounted as volumes in the instances, so you can modify them, restart/reload the process and change will be reflected in your cluster.
- docker-compose: different
docker-compose.yml
files that are used to build images or launch the cluster. - scripts: start scripts that are used when starting the Arvados' components.
- commands: commands that are used to manage Arvie. These files are read by the
arvie
command, and presented to you as a sub-command. Runarvie
with no parameters, and you'll get a list of the available commands. If you want to add another subcommand for your work, just drop a file in this directory and it will be automatically available as anarvie
subcommand. Check the files for examples on how to write/organize them. - dockerfiles: these are the Dockerfiles used to build the different Arvados' images.
The building of images requires the arvados source code to be in a subdir of this repository tree, due to docker design that does not allow to symlink to another directory in a parent/sibling dir nor copy from them, so the easiest way is to just create a git submodule directory inside this repo. The default subdir is arvados.
- Download this repo
git clone https://github.com/netmanagers/arvie.git
Check the variables in the
.env
file, which will be used in a few places. Quite possible you don't need to change them. If unsure, leave them as they are. Default configuration work creating everything under Arvie's own directory.Run
./arvie up
. The first time you run it, it will first run theprepare
subcommand, to populate the Arvados directory, create SSL certs for Postgresql and populate thecache
subdirs. As postgres needs the keys with certain permissions and ownership, the script will ask you for your sudo password.First run will take some time to start, as
prepare
will download and build a few gems that Arvadosi needs and then download the docker images from DockerHub.This is usually ~10 minutes (you'll see the build process on your screen).
Add an entry in
/etc/hosts
to get DNS entries for your cluster:
export ARVIE_CLUSTER_NAME=arvie
export DOMAIN=arv.local
echo \
127.0.0.2 \
api.${ARVIE_CLUSTER_NAME}.${DOMAIN} \
keep.${ARVIE_CLUSTER_NAME}.${DOMAIN} \
keep0.${ARVIE_CLUSTER_NAME}.${DOMAIN} \
collections.${ARVIE_CLUSTER_NAME}.${DOMAIN} \
download.${ARVIE_CLUSTER_NAME}.${DOMAIN} \
ws.${ARVIE_CLUSTER_NAME}.${DOMAIN} \
workbench.${ARVIE_CLUSTER_NAME}.${DOMAIN} \
workbench2.${ARVIE_CLUSTER_NAME}.${DOMAIN} \
${ARVIE_CLUSTER_NAME}.${DOMAIN} \
| sudo tee --append /etc/hosts
Add the CA certificate to your browser: this will prevent security errors when accessing Arvados services with your web browser.
- Go to the certificate manager in your browser.
In Chrome, this can be found under “Settings → Advanced → Manage Certificates” or by entering chrome://settings/certificates in the URL bar. In Firefox, this can be found under “Preferences → Privacy & Security” or entering about:preferences#privacy in the URL bar and then choosing “View Certificates…”. Select the “Authorities” tab, then press the “Import” button. Choose arvie.arv.local-CA.crt
The certificate will be added under “__ARVIE__”.
Enter the URL `https://workbench.arvie.arv.local:8443`_ in your browser.
Log in to your cluster (initial user/pass: alice/alice)
If you want to run an arvados' shell run
- Stop Arvie with
./arvie down
Data will be persisted under the ./local/arvie
subdir so, if you start arvie again, your work will be
available again.
If you want to build a local copy of any (or all) of Arvados' components, you can do it with the build
subcommand:
$ ./arvie build keepstore ws
to build those two images locally from the current Arvados tree in your working environment. If no image/s is/are given, all the images will be built again. Run:
$ ./arvie build -h
to get some help.
So far, the scripts can build docker images for the following components
REPOSITORY TAG IMAGE ID CREATED SIZE
nmarvie/compute latest 0de2ea413d7f 13 hours ago 190MB
nmarvie/shell latest b3cddf00f1e7 15 hours ago 757MB
nmarvie/keepstore latest 0e903cbefdf8 23 hours ago 92.8MB
nmarvie/keepproxy latest 3f97aa2cd894 23 hours ago 84.3MB
nmarvie/workbench latest b4871ce60674 23 hours ago 663MB
nmarvie/api latest accaca9f80a5 23 hours ago 635MB
nmarvie/keep-web latest 9f9396865106 7 days ago 86.3MB
nmarvie/keep-balance latest 0ce7ab96b18e 7 days ago 84.5MB
nmarvie/health latest a9ffa91bb6ff 7 days ago 84.2MB
nmarvie/crunch-dispatch-local latest d0d1a7fdde5b 7 days ago 123MB
nmarvie/git-httpd latest 09656234d70b 7 days ago 84.1MB
nmarvie/client latest cd95446a2bfa 7 days ago 85.7MB
nmarvie/server latest 808e8218a12c 7 days ago 111MB
As a convenience, there's a compose
subcommand, which is used to pass commands to docker-compose
.
Whatever you pass a parameters to the command ./arvie compose
will be passed verbatim to docker-compose
with the docker-compose/base.yml
config file.
Running ./arvie up
is equivalent to ./arvie compose up
and will start the cluster:
$ docker-compose ps
Name Command State Ports
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
api /scripts/ruby/app_start 8004 Up
controller ./executable controller Up
database docker-entrypoint.sh postg ... Up 0.0.0.0:5432->5432/tcp
dispatcher ./executable -poll-interval=1 Exit 1
keep ./executable Up
keep0 ./executable Up
keep1 ./executable Up
keepweb ./executable Up
nginx /docker-entrypoint.sh ngin ... Up 0.0.0.0:25101->25101/tcp, 80/tcp, 0.0.0.0:8000->8000/tcp, 0.0.0.0:8002->8002/tcp, 0.0.0.0:8443->8443/tcp,
0.0.0.0:9002->9002/tcp
shell irb Up
websocket ./executable ws Up
workbench /scripts/ruby/app_start 8002 Up
You can validate your docker-compose
configuration with
$ ./arvie compose config
$ ./arvie compose --file docker-compose/build.yml config
in this last example, remember that docker-compose/base.yml
is used by default with the compose
subcommand, so both files will be merged, by docker-compose
's rules.
- Get Arvie to a useful state (almost there)
- Testing (real testing)
- Improve configuration (too many hardcoded things atm)
- Add missing features/configs
and what's in the `TODO TODO`_ file 😄
Commit message formatting is significant!!
Please see :ref:`How to contribute <CONTRIBUTING>` for more details.