netmanagers / arvie Goto Github PK

api and workbench create a PID file under Arvados' directory (being removed with arvie clean atm). They should be configured to go to a temporary directory, keeping Arvados' source tree clean

This will prevent the permissions issues with the ssl cert and data dirs. See https://github.com/docker-library/docs/blob/master/postgres/README.md#arbitrary---user-notes

Every time I want to try CWL :

apt-get update
apt-get install python3-setuptools git python3-wheel python3-pip
pip3 install cwltest
git clone https://github.com/common-workflow-language/cwl-v1.2/  

and to run the test the following is neede (maybe this could be MOTD when you login to shell for the first time)

cd cwl-v1.2
# copy this from from token information
HISTIGNORE=$HISTIGNORE:'export ARVADOS_API_TOKEN=*'
export ARVADOS_API_TOKEN=....
export ARVADOS_API_HOST=test1.arv.local:8000
export ARVADOS_API_HOST_INSECURE=true
./run_test.sh RUNNER=arvados-cwl-runner EXTRA="--api=containers --compute-checksum --disable-reuse"

That way, we can probably use scratch as the final image for the binaries.

I solved this by doing

apt-get update
apt-get --no-install-recommends install gnupg
/usr/bin/apt-key adv --keyserver pool.sks-keyservers.net --recv 1078ECD7
echo "deb http://apt.arvados.org/buster buster main" | tee /etc/apt/sources.list.d/arvados.list
apt-get update

but is a nice to have for quick instalation / upgrade for things that don't come with arvie

To reduce build time and image sizes, we can try to use dir caches.

While using buildkit caches can help improve buildtimes, the RoR images are too big and they need to run bundler commands or rake tasks at run time, to either add npm packages that require the configuration in place, or update the container gems.

As during build time we can't mount volumes, I'm thinking that resourcing to some process can be done, like:

1. When building the images
   1.1- Create a cache dir
   1.2- Copy from the host to the cache their contents
   1.3- Build in a builder container
   1.4- Copy the gems/npms/other data that to the host. (see https://stackoverflow.com/a/51186557)
   1.5- Build a slimmer version of the image, without all those files

2. When running the instances
   2.1- Mount the cache dirs
   2.2- Run the scripts to update the images
   2.3- Launch the process

This way, step 2.2 will run faster if 1.4 happened, and step 1.3 will run faster because of the host caching done in 2.1 and 2.2