We need tasks with priorities since deletion of environments could take awhile.

Closing in favor of #62 . Duplicate

Example schema

docker run -it "localhost:5000/conda-store/numba/numpy/asdf.lt.3.0" /opt/conda/bin/python

Docker registry uses localhost:5000/<namespace>/<environment-name>. We will have the conda-store* namespaces reserved. And not allowed to end users. We propose conda-store-dynamic namespace be for dymamic environments. E.g.

And example image spec localhost:5000/conda-store-dynamic/numpy.ge.1.10/numba/flask

Comparison

• .lt. less than
• .gt. greater than
• .ge. greater than equal
• .le. less than equal
• .eq. equal

The following will render to

name: conda-store-dynamic/numpy.ge.1.10/numba/flask?
dependencies:
 - numpy >= 1.10
 - numba
 - flask

Conda-store will then dynanically build the docker image and then send it to the client.

Currently conda-store jupyterlab extension is polling as fast as possible. Need to change to polling every 5 seconds or so.

https://github.com/hpcng/singularity

Need an end to end test for minikube deployment.

https://github.com/Quansight/conda-store/blob/main/conda_store/build.py#L264

Depends on #61

issue to track things that needed to be added/deleted from the API:

[ ] expose docker image download
[ ] expose archive download

User story 1:

• user should be able to create and edit an existing environment
• after awhile should be able to change to the environment with nb_conda_kernel

User story 2:

• user should be able to delete an environment

Required dependency of #44

Often times there are additional actions that a user would like to do after a deployment:

• run a command after installation for example
• chmod, chown the files

It would be nice to support additional package managers that allow installation at custom prefixes. I'm thinking of node, go, and possibly a few others.

conda-store is not working on macOS and needs a simple fix. This is just an issue to track progress; should be done shortly!

Often times conda-store will fail due to exhausting the memory of the available memory when trying to solve for the conda environment. This issue is hard to catch since the conda solver is killed by the linux via OOM. In other places (e.g. dask workers) they use a nanny to monitor the process to see if it is killied. This is likely a hard issue to solve and with the prefered use of mamba now may be less of a problem since it uses significantly less memory.

I've had several env build failures recently (various reasons) that weren't caught by ci testing. On multiple occasions we've had to go into kubectl/k9s to see the failure or were it got hung up.

I like how everything is automated here, but when my gh action for deployment is showing "green" its misleading that the deployment is complete. I'd like some tooling (or documentation) to tell me how to set up kubectl to see what's going on after gh actions has done it's thing.

I know the pretense is that qhub is k8s for people who don't know k8s, but I need some additional monitoring capability.

conda-forge/staged-recipes#13933

@costrouc, as I use the API I am realizing there is a very real need for us to limit server side requests. I can easily put in a PR for this - is this something we have thought about doing yet?

The idea of using conda-store as a github action to create a docker image, conda lock, conda pack of a given environment.

cc: @jaimergp

Add a proper frontend to conda-store written in vanilla JS.

Will need to use https://github.com/regro/conda-docker

Current database model

Environment -> (specification_sha256, build_id)

build_id -> specification_sha256

specification_id (doesn't point to anything)

Proposal

If you build a conda environment at a set point in time it will result in a different set of packages than if it was built 1 day ago. Thus it doesn't make much sense for the artifact to have the specification sha 256 to be in the name.

I want to support a (rebuild) environment action. Thus I really don't think that a specficaition_id is useful when talking about an environment.

Environment -> build_id -> specification_sha256

Ping the rest api should be installable.

Need a similar environment to https://docs.coiled.io/user_guide/software_environment.html.

Depends on #61.

Copied from nebari-dev/nebari#310:

Conda-store provides a great solution for serving environments to jupyterlab instances.
Linuxbrew is a user level package manager that stores files in /home/linuxbrew.
I was thinking it would be useful to be able to serve up Linuxbrew packages centrally similar to conda-store.
I am not sure if something like this could be integrated directly into conda-store, or if a separate tool should be created for this.

Any thoughts, ideas, or suggestions would be appreciated.

@costrouc , Could conda-store be a solution to the issue below, providing a way for JupyterHub users to use Rstudio?
pangeo-data/pangeo-docker-images#163 (comment)

Create a mockup of how users will interact with conda-store.

Considered several options for making conda-store configurable. Ultimately the python configuration that is possible via traitlets allows for heavily customization that would be hard without it.

Packages worth looking at:

• dask-gateway
• jupyterhub
• jupyter-ssh

These can show how to configure traitlets in a way that is convenient to end users.

It looks like the tests/ folder is out of date. It tries to import conda_store instead of conda_store_server, and most are failing.

E.g. create Table artifacts

artifact:
    id: int
    type: enum.Enum (build artifact type)
    key: string
    sha256: string
    build_id: int

This would be required for garbage collection of builds.

Published packages do not contain LICENSE files.

Conda-store ran out of space on our qhub deployment, so I opened a shell through k9s to remove old copies of environments to free up space. Once complete, and the timeout elapsed, conda-store successfully built the updated environment.
However, now every 10s there is a message in the logs stating build starting again since past timeout. This seems to be a bug in conda-store. I suspect that for now the most appropriate course of action is to kill the pod so it can restart, but this will affect on-going processing so I will need to wait for the cluster to not be in use.

@kcpevey, @costrouc

https://github.com/Quansight/quansight-internal-support/issues/46

Can you modify README so I can hack on conda-store ?

Support for vms

Conda-store needs to have an authentication mechanism. The first step would be to make conda-store integrate with jupyterhub.

There would be a general authentication class similar to jupyterhub that will authenticate a given user and return user and groups that they are a member of.

Then a simple authorization mechanism would be added. Each namespace has read, write, list permissions.

A preliminary authorization mechanism would be read-only if username != namespace and namespace not in groups.

Would require token based authentication and oauth client authenticaiton.

Depends on #61.

For some reason when including pip dependencies conda env build ... expects the directory where it finds environment.yaml to allow writes. Why it doesn't try to create a requirements.txt file in a temporary directory I don't know.

      File "/opt/conda/lib/python3.7/site-packages/conda/exceptions.py", line 1079, in __call__
        return func(*args, **kwargs)
      File "/opt/conda/lib/python3.7/site-packages/conda_env/cli/main.py", line 80, in do_call
        exit_code = getattr(module, func_name)(args, parser)
      File "/opt/conda/lib/python3.7/site-packages/conda_env/cli/main_create.py", line 111, in execute
        result[installer_type] = installer.install(prefix, pkg_specs, args, env)
      File "/opt/conda/lib/python3.7/site-packages/conda_env/installers/pip.py", line 41, in _pip_install_via_requirements
        delete=False)
      File "/opt/conda/lib/python3.7/site-packages/conda/_vendor/auxlib/compat.py", line 80, in Utf8NamedTemporaryFile
        dir=dir, delete=delete)
      File "/opt/conda/lib/python3.7/tempfile.py", line 547, in NamedTemporaryFile
        (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags, output_type)
      File "/opt/conda/lib/python3.7/tempfile.py", line 258, in _mkstemp_inner
        fd = _os.open(file, flags, 0o600)
    OSError: [Errno 30] Read-only file system: '/opt/environments/condaenv.ry3f68ir.requirements.txt'

`$ /opt/conda/bin/conda-env create -p /home/conda/store/575144b1e7ed38e31d35f09c174c21151f66c7c7ae507d2f1eb13b4cb65beb88-scipy2020 -f /opt/environments/environment-scipy2020-geospatial.yaml`

Conda-Store environments return "None" for spec and build since they are not built at time yet.

The bigger discussion around this is: "What do we call environments that haven't had successful builds yet?"

cc: @costrouc

Make it smarter than envioronment.yml