swapnilpande / ml-experimental-framework Goto Github PK

Build application server on Docker to generate and run experiments.

• Web server to upload experiments & return results

• Run ExperimentInterpreter to accept experiments, generate training files/docker containers, and send experiments to SLURM

• Create Docker Compose files to run all components of the applications server

Configure SLURM workload manager on DGX.

• Configure node & GPU support on node

Base generator class to be used by plugins for docker container generation

Produce batch files to spawn docker container. This batch file will be queued by SLURM and will contain resource requirements for model training.

The DGX is currently running nvidia-docker v1. We should update to v2 since it provides a custom runtime for docker instead of being a wrapper for docker. Instead of executing docker commands using nvidia-docker run, we can use docker run --runtime=nvidia.

• Generate documentation for changes in command

• Run update

Create an ExperimentIntpreter class to accept experiments and generate training files using keras/docker plugins.

• Perform error checking on JSON file

• Interface with plugins to generate the training files