Run Jupyter notebooks on clusters.

If there is an anaconda module available on the cluster, then load it and create a conda environment:

conda create --name <env> ipykernel numpy matplotlib --channel conda-forge

The newly created environment can be activated and deactivated with conda <activate|deactivate> <env>. All the conda environments can be listed with conda info --envs. Conda environments can take between 1G to 10G bytes, make sure there is enough disk quota available with quota -u <username> or if there may be machine specific way of disk space distribution between users, e.g. WORK or SCRATCH space limitations on a specific machine. If the conda environment can't be created on the home directory, it can be created elsewhere, e.g. SCRATCH but then must be soft linked to ~/.conda:

ln -s /path/to/conda ~/.conda

To install any other packages later on, e.g.jupyter_contrib_nbextensions, after activating the environment, just install normally any package from any channel:

conda activate <env>
conda install jupyter_contrib_nbextensions --channel conda-forge

To enable an extensions, e.g. collapsible_headings:

jupyter nbextension enable collapsible_headings/main

Now to access the juyter notebook from a local machine, first initialize the jupyter-notebook server on the cluster:

jupyter-notebook --no-browser --port=8893 --ip=127.0.0.1

mind the url with the token. Now on the local machine forward the specified port to the cluster:

ssh -N -f -L localhost:8893:localhost:8893 <username>@<hostname> -i /path/to/parivate/key

Now the notebook server can be accessed locally with browser and the url generated at the last step.