The following lines could be run in the terminal to keep the virtual environment persistent in Jupyter Notebooks:

conda activate MLvenv
pip install ipykernel
python -m ipykernel install --user --name=MLvenv

Add a PyTorch dataset class for the penguin dataset.

Issue #46 made me wonder if we should perhaps re-name this repo as practical-ml-with-pytorch or similar as it better reflects the paper name, workshop name, and removes any confusion that this is intended for 'general ML' or similar.

Thoughts from @ma595 @surbhigoel77 @tztsai @dorchard @jdenholm appreciated.

It would be good to do this before the JOSE publication is accepted, which I hope is imminent.
Let me know and I will implement if we are happy.

Now the sessions have finished, we will begin adding worked solutions to the exercises.

Jack and Jim have quite a lot on this previously:

• we probably could condense this
• add a pytorch primer.

Mathematics and Machine Learning

Basic mathematics knowledge:

• calculus - differentiating a function
• matrix algebra - matrix multiplication and representing data as a matrix
• regression - fitting a function to data

Neural Networks:

• Awareness of high-level concepts
• We recommend the video series by 3Blue1Brown, at least chapters 1-3.

Python

The course will be taught in python using pyTorch.
Whilst no prior knowledge of pyTorch is expected we assume users are familiar with the basics of Python3.
This includes:

• Basic mathematical operations
• Writing and running scripts/programs
• Writing and using functions
• The concept of object orientation
  i.e. that an object, e.g. a dataset, can have associated functions/methods associated with it.
• Basic use of the following libraries:
  • numpy for mathematical and array operations
  • matplotlib for ploting and visualisation
  • pandas for storing and accessing tabular data
• Familiarity with the concept of a jupyter notebook

git and GitHub

You will be expected to know how to

• clone and/or fork a repository,
• commit, and
• push.

The workshop from the 2022 ICCS Summer School
should provide the necessary knowledge.

Preparation

In preparation for the course please ensure that your computer contains the following:

• A text editor - e.g. vim/neovim, gedit, vscode, sublimetext etc. to open and edit code files
• A terminal emulator - e.g. GNOME Terminal, wezterm, Windows Terminal (windows only), iTerm (mac only)
• python virtual environment (see Installation and setup)

Note for Windows users: We have linked suitable applications for windows in the above lists.
However, you may wish to refer to Windows' getting-started with python information
for a complete guide to getting set up on a Windows system.

If you require assistance or further information with any of these please reach out to
us before a training session.

The targets keys should be a List[str] and not str.

Surbhi and I plan to spend more time explaining the theory of NNs with more focus on:

• Datasets. Dimensions.
• An expanded introduction:
  • Potential applications. What NNs can do...
  • Other alternatives to NNs.
  • More of an overview of pytorch. What is jax etc.
• Links to previous statistical techniques like regression.

Specific notes on current slides:

11/29

• neural is spelt nerual
• some weird aberration on the slide
• weird formatting here.

13/29

• "workshop lecture thing"

General notes on slides

• quite a brief overview of nns
  - do mention 3blue1brown ml course, can't do much better than this.
  - could we include some of this content in additional slides to better visualise some concepts?
• jumps straight into SGD without much intro
  • more of a justification for why we're using NNs
• more images
  • visualising data
    • tabulated data
  • potentials of what NNs can do
• comparison of different models for different datasets.
  • Other methods like SVMs instead of NNs.
• Drawbacks of NNs?
  - black-boxes. https://towardsdatascience.com/the-math-behind-kan-kolmogorov-arnold-networks-7c12a164ba95
• pytorch vs scikit learn
  • why we even need pytorch

Summary of what is currently covered

Penguin classification:

• loading penguin dataframe and inspecting data
• introduce a torch.utils.data.dataset.
• split into train and validation
• transforming the data using torchvision.transforms.Compose

Suggestions for guided tutorial / workshp

• Add some boilerplate / template for them to add to. Notes.
• Extensions. Compare different methods?

Other resources

• Stanford ML resources

Todos

• Think about what content we want to include as part of the expanded introduction.
  • Look into Quarto
  • How to improve slides, instead of starting with SGD.
  • Might be helpful to look at other resources here. https://github.com/phlippe/uvadlc_notebooks
• Notebook improvements.
  • Adding more content to the exercises to make them easier to complete
  • Statistical summary of data. max. / min / missing values ...
  • [ ]

Here are a series of lecture notebooks I prepared as an ML instructor when I was in college (except review.docx). We can use them freely for the ML workshop.

Proposed sequence of topics

1. How ML works
2. Steps involved in developing a model - loading data, preprocessing, model definition (algorithm, optimiser, loss), training, prediction)
3. Pytorch intro (data type it works with - tensors, other key concepts)
4. Types of problems - Regression and Classification
5. Hands-on with a Regression example
6. Hands-on with Classification example

Algorithm used: Neural Network

Fixes #60

Too much boilerplate needs to be provided by the students. It would be better to have some of the boilerplate provided so that they concentrate on implementing the key details of the algorithm.

One idea is to do the following:

# code to complete the training loop should go here... (please change this line to fix the training loop). 

General comments

• Transformation of categorical data.

• more visualisations

  • data exploration steps
    • understanding of data. Plot distribution of each of the variables. Statistical information of each variable.
  • For evaluation we could provide a correlation matrix

• Look at CNN notebook. Currently it's MNIST. Could choose something more climate focussed?

Installing now leads to the following issue:

python3 
Python 3.12.4 | packaged by conda-forge | (main, Jun 17 2024, 10:11:10) [Clang 16.0.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch

A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.0.0 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.

If you are a user of the module, the easiest solution will be to
downgrade to 'numpy<2' or try to upgrade the affected module.
We expect that some modules will need time to support NumPy 2.

Traceback (most recent call last):  File "<stdin>", line 1, in <module>
  File "/Users/matt/miniforge3/envs/ml-workshop-2/lib/python3.12/site-packages/torch/__init__.py", line 1477, in <module>
    from .functional import *  # noqa: F403
  File "/Users/matt/miniforge3/envs/ml-workshop-2/lib/python3.12/site-packages/torch/functional.py", line 9, in <module>
    import torch.nn.functional as F
  File "/Users/matt/miniforge3/envs/ml-workshop-2/lib/python3.12/site-packages/torch/nn/__init__.py", line 1, in <module>
    from .modules import *  # noqa: F403
  File "/Users/matt/miniforge3/envs/ml-workshop-2/lib/python3.12/site-packages/torch/nn/modules/__init__.py", line 35, in <module>
    from .transformer import TransformerEncoder, TransformerDecoder, \
  File "/Users/matt/miniforge3/envs/ml-workshop-2/lib/python3.12/site-packages/torch/nn/modules/transformer.py", line 20, in <module>
    device: torch.device = torch.device(torch._C._get_default_device()),  # torch.device('cpu'),
/Users/matt/miniforge3/envs/ml-workshop-2/lib/python3.12/site-packages/torch/nn/modules/transformer.py:20: UserWarning: Failed to initialize NumPy: _ARRAY_API not found (Triggered internally at /Users/runner/work/pytorch/pytorch/pytorch/torch/csrc/utils/tensor_numpy.cpp:84.)
  device: torch.device = torch.device(torch._C._get_default_device()),  # torch.device('cpu'),

Pier Luigi's + Anna Sommer:

Summary:

• Practical part will include 3 examples from the paper with a reduced dataset.
• Students are required to be familiar with data visualisation ahead of time. Specifically cartopy and matplotlib. I have checked with James and this will be part of the visualisation lecture.
• Mostly using pandas.

Jack inquired about what we could teach to complement the follow session by Anna Sommer::

Plan is to do the following:

I’m preparing a practical part for our session now. It’ll be in a form of Jupyter Notebook with some theoretical explanation, questions, and code to run. The topic is “Observation System Simulation Experiences in the Atlantic Ocean for enhanced surface ocean pCO2 reconstructions”, it is based on my paper with the same title published in 2021 https://os.copernicus.org/articles/17/1011/2021/.
The practical part will contain three examples from this paper with reduced data set due to time restriction.
We would like to show students that physical models can be also used to plan a deployment of measuring instruments, particularly localizing the areas where the source of data is most important to understand some phenomena.

The packages I use in the exercise are

import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader, TensorDataset
from sklearn.model_selection import train_test_split
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import cartopy.crs as ccrs

I’m working with data in dataframe format. Data will be in .csv files.
An important part is a data visualisation. It would be great if students have experienced already with matplotlib and cartopy.

• First, we will plot the distribution of available data in the Atlantic Ocean.
• In the second step, we’ll run Machine Learning model and to look at the loss function.
• The third part will be a visualisation of results: maps of standard deviation, mean of differences between reference (ocean model) and ML output, correlation coefficient between reference and ML output.

It is three main steps of the exercise with questions and analysing work in each of them.

If you need more details or have comments/questions/suggestions, please don’t hesitate to contact me.

Hi, Laura, Pier Luigi, mentioned that you could help during the practical part. As we have something around an hour for practical part it would be great to have someone to help if students struggle technically or have questions etc. If you feel comfortable with this topic it would be helpful if you join us. As mentioned before if you need more details or have comments/questions/suggestions, please don’t hesitate to contact me. I’ll share the Jyputer Notebook file when it is ready (hopefully in a week's time).

It is not quite clear to me whether these exercises will be done in a guided, life-coding style fashion, or self-guided by the participants (with instructors available for questions). I assume the latter, but in that case some parts might need additional instructions.

Exercise1

• Task 2, Line 63: What are we supposed to do with the "magic methods" which are mentioned in brackets? Implement them? Use them?
• Line 65: spoiller -> spoiler
• Tasks 6 and 7: Will you guide participants through these exercises? Otherwise more info would be needed.
• Task 8, Line 189: Link could be integrated in the line above (not crucial, though).
• Tasks 8, 9 and 10 need more guidance what needs to be done.

Exercise 2

• Task 3, Line 73: Remeber -> Remember
• Task 6, Line 142: on scratch -> from scratch; can solve -> want to solve; is cumbersome -> would be cumbersome
• Line 144: very ugly -> not very nice (or something else less offensive)
• Line 155: Leave out the "while my peers..." bit, just say "It is useful to know"
• Again, more guidance for tasks 8, 9 and 10

Exercise 3

• Task 1a, Line 37 - Do we know how to plot image tensors, or does that need more guidance?
• Task 1a, Line 38 - Do we know how to create a validation data set, or does that need more guidance?
• Numbering inconsistent: 1, 1a, 2a, 3 ?
• Task 4, Line 101 wont ->won't
• Task 5, Line 442: choose -> chose
• "Test 6"? -> Task 6?: "But I forgot" -> Is that intended, or do you need to change something above?

Relatively low priority, but when reviewing #29 I noticed that there are a few minor typos in the readme (e.g. "ploting", "neccessary", "sesison", "systems's") that would be nice to clean up during the next pass.

As mentioned in #29, some of text in the exercise/solution notebooks is also a little out of sync.

Add a license.

VESRI prefers MIT.

Suggest code/general repo is licensed under MIT with Teaching materials under CC4 https://creativecommons.org/licenses/by-nc-sa/4.0/

To make the data reading aspect a little easier to understand, we intend to embed a simpler version of the PenguinDataset directly into the notebook. The src/ml_workshop/_penguins.py will remain untouched and can still be used as before.

Thought process around the 'simpler' class (discussion between @jatkinson1000 and @ma595)

Load pandas df in notebook:

Put the definition of the PenguinDataset in the notebook:

• Remove x_tfms and y_tfms
• Hardcode one_hot, tensor, fp32 into __getitem__

Propagate change to

• solution notebook
• colab
• ex 2+? Consider leaving this as is.

Can we deal with the various hanging branches, please? For example, the patch opened by Dom?

It would be good to add information on how people can:

• Raise Issues
• Contribute
• Seek support

I'd suggest mentioning which versions you used for major python libraries, especifically pytorch, matplotlib, pandas, numpy. For example, just saying: "The exercises were tested with versions: pytorch=2.2.2, pandas=2.1.0, etc".

In general, I think this is a good idea for future-proofing the repository -- is useful for users to know which versions should work. (For me it's ok that versions are not specified in the pyproject.toml, in some cases I've found problems when too many packages and versions are specified)

Going through the README I noticed problems with these links:

• the URL jackatkinson.net is not working. If the website does not exist anymore, I'd suggest removing the link from the README

• There is a mention to "existing open issue" in the Contribution Guidelines with a link to open a new issue, but I'd say it should be a link to issues

Add instructions for creating a virtual env and installing the dependencies.

Running CNNs on CPU with apple silicon is prohibitively slow.

Pytorch now has an mps backend that works very similarly to cuda to alleviate this.

Consider adding to exercises.

Should we consider changing the repo name to practical DL with pytorch?

For some reason the non-linear activation functions have been removed from the models in the solutions. This may well work, but it seems very irregular and, since this is for educational purposes, it should be corrected.