wandb / edu Goto Github PK

Matching the questions on slides

Need to refactor it into a base class and two dataloaders for classification and for image tasks

2. Choosing an output activate
   • Review the logged outputs of your neural network in the Weights & Biases interface and look for issues caused by this choice of activation.
   • For example, here they are always between 0 and 1, to match the data they are compared against, while while activation values have no such restriction.
3. Change Hyperparameters
   • Better results can be obtained by tweaking the hyperparameters.
4. Challenge: Regularization and Learned Weights
   • With the default settings, the leanred filters don't look like that at all.

https://github.com/wandb/edu/blob/4a438532a4cd53a025c348ecabed93bcc5dee646/lightning/autoencoder/autoencoder-mnist.ipynb

Based on new art collateral

Ideas

1. Little-o exercises. They'd be multiple choice, basically. And/or: provide a function that is little-o of x^2? I can check this with sympy.
2. Linearity of the gradient approximation. Write a function to compute gradient approx, given the function and its gradient. Compare to finite differences?

this way, folks are always logged in before they get to the meat of the code

in particular, how much can i change the sizes of the convs before the adaptive pooling layer complains? how much can i change the target size of the adaptive pooling layer? how deep can i go before MNIST images are too small?

• better naming and saving
• add config metadata

This was written when I had half as much experience with Lighting as I have now, and before the most recent integration.

I should rethink it, with emphasis on the following:

• moving material into callbacks
• cutting down on the magic parameter inference
• improve robustness (see #25)
• reorganize and make hierarchical (see #20)
• writing some tests, including e2e tests that write runs to wandb and pull metrics

right now, the installs are run regardless of environment, but they should only be run in Colab.

just need to move the !pip install commands into the appropriate if branch and then apply %%capture to the cell

On the one hand, it will reduce code duplication across Colabs, but on the other, it really needs to be done well if it's going to be used everywhere -- have to make sure it's e.g. DDP compatible, the logging could be done better (more callbacks?), and want to use PL best practices as much as possible.

See projects/constrained_emotion_classifier.ipynb for an example. This is better than the name-sensitive style being used elsewhere.

beginning to be deprecated

waiting on the branches to be correct

Doesn't need to be provided for convolutional networks.

Should fix and possibly update the docs example.

Adding more content and at the same time de-emphasizing existing content, but we don't want to delete that content, so let's make a nested structure with folders and subfolders

-- 00_{topic}/
  |  exercises.ipynb
  --  extras/
    |  {subtopic}.ipynb
-- 01_{topic}/ 

Retain only the components that are easily autograde-able or could be converted to autograding.

crib the HTML from the notebooks in the examples repo.

Should be as easy as calling print(model) inside the right hook, which might be on_train_start, with the LoggedLitModule

Caught on a dilemma with saving PyTorch models for viewing in Netron

• OT1H, just saving as a .pt file results in unreliable performance by Netron, who can't really be expected to handle all the possible choices in both major libraries, and so prefers ONNX and lets them handle the conversion, but
• OTOH, ONNX can't handle e.g. the AdapativeAvgPool2d layer, important for CNNs that are easy to play with. Seems like a pretty fundamental limitation w.r.t adaptive layers.

In particular, for the MLP and CNN in PyTorch, I want to emphasize reusability and enable easy extension, and so I'm using ModuleLists and custom Modules. Neither is really gonna play nicely with Netron.

For now, I'm sticking with .pt files, which save but aren't visualized well (the ModuleLists aren't expanded, and that's where all the action is!).

Would like to drop reliance on the W&B Hub, and Colab seems like the least-bad choice.

Options:

1. Local. No lol. Installation issues are a dealbreaker.
2. Binder. Ephemerality will be frustrating for students, and is a dealbreaker for classes where the HW is done between sessions.
3. Colab. Avoids ephemerality. The limitation to the matplotlib inline backend for interactive charts is frustrating but not a showstopper.
4. Gradient. Requires a login, possibly expensive, unclear if better, GPUs not needed for this class.

torch.nn.Identity serves the same purpose, but is less jank.

See cnn.ipynb.

Architecture search was put under cnn/ when it fits much better under projects/. For now it's duplicated, but it should only be in projects/.

There are currently two versions of the FER project, but there should only be one. Once the more robust NN utils are in place (#20, #25, #26) can reduce.

The profiling tool may only be compatible with Chrome.

• ✅ Chrome
• ❌ Safari
• ❔ Firefox
• ❌ Edge (not fully supported by Colab)

Networks that are being pruned with torch.nn.prune break certain assumptions in e.g. the parameter counting, as do I believe the quantized networks.

These should be resolved (incorporating fixes from the relevant notebooks, when possible) so that the utils are more robust.

• linearalgebra
• calculus
• probability

it's interesting, but not autogradable.

this needs to be installed locally, but is not a pip package

Right now, doesn't appropriately test whether there are values above 1 or below 0 because the examples don't add up to 1, which is what most folks test for.

np.array([-1, 2]) and/or np.array([-0.5, 1.1, 0.2, 0.2]) would do it.

_Note_: renders differently between platforms because Markdown is an incomplete spec, resulting in some very wonky formatting in places; use _Note:_ instead.

based off of README in ml-class

Colab defaults to 2-space but much of the code is in 4-space -- and other Jupyter instances don't like 2-space indentation

In calculus exercises: is_little_o, identity, constant

The SVD material is interesting, but hard to make concrete and compelling with the constraints we have (unless I come up with a slick "LA-as-programming" explanation of kernels and maybe also eigenvalues, which is tougher).

I should move it into a separate notebook.

should make the dataloaders more configurable for the AbstractMNISTDataModule -- can probably fix pin_memory to True, but should allow configuration of num_workers (with a default of 2 or nproc, depending on how far we want to go)

Should be ## Setup Code, # Section X, etc.

This is the core idea of the lecture slides, but there aren't enough exercises for it. They require a certain amount of creativity, but here's a few possibilities:

1. Shapes and types. Debug a shape issue? Write code that checks a shape, by analogy with checking a type?
2. Batch dimensions/broadcasting and for loops. Write a for loop that does the same thing as a broadcasted multiply. Make it a "batch" application.
3. Convolutions and zips. Harder: write a zip that's equivalent to a convolution.
4. Parallel composition. Use concat/stack operations to implement applying two "functions" to the same input. Something else with the Kronecker product?
5. repeat. Use matrix multiplication (an outer product?) to copy the input k times.

Ideas:

1. PMFs and PDFs. Dictionaries versus densities.
2. Softmax. Define a softmax function for a PMF.
3. Cross entropy loss. Implement it, based on a formula? Based on PMF or PDF?

Given the gradient and parameters, apply one gradient descent step and return the new parameters.

Check that:

• Stationary at a maximum and a minimum
• Stationary if lr is 0
• Optimizes quadratic in one step with the right lr
• Works on vector-valued versus scalar-valued inputs?

should move to /content before git clone

Put the links

• in a table in the README
• in the notebooks themselves, where appropriate

Logger is used elsewhere in the Lightning API and so I should avoid the name collision

The Binder setup needs to be tweaked now that we're in v2. The Dockerfile is no longer in the right place, which is going to be a PITA to fix. That Dockerfile also needs to be updated.

The utils I wrote for the lightning/mlp/ notebooks are useful more broadly in the lightning material.

They should perhaps be moved up to the lightning/ folder. This will require changing some code in the extant lightning colabs.

in v2, this is pointing to the v2 branch