This looks like a great initiative, and it's something that was much needed.

But if you could also add the explanation/resources of some of the 'elementary' research papers (and by elementary I do not mean simple - rather the basic/foundation), like papers introducing CNNs or RNNs, or some regularization technique like dropout, it would be really helpful.

The reason for this is that if you have to learn about GAN (say a specific use-case of GAN like neural-style transfer), it's necessary that you should have the required knowledge regarding CNNs.

Although I know that explaining each and every paper out there in the domain of ML is not plausible, but maybe this could be done for some of the foundational papers.

I think there may be an error in Problem Set 1 2e:

Specifically, x = phi(h), whereas the integrand term has phi(x). If this is a mistake, the problem is carried forward to

The dependency graph is the thing that looks like this:

Note that these links live inside an IFRAME (that contains the SVG content), so it may take a bit of custom JavaScript.

I'd like to add a curriculum for the recent paper Neural mechanics: symmetry and broken conservation laws in deep learning dynamics. There is a lot of depth to this paper, and the payoff (being able to calculate exact training dynamics of certain parameter combinations) makes that worth studying. Like the Resurrecting the Sigmoid curriculum, I'd probably structure it to be mostly based around problem sets, except with more interactive (coding/jupyter notebook) problems. If this sounds good, I can get started.

Thanks for the exceptional depthfirstlearning material!

I am currently working through the InfoGAN material and put together a small repo with my solutions.
If the solutions are of interest for the InfoGAN page I could add them to your repo.

I would be happy contribute to this great project.

Kind regards
Michael

http://www.depthfirstlearning.com/2018/DeepStack
Noam Brown's same video is the 2nd optional reading in Part 1 and also the 2nd optional reading in Part 6

http://www.depthfirstlearning.com/2018/AlphaGoZero
Optional readings 3 and 4 in Part 1 are dead links.
Working links are respectively: https://www.chessprogramming.org/index.php?title=Minimax, https://www.chessprogramming.org/Alpha-Beta

Hi,
I am a CS undergrad. i would like to implement the following papers.
Please let me know the subjects that i need to know to understand it. Could you please help me ?

https://people.csail.mit.edu/rishabh/papers/cav15-ranking.pdf

https://www.semanticscholar.org/paper/RobustFill%3A-Neural-Program-Learning-under-Noisy-I%2FO-Devlin-Uesato/3ff0af64279929a952ee340e645256b7e0580f65

Thank you.

Why is Jensen-Shannon divergence called this way?

The answer is something like this: https://dit.readthedocs.io/en/latest/measures/divergences/jensen_shannon_divergence.html#derivation (where "x" is a convex combination of P and Q), and the expectation is taken over the binary variable defining which of {P,Q} to take

We'd like a clear write-up of this definitely of the JS divergence, including a proof of the equivalence between this definition and the others.

It appears you are trying to achieve the same goals as https://distill.pub/. Why having a distinct platform?

It looks pretty -- we should put it more places!

Hey,

as I am strong supporter of "Depth First Learning" and am very intrigued by the material, I wanted
to suggest to include a curriculum on "A correspondence between Random neural networks and statistical field theory" (Pennington, Schoenholz, 2017).

Even though there are some more recent developments on statistical field theory for neural networks (e.g. Helias, Dahmen, 2020 or Grosvenor, Jefferson, 2021) I think, that the paper of Pennington and Schoenholz provides rich opportunities to create a curriculum which covers not only NN and statistical field theory but also some basic introduction in to replica methods.

As the publication is in the realm of materials which I study for my PhD, I could design a curriculum.
Greetings
Javed (Lindner), (PhD student, RWTH Aachen University)

(Just a GAN implementation, no InfoGAN).

I'd like to show the specific differences in training for the two losses described in the original GAN paper.

A good Colab would include an abundance of text cells explaining exactly what each part is doing. You can put math in text cells, followed by TensorFlow code implementing that math.

I am wrapping my head around the explanation of the vanishing gradients problem of GANs for quite some time:

The current solution pdf document plots the input of the sigma functions over the loss function values to explain the (non-)saturating behavior. However, I am asking myself if that plot just captures the saturation of the sigma function and not the saturation behavior of the G loss function itself.

The NIPS 2016 GAN tutorial shows in figure 16 (p.26) an explanation of the saturating loss without taking the sigma function into account. With this explanation, I guess, the saturation behavior is explained through the gradients for G when G is not (yet) able to generate good fakes and D can easily identify them as fake (x = 0 or close to 0).
See a plot of the saturating and non-saturating loss function and their derivations. There, the saturating loss has a small gradient of around -1 and the saturating loss of -infinity at x = 0.
When I plot the gradients over the training for both loss functions I also get higher gradient means and higher standard deviations for the non-saturating loss when compared to the saturating loss (see notebook).

Maybe I am missing something?

I would be happy if somebody could point me in the right direction.

http://www.depthfirstlearning.com/about/
Typo 1:
Second paragraph, third sentence, extraneous "is".
"We spent some time understanding each paper and writing down the core concepts on which they were is built."

Typo 2:
Fourth paragraph, fourth sentence, duplicate "out".
"Check out out github for examples and open an issue detailing which paper you want to study."

People don't notice that the dependency graph at the top of InfoGAN is clickable.

The link for the resource is https://colah.github.io/posts/2015-09-Visual-Information/

I found this article a great resource for introduction to Information Theory, and it might be helpful to have as many resources as possible available optionally for the in-depth style.

Hey all!

I've just been taking a look at the SVGD curriculum and unfortunately, there are a number of broken links. Hopefully, some of these can be fixed without too much trouble! Here is a non-exhaustive list:

1. https://www.win.tue.nl/~rvhassel/Onderwijs/Old-Onderwijs/2DE08-1011/ConTeXt-OWN-FA-201209-Bib/Literature/sigma-algebra/gc_06_measure_theory.pdf
2. https://www.ims.nus.edu.sg/Programs/stein09/files/A%20Gentle%20Introduction%201.pdf
3. https://www.cs.dartmouth.edu/~qliu/PDF/ksd_short.pdf
4. https://www.cs.dartmouth.edu/~qliu/PDF/slides_ksd_icml2015.pdf
5. https://www.cs.dartmouth.edu/~qliu/PDF/steinslides16.pdf

I found (I think) an alternative source for 1) here: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.124.3854&rep=rep1&type=pdf, but the rest I didn't manage. There could also be more missing links.

Cheers,

James