2020/02/illustrated-self-supervised-learning/ about blog-comments HOT 19 CLOSED

@vs74 It's draw.io

from blog-comments.

Sure @asanokoy

from blog-comments.

@numanai Sure.

Self-supervised: learn by trying to predict some part of the data itself
Semi-supervised: use both labeled(by humans) + unlabeled data to learn. I've written an overview here.
Meta-learning: This is basically "learning to learn". The idea is that if we expose a model to tons of small tasks and training data, then it can generalize to a new similar task very quickly.

from blog-comments.

@ariG23498 Thank you. I'm glad you liked it.

The sidebar navigation is a great suggestion. I'll try incorporating it.

from blog-comments.

For the section abut learning using Image clustering as self-supervision, you could also add our paper as a reference.
CliqueCNN: Deep Unsupervised Exemplar Learning, NIPS2016
https://arxiv.org/pdf/1608.08792.pdf

from blog-comments.

Gidaris et al. - link to this paper under the section of Geometric Transformation recognition is redirecting to a wrong paper - Unsupervised Visual Representation Learning by Context Prediction

Please correct - https://arxiv.org/abs/1803.07728

from blog-comments.

@sankalp-s Thanks for pointing out. I've corrected the link.

from blog-comments.

Great read!

from blog-comments.

Amazing way of explaining, an introuduction to "self-supervised" learning, especially with great images and animations. By the way, which tool do you use for creating images?

from blog-comments.

Hello Please have you a code for SimCLRv2 ?

from blog-comments.

@aynesss It's available here: https://github.com/google-research/simclr

from blog-comments.

Thanks for such a detailed explanation. Please can you comment on the difference between self supervised, semi supervised and meta learning?

from blog-comments.

Will you include some introduction about contrastive learning, like MoCo?

from blog-comments.

@iamhankai Ankesh Anand has written a great overview on contrastive methods here.

from blog-comments.

@amitness for SimCLR v2, in finetune, we must keep or select which layer of projection head ? the middle one? (Dense)

from blog-comments.

@ines321 As per the paper, finetuning from the middle layer of the projection head gives better performance. See Figure 5 of the paper.

from blog-comments.

Hey @amitness great read!
The structure of the blog is top notch.

Feature request: A sidebar with the heading pointers would be great to have. We could then eventually jump from one section to other as need be.

from blog-comments.

Hey @amitness

What is the real purpose of term temperature in the loss function of SimCLR? Please can you help in understanding it with some intuitive example. Also, I found this temperature term in the MoCo paper; both of them means the same?

I found the following comment on this blog post (https://towardsdatascience.com/contrasting-contrastive-loss-functions-3c13ca5f055e), but I don't think that I really understood what does it mean.

"Chen et al. found that an appropriate temperature parameter can help the model learn from hard negatives. In addition, they showed that the optimal temperature differs on different batch sizes and number of training epochs."

Thanks

from blog-comments.

Hey @amitness
After pretraining with SimCLR, I save the model pretrained with SimCLR. After that, I load the saved model, get the first layer of projection head and finetuning. But accuracy is bad.
Why accuracy of finetuning is very bad (40 %)

from blog-comments.