datalass1 / fastai Goto Github PK

Testing new laptop 21/12/2018. Seems to work fine technically so far if not a little slower (freezes) and stickier on the mouse pad.

Issue: with Jupyter notebook. Error when opening a fastai notebook 'The error was: [sprintf] expecting number but found string See the error console for details.'
solution: https://forums.fast.ai/t/truncated-notebooks/24533

1. work from the latest code in the fastai repo
2. update conda (and pip too, why not) and ensure you can run conda activate
3. destroy the fastai conda environment
4. setup the fastai conda enviroment
5. re-configure the IP address for Jupyter Notebook
6. rm -r fastai
7. git clone https://github.com/fastai/fastai.git

We start lesson 3 looking at an interesting dataset: Planet's Understanding the Amazon from Space. In order to get this data in to the shape we need it for modeling, we'll use one of fastai's most powerful (and unique!) tools: the data block API.

One important feature of the Planet dataset is that it is a multi-label dataset. That is: each satellite image can contain multiple labels, whereas previous datasets we've looked at have had exactly one label per image. We'll look at what changes we need to make to work with multi-label datasets.

Next, we will look at image segmentation, which is the process of labeling every pixel in an image with a category that shows what kind of object is portrayed by that pixel. We will use similar techniques to the earlier image classification models, with a few tweaks. fastai makes image segmentation modeling and interpretation just as easy as image classification, so there won't be too many tweaks required.

We will be using the popular Camvid dataset for this part of the lesson. In future lessons, we will come back to it and show a few extra tricks. Our final Camvid model will have dramatically lower error than an model we've been able to find in the academic literature!

What if your dependent variable is a continuous value, instead of a category? We answer that question next, looking at a keypoint dataset, and building a model that predicts face keypoints with high accuracy.

image classification, covering several core deep learning concepts that are necessary to get good performance: what a learning rate is and how to choose a good one, how to vary your learning rate over time, how to improve your model with data augmentation (including test-time augmentation). We also share practical tips (such as training on smaller images), an 8-step process to train a world-class image classifier

Overview
In the final lesson of Practical Deep Learning for Coders we'll study one of the most important techniques in modern architectures: the skip connection. This is most famously used in the resnet, which is the architecture we've used throughout this course for image classification, and appears in many cutting edge results. We'll also look at the U-net architecture, which uses a different type of skip connection to greatly improve segmentation results (and also for similar tasks where the output structure is similar to the input).

We'll then use the U-net architecture to train a super-resolution model. This is a model which can increase the resolution of a low-quality image. Our model won't only increase resolution—it will also remove jpeg artifacts, and remove unwanted text watermarks.

In order to make our model produce high quality results, we will need to create a custom loss function which incorporates feature loss (also known as perceptual loss), along with gram loss. These techniques can be used for many other types of image generation task, such as image colorization.

Finally, we'll learn about a recent loss function known as generative adversarial loss (used in generative adversarial networks, or GANs), which can improve the quality of generative models in some contexts, at the cost of speed.

The techniques we show in this lesson include some unpublished research that:

Let us train GANs more quickly and reliably than standard approaches, by leveraging transfer learning
Combines architectural innovations and loss function approaches that haven't been used in this way before.
The results are stunning, and train in just a couple of hours (compared to previous approaches that take a couple of days).

https://forums.fast.ai/t/useful-jupyter-notebook-tips-plugins-collapsible-sections/17919

Today we discuss some powerful techniques for improving training and avoiding over-fitting:

Dropout: remove activations at random during training in order to regularize the model
Data augmentation: modify model inputs during training in order to effectively increase data size
Batch normalization: adjust the parameterization of a model in order to make the loss surface smoother.
Next up, we'll learn all about convolutions, which can be thought of as a variant of matrix multiplication with tied weights, and are the operation at the heart of modern computer vision models (and, increasingly, other types of models too).

We'll use this knowledge to create a class activated map, which is a heat-map that shows which parts of an image were most important in making a prediction.

Finally, we'll cover a topic that many students have told us is the most interesting and surprising part of the course: data ethics. We'll learn about some of the ways in which models can go wrong, with a particular focus on feedback loops, why they cause problems, and how to avoid them. We'll also look at ways in which bias in data can lead to biased algorithms, and discuss questions that data scientists can and should be asking to help ensure that their work doesn't lead to unexpected negative outcomes.

Lesson 3 30mins in shows how to upload competition data results to Kaggle.

Make sure this is tested in next notebook using Kaggle.

lesson 1 suggests at 30mins to use the classifier on my own imagery.

For example, test on gull and auk species. panda and polar bear. elephant and whale.

https://arxiv.org/pdf/1506.01186.pdf

Overview

In lesson 5 we put all the pieces of training together to understand exactly what is going on when we talk about back propagation. We'll use this knowledge to create and train a simple neural network from scratch.

We'll also see how we can look inside the weights of an embedding layer, to find out what our model has learned about our categorical variables. This will let us get some insights into which movies we should probably avoid at all costs…

Although embeddings are most widely known in the context of word embeddings for NLP, they are at least as important for categorical variables in general, such as for tabular data or collaborative filtering. They can even be used with non-neural models with great success.

Great notes: https://forums.fast.ai/t/deep-learning-lesson-5-notes/31298

We start today's lesson learning how to build your own image classification model using your own data, including topics such as:

• Image collection
• Parallel downloading
• Creating a validation set, and
• Data cleaning, using the model to help us find data problems.

I'll demonstrate all these steps as I create a model that can take on the vital task of differentiating teddy bears from grizzly bears. Once we've got our data set in order, we'll then learn how to productionize our teddy-finder, and make it available online.

In the second half of the lesson we'll train a simple model from scratch, creating our own gradient descent loop. In the process, we'll be learning lots of new jargon, so be sure you've got a good place to take notes, since we'll be referring to this new terminology throughout the course (and there will be lots more introduced in every lesson from here on).

To make sense of learning rates, augmentation, stochastic gradient descent with restarts, precompute=True/False, freeze and unfreeze, epochs, over-fitting, training time augmentation (TTA), confusion matrix, cycle_len and cycle_mult.

Also:
Exercise: look at the other cases and convince yourself that this make sense.
Exercise: how would you rewrite binary_loss using if instead of * and + ?

Learning about metrics, loss functions, and (perhaps the most important machine learning concept) overfitting. We discuss using validation and test sets to help us measure overfitting.

Then we’ll learn how random forests work - first, by looking at the individual trees that make them up, then by learning about “bagging”, the simple trick that lets a random forest be much more accurate than any individual tree.

Next up, we look at some helpful tricks that random forests support for making them faster, and more accurate.

Bulldozers Kaggle dataset

14mins: R2 is a measure of score for random forest: Returns the coefficient of determination R^2 of the prediction.
In statistics, the coefficient of determination, denoted R2 or r2 and pronounced "R squared", is the proportion of the variance in the dependent variable that is predictable from the independent variable(s). source: Coefficient_of_determination
If R2 is less than 0 then the model is worse than predicting the mean. R2 is anything less than 1 with 1 being the best score for the model.

A great github resource: https://github.com/cedrickchee/knowledge/blob/master/courses/fast.ai/machine-learning/2017-edition/lesson-2-random-forest-deep-dive.md

What ssh keys exist:
$ ls -al ~/.ssh

Generating a new SSH key:

ssh-keygen -t rsa -b 4096 -C "[email protected]"

Adding your SSH key to the ssh-agent

Start the ssh-agent in the background:
$ eval "$(ssh-agent -s)"
Add your SSH private key to the ssh-agent:
$ ssh-add ~/.ssh/id_rsa

Add the SSH key to the GitHub account:

Copy the SSH key to your clipboard:

$ sudo apt-get install xclip
$ xclip -sel clip < ~/.ssh/id_rsa.pub

On GitHub go to profild pic > settings > SSH and GPG keys > New SSH key > Add SSH key

source

I have a Google Cloud Platform account. Previously I used the GeoCode API to change postcodes from a housing dataset to coordinates for mapping house prices in Somerset.

https://console.cloud.google.com/home/dashboard?project=geopackage-223116

I have £230.58 credit which will end on 9th November.

Using Multilabel classification create a notebook for Understanding the Amazon from Space competition.

Using Structured data approach with some Machine learning techniques create a notebook for the Rossmann Store Sales competition.

can I store the code written in paperspace machine to my machine and GitHub repo for future reference/portfolio.

Start of your fast.ai journey! In today’s lesson you’ll set up your deep learning server, and train your first image classification model (a convolutional neural network, or CNN), which will learn to distinguish dogs from cats nearly perfectly.

Today we discuss some powerful techniques for improving training and avoiding over-fitting:

Dropout: remove activations at random during training in order to regularize the model
Data augmentation: modify model inputs during training in order to effectively increase data size
Batch normalization: adjust the parameterization of a model in order to make the loss surface smoother.
Next up, we'll learn all about convolutions, which can be thought of as a variant of matrix multiplication with tied weights, and are the operation at the heart of modern computer vision models (and, increasingly, other types of models too).

We'll use this knowledge to create a class activated map, which is a heat-map that shows which parts of an image were most important in making a prediction.

Finally, we'll cover a topic that many students have told us is the most interesting and surprising part of the course: data ethics. We'll learn about some of the ways in which models can go wrong, with a particular focus on feedback loops, why they cause problems, and how to avoid them. We'll also look at ways in which bias in data can lead to biased algorithms, and discuss questions that data scientists can and should be asking to help ensure that their work doesn't lead to unexpected negative outcomes.

The key outcome of this lesson is that we'll have trained an image classifier which can recognize pet breeds at state of the art accuracy. The key to this success is the use of transfer learning, which will be a key platform for much of this course. We'll also see how to analyze the model to understand its failure modes. In this case, we'll see that the places where the model is making mistakes is in the same areas that even breeding experts can make mistakes.

learning rates:

https://towardsdatascience.com/estimating-optimal-learning-rate-for-a-deep-neural-network-ce32f2556ce0
https://miguel-data-sc.github.io/2017-11-05-first/

pytorch:

https://towardsdatascience.com/a-practitioners-guide-to-pytorch-1d0f6a238040

advanced:

http://teleported.in/posts/decoding-resnet-architecture/

download bulldozers data from https://www.kaggle.com/c/bluebook-for-bulldozers/data (firefox browser)

• sign in
• crtl shift i
• developer console go to network
• click to download train.zip
• don't download it but see that googleapis opened
• right click and copy cURL
• in paperspace cmd line paste cURL and type in '--output bulldozers.zip'
• ls, mkdir bulldozers, mv bulldozers.zip bulldozers, cd bulldozers/, unzip bulldozers.zip

Find the chapters to optimise learning in the Deep Learning Book

https://arxiv.org/pdf/1609.04802.pdf

https://arxiv.org/pdf/1606.08921.pdf - good link https://github.com/vdumoulin/conv_arithmetic

https://github.com/alexjc/neural-enhance

Cloud GPU not installed

Might have to re-visit lesson 1 first 15 minutes to set up Paperspace

£££ to set up and use a GPU

The key piece of a convolutional neural network is convolution.
A good example is http://setosa.io/ev/image-kernels

Most kernels in deep learning are 3x3 really fantastic interactive book http://neuralnetworksanddeeplearning.com/chap4.html

A great paper about what the convolutional layers are learning: Visualizing and Understanding Convolutional Networks, Matthew D. Zeiler, Rob Fergus: https://arxiv.org/pdf/1311.2901.pdf

In lesson 4 we'll dive in to natural language processing (NLP), using the IMDb movie review dataset. In this task, our goal is to predict whether a movie review is positive or negative; this is called sentiment analysis. We'll be using the ULMFiT algorithm, which was originally developed during the fast.ai 2018 course, and became part of a revolution in NLP during 2018 which led the New York Times to declare that new systems are starting to crack the code of natural language. ULMFiT is today the most accurate known sentiment analysis algorithm.

The basic steps are:

Create (or, preferred, download a pre-trained) language model trained on a large corpus such as Wikipedia (a "language model" is any model that learns to predict the next word of a sentence)
Fine-tune this language model using your target corpus (in this case, IMDb movie reviews)
Extract the encoder from this fine tuned language model, and pair it with a classifier. Then fine-tune this model for the final classification task (in this case, sentiment analysis).
After our journey into NLP, we'll complete our practical applications for Practical Deep Learning for Coders by covering tabular data (such as spreadsheets and database tables), and collaborative filtering (recommendation systems).

For tabular data, we'll see how to use categorical and continuous variables, and how to work with the fastai.tabular module to set up and train a model.

Then we'll see how collaborative filtering models can be built using similar ideas to those for tabular data, but with some special tricks to get both higher accuracy and more informative model interpretation.

This brings us to the half-way point of the course, where we have looked at how to build and interpret models in each of these key application areas:

• Computer vision
• NLP
• Tabular
• Collaborative filtering

For the second half of the course, we'll learn about how these models really work, and how to create them ourselves from scratch. For this lesson, we'll put together some of the key pieces we've touched on so far:

• Activations
• Parameters
• Layers (affine and non-linear)
• Loss function.

We'll be coming back to each of these in lots more detail during the remaining lessons. We'll also learn about a type of layer that is important for NLP, collaborative filtering, and tabular models: the embedding layer. As we'll discover, an "embedding" is simply a computational shortcut for a particular type of matrix multiplication (a multiplication by a one-hot encoded matrix).

1. what is numpy and fastai libraries doing
2. what will more epochs do, will I overfit my data? What is the optimum number of epochs
3. do this with the 'test lesson 1 code on my own imagery' lesson and timebox to 2hours

We explain convolutional networks from several different angles: the theory, a video visualization, and an Excel demo. You’ll see how to use deep learning for structured/tabular data, such as time-series sales data.

We also teach a couple of key bits of math that you really need for deep learning: exponentiation and the logarithm. You will learn how to download data to your deep learning server, how to submit to a Kaggle competition, and key differences between PyTorch and Keras/TensorFlow.

A key advantage of neural networks as opposed to traditional machine learning techniques is that we can avoid feature engineering our raw input, i.e. constructing pre-determined features from our raw data that we think is important. (source: fastai wiki)

In training convolutional neural networks, we allow stochastic gradient descent to find optimal values for filter weights based off of the raw data itself, and in this way we've allowed our network to "learn" what the best features are from the raw data given labelled output. This is incredibly powerful, and it removes the onerous task of manually designing features. (source: fastai wiki)

Extra stuff I found:

• this really cool video visualising what happens inside of neural nets: Deep Visualization Toolbox and associated webpage.

install using the linuxbabe guide.

wget https://dl.google.com/linux/linux_signing_key.pub

Then use apt-key to add it to your keyring so the package manager can verify the integrity of Google Chrome package.

sudo apt-key add linux_signing_key.pub

Now update package list and install the stable version of Google Chrome.

sudo apt update

sudo apt install google-chrome-stable

Launch with:
google-chrome-stable

Entity Embeddings of Categorical Variables
Cheng Guo, Felix Berkhahn

• discussed in lesson 5 @ 1hr07mins

Create a PyTorch linear function, write function from scratch, learn to debug.
See Lesson 5: Back propagation; Accelerated SGD; Neural net from scratch at 1hr31mins for reference to this issue.

Tool for deleting files on the Google Image Search page before downloading
Tips for building large image datasets

Check out computation linear algebra with Rachel Thomas.