As the title says, the scope is:

• find a custom unlabelled dataset, or find a labelled dataset and show how to create segmentation labels yourself. Dataset has to have continuity to datasets used in #49 and #47
• show how to label this dataset (create semantic segmentation labels for each pixel) with a custom smart labelling tool such as http://www.cs.toronto.edu/~amlan/demo/ (polygon RNN)
• write notebook detailing this work training a simple DNN model for semantic segmentation.

The scope of the work includes using a pre-trained semantic segmentation tool (does not include showing how to train such a tool)

Similar to IC, build a notebook called "01_training_introduction.ipynb" which introduces the problem, without going into details.
This notebook can be heavily based on the fast.ai's 2019 lesson3-camvid.ipynb and lesson3-camvid-tiramisu.ipynb notebooks:
https://github.com/fastai/course-v3/blob/master/nbs/dl1/

Similar to IC, find good default parameters for (i) fast accuracy (e.g. deep model, high resolution); and for (ii) fast inference (e.g. low resolution) using ~5 image segmentation datasets.

Using e.g. flake

TODO: placeholder for 2.5D semantic segmentation regarding how to work with 2.5D data in general - nothing to do with oil and gas.

E.g. using cans, bottles, and milk boxes.

• Single-label image classification: single object per image. Maybe also some "negative" images without a single object-of-interest in them.
• Multi-label image classification: 0 or more objects in image.

Maybe 50 images for each of the two datasets?

Important to show if/what the value is of parameter optimization using hyperdrive versus using default parameters.

tests/unit/test_experiments.py:26:

>           mkdir(name, mode)
E           FileExistsError: [Errno 17] File exists: 'tmp_data'

tests/unit/test_experiments.py:71: AssertionError

>       assert df.mean(level=(2)).loc["fridgeObjects", "accuracy"] > 0.85
E       assert 0.7954545617103577 > 0.85

Create a notebook which compares fast.ai U-net model with pixel-level classification model https://github.com/waldeland/CNN-for-ASI and explains how classification compares to segmentation (semantic 2D segmentation in this case).

Description

The default fastai record callback is useful but limited.
Add custom callback function so that we can track more information while training.

Implement model https://github.com/waldeland/CNN-for-ASI for fast.ai in pyTorch, push to fast.ai and expose as baseline learner (show pixel-level classification model from this link compared to say UNet model here #47). Could add to #47 notebook to show how much slower classification models work for segmentation.

In the webcam notebook, we print out the name of the device that will be used to train your model. Would it be possible to provide some guidance, for the users, on:

• how to know whether their machine has a GPU installed or not
• what they need to do to get the model to train on the GPU instead of a CPU, if the former is present
• what to expect if their machine doesn't have a GPU.

Might want to support both Windows and Linux.

Compile ~5 image segmentation datasets. Similar to IC, these will be used to find good default parameters, for integration tests, etc.
The datasets should be small since (i) otherwise running parameter tuning takes a long time; and (ii) most customer only have 100s of images, even for image classification.

TODO: placeholder for 3D semantic segmentation specific for oil and gas based on existing MSRA work (Beijing).

Find default parameteres (e.g. learning rate) which work well for many IC problems, for shallow/deep models, for low/high res image, etc. See CVTK's default parameters table. Uses the internal datasets for single-class and multi-class image classification.

Remark:

• we already have 4-5 datasets for single-class problems which were used to evaluate the CVTK.
• datasets should be diverse to cover large spectrum of use cases
• datasets should be small (500-5000 images) so that can run parameter sweep efficiently

Use the basic notebook in combination with an oil & gas dataset. Tweak parameters to improve accuracy, etc. As a second step, evaluate accuracy improvements using spatial consistency smoothing.

Push DeepLab model https://github.com/lexfridman/mit-deep-learning/blob/master/tutorial_driving_scene_segmentation/tutorial_driving_scene_segmentation.ipynb to fast.ai and show how it compares to other models which we already wrote about in #47

Setted up - https://dev.azure.com/best-practices org and https://dev.azure.com/best-practices/computervision project for CV
Setted up DSVM (Linux and Windows, GPU supported) to use as private agents.

Nominate a code champion which will monitor and drive code quality

Widgets may not rendered correctly on some browsers. Should mention that from the notebook. (mainly in the web-cam notebook).

Ideally show (at the very least explain) in an example where hard-negative mining makes a big difference. E.g. training a deer classifier, which also fires on trees.

In addition to code, this should also include a description when to use and when not to use various image augmentations.

Measure how/if training time decreases when using an optimized Pillow package.

Fastai's conda (test) channel has an experimental pillow package built against a custom build of libjpeg-turbo. See: https://docs.fast.ai/performance.html

TODO: this is a placeholder for hyperparameter tuning. Pending on 2D semantic segmentation and 2D Oil and Gas scenarios, we will see what dataset and model we want to pick for this notebook - but the accent should be on scalable hyperparameter tuning of segmentation models.

Should at the very least support a high-accuracy model (e.g. ResNet) and a high-speed model (e.g. MobileNet)

Added documentation for working with git hooks and pre-commit
how to use black and flake8
How to work with notebooks.

Ideally this should use batching to speed up feature computation. Also, could include a toy example, e.g. k-means clustering of an image dataset

• Examples which are, and which are not suited for image classification and object detection
  • For example:
    • Image classification: area of interest is tiny in the image (e.g. logo far in the background)
    • Object detection: object to be found is hard to define where it starts/ends
• Annotation
  • How to annotate / how to not annotate
• Common pitfalls and how to avoid
• Trade-off speed versus accuracy in training and inference
• etc.

To use gpu for training, users will need certain size of gpu memory, depending on the image and batch size. Would be good to have minimum requirements to each notebook to use the default parameters shown in the notebook.

I got an gpu memory error while testing the training notebook from my laptop which has a tiny gpu (gtx1050 2g).

Use findings from #13 to create a notebook that shows the spectrum of fast/accurate DNNs, low/high resolution, low/high memory (Resnet-18 and 299p, Resnet-50 with 499p, MobileNet with 299p)

Remark: maybe make this a a single notebook which can handle single-label classification and multiple-label classification problems, think the IC "hello world". Then other notebooks will focus on more advanced topics.