microsoft / arcticseals Goto Github PK

A deep learning project in cooperation with the NOAA Marine Mammal Lab to detect & classify arctic seals in aerial imagery to understand how they’re adapting to a changing world.

License: MIT License

Batchfile 0.01% JavaScript 1.40% C# 0.60% Python 12.13% Jupyter Notebook 84.58% CSS 0.01% HTML 0.15% Dockerfile 0.15% Shell 0.04% C++ 0.32% Cuda 0.54% C 0.06% Cython 0.02%

aiforearth

arcticseals's Introduction

Overview

This is the workspace for the a collaboration between Microsoft AI for Earth, NOAA Fisheries, and the University of Washington, aimed at automating the detection of arctic wildlife in aerial imagery.

Imagery

Imagery is now available publicly on lila.science, an open data repository for labeled images related to conservation biology.

Labels

The data directory contains the following label/metadata files:

train.csv (5,110 records): Hotspot detection data for which we have all corresponding imagery data (see below). Currently all of these hotspots refer to images in dataset ArcticSealsData01.
test.csv (1,314 records): Same format and distrbution of train.csv, suitable for cross-validation.

Each record in the CSV files refers to a hotspot that the NOAA thermal detection system picked up and that was classified by a human into either "Animal" (true positive) or "Anomaly" (false positive). Each hotspot is unique (no duplicates). The column schema is as follows:

hotspot_id: unique ID
timestamp: GMT/UTC timestamp (always corresponds to thermal image timestamp)
filt_thermal16: Filename of the 16-bit PNG containing the raw FLIR image data
filt_thermal8: Filename of the 8-bit JPG containing the annotated FLIR image data (hotspots circled)
filt_color: Filename of the 8-bit JPG containing a color image taken at or near the same time as the thermal image. The timestamp encoded in the filename may be different from the thermal timestamp by up to 60 seconds (but typically less than 1 second).
x_pos/y_pos: Location of the hotspot in the thermal image
thumb_*: Bounding box of the hotspot in the color image. NOTE: some of these values are negative, as the bounding box is always 512x512 even if the hotspot is at the edge of the image.
hotspot_type: "Animal" or "Anomaly"
species_id: "Bearded Seal", "Ringed Seal", "UNK Seal", "Polar Bear" or "NA" (for anomalies)

Raw Hotspot Data

In the data directory there is also a raw.csv (14,910 records) containing all hotspot detections from the NOAA 2016 survey flights (includes more seals but also more types of animals, more anomalies, hotspots marked as duplicates, etc.).

Code

The project is meant to accomodate many different approaches, frameworks, languages, etc. Linux is the primary supported dev environment, though some GUI tools are Windows-only.

Organization

Team members are welcome to add whatever code you like to this repo, but please follow these guidelines:

Put your source code in its own directory inside the src directory.
Add a README.md file to your code directory explaining what your code does and how it works.
If there are dependencies that need to be installed and/or build steps that need to be performed, add any necessary code to the build.bat script to run the relevant package manager commands, compile steps, etc., to ensure your code is fully runnable locally.
- Alternatively, it is also ok if your code only builds from within an IDE; if so just make a note of that in your README.md.
If applicable, add a script that runs your code to the root directory. If it takes command line arguments, please show help text if it is run without arguments.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

arcticseals's People

Contributors

Stargazers

Watchers

Forkers

cipop martinjm97 marcel-simon readicculus mzkaramat vdedyukhin bipulroy8 bhaskers-blu-org2 taffywrinkle claudiusgonzo daukantas smj111

arcticseals's Issues

This repo is missing important files

There are important files that Microsoft projects should all have that are not present in this repository. A pull request has been opened to add the missing file(s). When the pr is merged this issue will be closed automatically.

Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.

Merge this pull request

Validation (and API for using our tools)

It might be very helpful, if we had a way to provide our model in such a way, that it is very easy to use even with little to no technical knowledge. I was thinking, that we could provide it using an API and some GUI tool, which allows using this API. What do you think?

Docker image management

It would be great to have someone with Docker expertise help manage images containing all of the GPU training prerequisites. That would make it easy to spin up training on additional machines without having to tweak all the dependencies to get them just right; people can just pick up an image and use it as a starting point for further work.

See https://github.com/Microsoft/arcticseals/blob/master/src/SealDetectionRCNN/README.MD

This could be of interest:

https://docs.microsoft.com/en-us/azure/container-registry/container-registry-get-started-portal

IR normalization with current variable px distributions

Not a collaborator so could not re-open the closed normalization issue but I've been working on this on my own and have spent a lot of time on the IR imagery. The current issue, which I think its not possible to get around, is that there is no standard distribution of values nor can it be expected. This means that it's not too difficult to do percentile normalization on the given dataset but when it comes to images without hotspots the normalization causes warmer areas to look like hotspots.

There are a lot of pools of water that look very similar to seals in shape and size but are not. Without some sort of temperature to pixel correlation I've not found a way to do this so that it won't detect a ton of FPs on images without seals or other animals.

It sounds like in future flights the camera will include a temp correlation to at least the max pixel value but until then I'm thinking that the IR data is unfortunately relatively useless. IR imagery is suuuper important as I was getting an F1 score of around .93 on the labeled images and its WAYYYY faster to localize on them than the RGB imagery. Ideally we need to achieve about 3fps localization to be able to run this live on the plane which is seeming impossible with the large rgb images where it's taking me about 1-4 secs per image depending on the model and chip size.

Any feedback would be great as getting IR to work would be a massive step forward in this project.

Process raw.csv to append more records to training.csv

The new raw.csv file contains all hotspot detections, including ones we don't care about (duplicates) and ones we don't have imagery data for yet. It's also in a different schema than the training.csv file. This task is to process raw.csv to extract only the records that have at least thermal data in our datasets (01-99), convert those records to the training.csv schema, and append the records that are not already in training.csv (raw.csv is a superset of training.csv).

Make 4-channel RGBI (RGB-infared) images

Script almost done. Run code and check in blob with 4-channel images for the dataset. where we have thermal+color+detected hotspots.

Thermal imagery normalization

Our thermal data is in a raw 16-bit PNG format. We need to both a) figure out exactly what the raw values correspond to so that we can normalize the data (i.e. 0 degrees Celsius should be the same 16-bit pixel value in any image) for training and b) convert it an 8-bit representation for human visual inspection.

Thermal/color imagery registration

We will need the ability to register our thermal and color imagery pairs. Due to shifts in aircraft orientation, etc., image pairs taken at the same time will have some arbitrary offset relative to each other.

Question on IR normalization

Hello,
I'm working on a completely unrelated project where I also need to normalize IR images from different cameras. I want to ask you if you can explain how you derive the top and bottom parameters for each IR camera. I read that your goal is to adjust the histogram of all measured values for each camera to be approximately the same. I understand what you did in src/ir-normalization/gen_coeffs_playground.py, but then what to do next? @jomalsan

Thank you in advance!

Object detection approaches

Another way to contribute is training an actual object detection model, which does not require the hotspot detections. The input is an image, the output is a set of bounding box. I would consider this more difficult then the hotspot classification, but I would be happy to work with people on this task as well. First ideas for action items:

Cropping large patches (such as Full HD) out of all images and making a nice dataset with a reasonable number of seals in it.
Converting the annotations in a common object detection dataset format, such as VOTT or Coco
Training an off-the-shelf CNN object detector. If you have never worked in this area before, you could use the tensorflow detection API (https://github.com/tensorflow/models/tree/master/research/object_detection) as it is very complete and nicely written. However it is also quite a large piece of code. If anybody knows an easy to use framework, please mention it below.
On the long-term / more advanced: training of a cascade of detector, which allows for scanning huge amounts of images. The first detector is very fast, but has a high false positive rate. It is used to discard images which certainly do not contain seals. The second detector is slower and only scans images, which were not discarded by the last detector, and so on. This is similar to the Haar cascade for face detection.

Pretrained weights

In the README you elude to having pretrained models -

Copy the folder /ai4edevfs/models/seals-detection-ir-n-large-86.0 to the models directory of the SealDetectionAPI folder. This models folder will be copied to the docker image as /app/models. Now adjust the path to the model in detection_api/runserver.py to point to the correct path, e.g. /app/models/seals-detection-ir-n-large-86.0/fasterrcnn_07281142_0.8598245413189538.

But I cannot find those files anywhere on the repository. Are those files available? if not, is it possible to make them available?

Image previewer tool

It would be awesome to have a tool that makes it easier to examine the data we have. I'm thinking something like a simple GUI app that loads up a CSV and lets you quickly iterate through thermal/color pairs...

Hotspot classification

One low hanging fruit in our project is the hotspot classification, i.e. taking the hotspot detections provided by their algorithm and deciding which one actually corresponds to a seal and which not. Here are possible action items:

Crop color image and IR image patches at the annotated hotspot locations and create an easy to use dataset out of it. The cropping can be done as in the visualization tool of @Athapan Make sure the seals are not tightly cropped, i.e. there is still at least 30 pixels border around the seal.
Train an image classification model on color images only with any library you like on this dataset. This is a nice task for somebody, who has not much experience with machine learning but who would be interested in learning it. You can use any library and model you like, but a good starting point is pytorch and AlexNet / ResNet-50. It would be great to have an overview how different models perform. Feel free to ask for more details, if needed, in this issue thread.
Investigate the fusion of color and IR images for classification. Possible would be stacking images, i.e. making a 4 channel image out of color + IR (more promising but can be tricky to get this running), or separate models for each type of image (easier to get it running, but maybe worse accuracy).

Add filter to viewer

create website/tools to view navigate trainning/valiation

bounding rect around thumb nail data.