airctic / icevision Goto Github PK

🚀 Feature

Make .predict method on MantisRCNN models return standard objects BBox, Mask...

This parser does not follow the structure of the other parsers and it's not very useful anyways

Following discussion on Slack and on issue #60
We come to a structure as follows

so we can have a structure something like this
-> backbones (use torchvision + custom)
-> layers (for some layers that you would use in the backbone)
-> models
-----> model_name_folder (e.g. fasterrcnn)
----------> model.py (take help of backbones here)
----------> dataloader.py (with minor edits for every mode)

model.py includes train_step, validation_step, test_step.

And train.py (code for training and inference from the model) Would be in the examples folder.

Inheriting from rcnn to faster rcnn is extra inter code coupling which we might avoid.
Let's have seperate structures for rcnn, fast rcnn and faster rcnn. It would make debugging easier as well.

Also, I will raise a PR for contributing.MD and FAQs.MD (will check how to make .rst)

Transform that resize the image changes the positions of bboxes and segs.

Currently COCOMetric will use the positions of the original records to calculate it's metrics. There are three possible solutions:

• Never scale validation images
• Don't use pycocotools, write metrics from scratch (good for longterm, pycocotools is causing a lot of minor issues)
• Apply transforms to the records passed to COCOMetric

If a transform removes an item from the image, the corresponded iscrowd also has to be removed

For instance, when we use COCOParser we should not need to pass a catmap

Absolute imports are easier to refactor (automatically refactor when moving packages) and offer no drawbacks

Is a high level Learner class a good idea?

This class would behave similarly to fastai, but would differ from the lightning workflow.

Maybe we can think of this like the high level API for training models, while lightning would be the mid level.

We need the workflow to be able to do something like this:

model.freeze_to(-1)
trainer.fit(...)
model.freeze_to(0)
trainer.fit(...)

Trainer already correctly resumes training, but we need to reset the lr_scheduler.

🚀 Feature

• It would be hard for end-user to understand where to implement custom architectures and backbones. They require same training code as there is a change in backbone which is feature extractor.

• Provide different backbone configurations that can be extended easily, something like a model file
  https://github.com/oke-aditya/pytorch_fasterrcnn/blob/dev/src/model.py

• Then use pytorch lightning trainer which is same for all these models. Something that simplifies https://github.com/oke-aditya/pytorch_fasterrcnn/blob/dev/src/engine.py

using torch lightning.

• Also it would make a standard API, define the model, using lightning as trainer, user need not edit and inherit lot of stuff for backbone changes, architecture changes, num_classes etc. He can simply edit the model.

What is FrozenBatchNorm?

Why is it used on models like FasterRCNN and MaskRCNN?

How does it impact fine tuning? Because it's often a good idea to never freeze any batch norm layer while training (even if the other layers are freezed)

Is your feature request related to a problem? Please describe.
Use models from pytorch hub

Describe the solution you'd like
Easily use models from hub, with minimal setup.

We can either always return a list, or wrap the returned object with a list if it's not already (might lead to future problems)

Currently show_pred is specify to RCNN models

I think it go out of visualize and inside RCNNModel, I'm open to new ideas

Figure out a better way to handle the vocab/id2cat

This can probably be done automatically by pytorch-lightning. More info here

Add lr schedule to at least one example, person.ipynb

📓 New example

What is the task?
Object detection

Is this example for a specific model?
FasterRCNN

Is this example for a specific dataset?
wheat

Don't remove
Main issue for examples: #39

Option to pickle records so we don't have to parse all data everytime

This option should be transparent to the user, we can expose it by a optional argument passed to DataParser.

Always a good discussion is where to store this data. Do it store it relative to the current file? Into /tmp? Or into a .mantisshrimp folder in the home directory?

Storing relative to the current file is always annoying when using version control, we have to explicitly not add it to checkout

Example

COCOParser(data, source, use_cached=True)

Calling fit multiple times should start where the last call ended.

Need to take care of steps in the trainer

The modification previously made on layers.ipynb were affecting the model performance on evaluation. It's needed to better understand what is happening in roi_heads.forward before modifying the method.

For now it's okay to feed the model twice for getting the loss and then the predictions.

Getting the validation loss by using model.train also disconsiders other important effects like Dropout and BatchNorm

Stop using pycocotools and implement metrics ourselves, this way them can be easily controlled and logged.

It's good that each model knows how to create it's own dataloader, but I don't like the fact that we need to instantiate the model to have access to the dataloader.

Previously we were using staticmethod, that got removed because we could not call super.

I think it's a good idea to bring staticmethod back, and instead of calling super we can just call the appropriate function

When should we start working in synth data?

References:
https://github.com/debidatta/syndata-generation

Also explain the concept of the mid/high level API for training

Currently we have an Item class that is handling all use cases, this introduces a lot of complexity because we have to keep checking for Nones.

Because each model is specific to a single task, we could instead use specific items for each task. Something like:

class MaskBBoxItem(Item):
    ...

Or even specific to each model like:

class FasterRCNNItem
    ...

If we go specific to each model, we can insert item2training_sample in the class

The error happens at mask_utils.decode from pycocotools.

Related issue can be found here

📒 Tutorials

Tutorials are in .ipynb format, explaining each step of the process, really detailed, not production like.

Core

• parser

Object detection

• wheat
• wheat with detr

Segmentation

• penfundan

Keypoints

📓 Examples

Examples are be in the .py format, more production oriented. Ready to be run with arguments from the command line and easy to integrate with wandb sweeps and alike.

Object detection

• wheat

Segmentation

Keypoints

Is there a new tutorial or example you would like to add? Comment below and we talk about it 😁

Once we agree, create an Tutorial or Example request issue (use the template) and I'll edit this post with your new cool example!

The main question is: Do we want to keep utils functions like L , lmap, ifnotnone and stuff like that?

While this functions are really helpful to who already is used to then, it elevates the barrier for new contributors.

Like fastai dl.show_batch

For fine-tuning, differential learning rates.

Would be good to have something like fastai freeze_to

Need to check if the DataParser is removing images without annotations.

🚀 Feature

Is your feature request related to a problem? Please describe.
Use model from pytorch hub

Describe the solution you'd like
Use models available on hub with minimal setup

It might be a good idea to embed the dataloader inside the model, because each dataloader is specific to a model anyways, this would also more closely follow the lightning guidelines

class MantisRCNN(MantisModule):
    @staticmethod
    def dataloader(self, <pytorch dataloader kwargs>):
        # Do the specialization
        return dataloader

It would also be logical to bring item2training_sample inside the model

class MantisRCNN(MantisModule):
    @staticmethod
    def item2training_sample(item):
        # convert item to training sample
        ...

Currently the images are only being divided by 255 on the call 'im2tensor', we need to divide by the pretrained mean

related to #1

The torchvision model will throw an error if images with no annotations are passed to it. I think we need to preemptively remove these images.

Note that a transform (like random crop or zoom) can remove the items from an image (potentially leaving it with no objects).

Related to this and this

🚀 Feature

Detr is an amazing new approach to object detection just launched by facebook. Pretained weights are available to hub, so naturally this issue is a bit related to #38.

Describe the solution you'd like
Let's divide this task into three separate parts:

• Model inference
• Train from scratch: Should be easier to implement and is supported in the original code
• Fine tuning: Not officially tested in the original code, will be a bit harder to implement.

Additional context
No other library supports this yet, let's goooo!! 🚀 🚀 🚀