RandomCutMix, RandomMixUp augmentation about keras-cv HOT 17 CLOSED

Thanks @innat! I’ll take a look at this early next week.

from keras-cv.

We could port these from:
https://github.com/tensorflow/models/blob/master/official/vision/beta/ops/augment.py#L20

from keras-cv.

@LukeWood
The above implementation of mixup - cutmix should be the most optimized. Here is another implementation, a very basic starter of it. CutMix - MixUp.ipynb

rand_cutmix

rand_mixup

from keras-cv.

Began in PR #22

from keras-cv.

@bhack does label smoothing need to inherently be a part of MixUp & CutMix? (I'm not intimately familiar with the operations)

from keras-cv.

@LukeWood No but in the original papers they are using the label smoothing regularization as one of the experiment to improve the performance.

As you can see in the table Cutmix+Shakedrop has a better result in the original paper on CIFAR-100

We had Skaedrop in research at:
https://github.com/tensorflow/models/blob/master/research/autoaugment/shake_drop.py

Label smoothing is mainly from 2016:
https://arxiv.org/abs/1512.00567

As Cutmix (2019) was cited by other 940 papers now it is hard to understand what is a stable SOA.

Probably at API level we could try to design a pluggable composition for different regularizers if this is still a stable pattern in SOA.

from keras-cv.

Thanks for the info, I'll see if I can figure out exactly what we would want this layer to do.

Keras loss functions already smooth labels, so perhaps we don't need to accept labels as an argument regardless? https://www.tensorflow.org/api_docs/python/tf/keras/losses/BinaryCrossentropy

I'll do some further reading and look into everything the labels are used for internally.

from keras-cv.

Keras loss functions already smooth labels, so perhaps we don't need to accept labels as an argument regardless?

Do you need to apply the conditional at https://github.com/tensorflow/models/blob/master/official/vision/beta/ops/augment.py#L2219 ? I don't know if in the loss it is too late..

from keras-cv.

Ah Yeah, I think you are right.

One issue I am attempting to account for is composability. I.e.

preprocessing_model.add(RandomCrop())
preprocessing_model.add(RandomMixUp())

note randomCrop.call only receives images as an input, whereas right now RandomMixUp requires both images & labels. The standard is to only receive the x to a preprocessing layer, but if MixUp and CutMix require labels then maybe they’ll need to deviate from the standard a bit.

https://keras.io/guides/preprocessing_layers/

from keras-cv.

@mattdangerw are there any KPLS as of now that rely on label information?

from keras-cv.

I think the problem is that generally sample and label pair manipulation was more relegated to the tf.data API.
But here ware are going to manipulate sample a label pairs to create virtual and a sort of interclass samples.

from keras-cv.

E.g. check the dataset API example at:
https://keras.io/examples/vision/cutmix/

from keras-cv.

@bhack I went ahead and created PR #24 this morning. Since there's no way to avoid having to have two inputs & two outputs I went ahead and included label smoothing, with it disabled by default. I think this is a happy medium as it won't surprise users, but it's still an option. I'll create a RandomCutMix layer next.

from keras-cv.

Hey @bhack if you have any time a review here would be really helpful. Thanks, and no worries if you don't have the time!

from keras-cv.

@LukeWood no we don't have any preprocessing layers that depend on layer information. We do have a hashed crossing layer which takes multiple inputs as a tuple, but not really the same.

I am not sure if it would better to accept two separate inputs as you are doing in the PR currently, or to take in a single input that needs to be a (x, y) tuple.

Either way, doing this would restrict this layer to being something that needs to applied outside a model (e.g. a dataset.map), or to a subclassed model. I do not believe we have any way to access label data from layers inside a Functional/Sequential model.fit call. Though there has I believe been some discussion of it in the past (@fchollet might know more).

from keras-cv.

Perhaps a preprocessing model could be defined with an input for the labels as well here if the users want.

I am not sure if it would better to accept two separate inputs as you are doing in the PR currently, or to take in a single input that needs to be a (x, y) tuple.

thanks for the heads up, I updated it to take a tuple.

Though there has I believe been some discussion of it in the past (@fchollet might know more).

yeah there are some interesting regularization losses that actually require label information that would be great to be able to add from inside of Keras layers instead of just at the end of the model.

from keras-cv.

hey @LukeWood. Just to add two cents on this augmentation.

Currently, this aug can be applied to classification tasks but as the kerascv sets the target to general vision task, that's why to support vision tasks like object detection, we may need to add utility to accept the bbox_params argument too.

example of mixup-object-detection - region=full-images

example of mixup-object-detection - region=random

example of cutmix-object-detection

ref: https://www.kaggle.com/ankursingh12/data-augmentation-for-object-detection
ref: https://www.kaggle.com/shonenkov/oof-evaluation-mixup-efficientdet

from keras-cv.