getcrest / all-conv-keras Goto Github PK

View Code? Open in Web Editor NEW

118.0 13.0 41.0 15 KB

All Convolutional Network: (https://arxiv.org/abs/1412.6806#) implementation in Keras

License: MIT License

Python 100.00%

data-augmentation keras convolutional-neural-networks deeplearning

all-conv-keras's Introduction

Implementation of the All Convolution model in keras

Source

'Striving for Simplicity: The All Convolutional Net' The original paper can be found here.

We have also tried explaning the important concept of the paper on our Medium Blog.

Requirements

keras with Tensorflow backend (keras version 1.0.4 or later)
h5py (if you want to save your model)
numpy
pandas (if you want to save the logs of your model)
cv2 (for image resizing)

External data

In this implementation we are using the Cifar10 dataset. Either you can import the dataset from keras.datasets

You can Download the dataset from here.

Usage

If you want to run the model without using the pretrained weights then: Run python allconv.py

You can download the weights from here: weights.994-0.56.hdf5 And use the weights to retrain your model.

Results

The above model easily achieves more than 90% accuracy after the first 350 iterations. If you want to increase the accuracy then you can try much more heavy data augmentation at the cost of computation time.

Licensing

MIT License

Additional Notes

Use of Scheduler:

In the original paper learning rate of 'γ' and scheduler S = "e1 ,e2 , e3" were used in which γ is multiplied by a fixed multiplier of 0.1 after e1. e2 and e3 epochs respectively. (where e1 = 200, e2 = 250, e3 = 300) But in our implmentation we have went with a learning rate of 0.1, decay of 1e-6 and momentum of 0.9. This is done to make the model converge to a desirable accuracy in the first 100 epoch (Benificial for those who have a constrain on the computation power, feel free to play around with the learning rate and scheduler)

Data Augmentation:

In the original paper very extensive data augmentation were used such as placing the cifar10 images of size 32 × 32 into larger 126 × 126 pixel images and can hence be heavily scaled, rotated and color augmented. In our implementation we have only done very mild data augmentation. We hope that the accuracy will increase if you increase the data augmentation.

all-conv-keras's People

Contributors

Stargazers

Watchers

all-conv-keras's Issues

Strided convolutions missing activation?

The MaxPooling2D layers perform similar nonlinearity as ReLu so there is no need for activation function.
By replacing them with strided Conv2D we lose the nonlinearity effect and should add ReLu activation or the layer is basically useless (could be merged with next layer, because linear system).

Also the paper indicates that all Conv2D layers have ReLu activation.
@marcj maybe this gives your missing performance #4

Does not work out of the box

Thanks for the code. Without trying to look for reasons, it seems to fail for newer tensorflow/keras versions - though that could also be specific to my setup.

Anyway, I get:
tensorflow.python.framework.errors_impl.InvalidArgumentError: Dimension 0 in both shapes must be equal, but are 1 and 3. Shapes are [1] and [3].

Full trace:

Using TensorFlow backend.
C:\Users\Deeplearning.keras\datasets\cifar-10-batches-py
X_train shape: (50000, 32, 32, 3)
50000 train samples
10000 test samples
(32, 32, 3)
C:/Users/Deeplearning/Desktop/DeepRepo2/greendatamining/DeepLearn/DeepCodeOwnNetwork/simplenet.py:78: UserWarning: Update your Conv2D call to the Keras 2 API: Conv2D(96, (3, 3), input_shape=(3, 32, 32..., padding="same")
model.add(Convolution2D(96, 3, 3, border_mode='same', input_shape=(3, 32, 32)))
C:/Users/Deeplearning/Desktop/DeepRepo2/greendatamining/DeepLearn/DeepCodeOwnNetwork/simplenet.py:80: UserWarning: Update your Conv2D call to the Keras 2 API: Conv2D(96, (3, 3), padding="same")
model.add(Convolution2D(96, 3, 3, border_mode='same'))
C:/Users/Deeplearning/Desktop/DeepRepo2/greendatamining/DeepLearn/DeepCodeOwnNetwork/simplenet.py:82: UserWarning: Update your Conv2D call to the Keras 2 API: Conv2D(96, (3, 3), strides=(2, 2), padding="same")
model.add(Convolution2D(96, 3, 3, border_mode='same', subsample=(2, 2)))
C:/Users/Deeplearning/Desktop/DeepRepo2/greendatamining/DeepLearn/DeepCodeOwnNetwork/simplenet.py:85: UserWarning: Update your Conv2D call to the Keras 2 API: Conv2D(192, (3, 3), padding="same")
model.add(Convolution2D(192, 3, 3, border_mode='same'))
C:/Users/Deeplearning/Desktop/DeepRepo2/greendatamining/DeepLearn/DeepCodeOwnNetwork/simplenet.py:87: UserWarning: Update your Conv2D call to the Keras 2 API: Conv2D(192, (3, 3), padding="same")
model.add(Convolution2D(192, 3, 3, border_mode='same'))
C:/Users/Deeplearning/Desktop/DeepRepo2/greendatamining/DeepLearn/DeepCodeOwnNetwork/simplenet.py:89: UserWarning: Update your Conv2D call to the Keras 2 API: Conv2D(192, (3, 3), strides=(2, 2), padding="same")
model.add(Convolution2D(192, 3, 3, border_mode='same', subsample=(2, 2)))
C:/Users/Deeplearning/Desktop/DeepRepo2/greendatamining/DeepLearn/DeepCodeOwnNetwork/simplenet.py:92: UserWarning: Update your Conv2D call to the Keras 2 API: Conv2D(192, (3, 3), padding="same")
model.add(Convolution2D(192, 3, 3, border_mode='same'))
C:/Users/Deeplearning/Desktop/DeepRepo2/greendatamining/DeepLearn/DeepCodeOwnNetwork/simplenet.py:94: UserWarning: Update your Conv2D call to the Keras 2 API: Conv2D(192, (1, 1), padding="valid")
model.add(Convolution2D(192, 1, 1, border_mode='valid'))
C:/Users/Deeplearning/Desktop/DeepRepo2/greendatamining/DeepLearn/DeepCodeOwnNetwork/simplenet.py:96: UserWarning: Update your Conv2D call to the Keras 2 API: Conv2D(10, (1, 1), padding="valid")
model.add(Convolution2D(10, 1, 1, border_mode='valid'))
Traceback (most recent call last):
File "C:\Users\Deeplearning\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\framework\ops.py", line 1576, in _create_c_op
c_op = c_api.TF_FinishOperation(op_desc)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Dimension 0 in both shapes must be equal, but are 1 and 3. Shapes are [1] and [3].
From merging shape 0 with other shapes. for 'tower_0/lambda_1/concat/concat_dim' (op: 'Pack') with input shapes: [1], [3].

Code execution error?

hi vibrantabhi19 :

I think your article is great, so I try to execute the code you've provided. But I encountered some errors when I executed the code.

The last error message is as follows:
ValueError: Dimension 0 in both shapes must be equal, but are 1 and 3
From merging shape 0 with other shapes. for 'lambda_1/concat/concat_dim' (op: 'Pack') with input shapes: [1], [3].

So I would like to ask you may cause the wrong reason.You mentioned in the project environmental requirements are as follows:
Requirements:
keras with Tensorflow backend (keras version 1.0.4 or later)
h5py (if you want to save your model)
numpy
pandas (if you want to save the logs of your model)
cv2 (for image resizing)

And my operating environment is as follows:
Python 3.5.2 :: Anaconda 4.2.0 (64-bit)
TensorFlow Version:1.1.0
Keras Version:2.0.3
Using TensorFlow backend.

Is this the cause of the error?
Please help me deal with this error.
Thank you.

Accuracy is ~80 after 350 epochs

hi vibrantabhi19 :

Thank you for sharing your code! That's very helpful for me to understand All-CNN.

In addition, I've trained it last with your model night with 350 epochs, however found its accuracy (i.e. val_acc) became stable (about 0.81) after epoch 49 and remained the same to the end

Any ideas? :) 👍

The model I used:

`
model = Sequential()

model.add(Conv2D(96, (3, 3), padding="same", input_shape=(32, 32, 3)))
model.add(Activation('relu'))
model.add(Conv2D(96, (3, 3), padding="same"))
model.add(Activation('relu'))
model.add(Conv2D(96, (3, 3), padding="same", strides=2))
model.add(Dropout(0.5))

model.add(Conv2D(192, (3, 3), padding="same"))
model.add(Activation('relu'))
model.add(Conv2D(192, (3, 3), padding="same"))
model.add(Activation('relu'))
model.add(Conv2D(192, (3, 3), padding="same", strides=2))
model.add(Dropout(0.5))

model.add(Conv2D(192, (3, 3), padding="same"))
model.add(Activation('relu'))
model.add(Conv2D(192, (1, 1), padding="valid"))
model.add(Activation('relu'))
model.add(Conv2D(10, (1, 1), padding="valid"))

model.add(GlobalAveragePooling2D())
model.add(Activation('softmax'))

sgd = SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)
model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])`