A semantic segmentation CNN for cloud detection

License: GNU General Public License v3.0

Python 100.00%

cloud-net-a-semantic-segmentation-cnn-for-cloud-detection's Introduction

Cloud-Net: A semantic segmentation CNN for cloud detection

Cloud-Net is an end-to-end cloud detection algorithm for Landsat 8 imagery. However, it can be used for other semantic segmentation applications, too. It gets a four-channel RGBNir image of Landsat 8 and predicts the location of clouds.

Cloud-Net has been introduced in the following IGARSS paper:

S. Mohajerani and P. Saeedi. "Cloud-Net: An End-to-end Cloud Detection Algorithm for Landsat 8 Imagery". (forthcoming) 2019, to appear at IEEE International Geoscience and Remote Sensing Symposium (IGARSS). URL: https://arxiv.org/pdf/1901.10077.pdf

Cloud-Net is a modification of CPAdv-Net, which is proposed in the following paper:

S. Mohajerani and P. Saeedi, "Shadow Detection in Single RGB Images Using a Context Preserver Convolutional Neural Network Trained by Multiple Adversarial Examples," in IEEE Transactions on Image Processing, vol. 28, no. 8, pp. 4117-4129, Aug. 2019. doi: 10.1109/TIP.2019.2904267, URL: https://ieeexplore.ieee.org/document/8664462

Training Cloud-Net on 38-Cloud Training Set

Requirements

The network has been tested with the following setup:
Windows 10, CentOS Linux release 7.5.1804
Python 3.6
Tensorflow 1.9.0, 1.10.0, 1.12.0
Keras 2.2.4
Scikit-image 0.15.0

Scripts

Run python main_train.py to train the network on 38-Cloud training set. The path to the dataset folder should be set at GLOBAL_PATH = 'path to 38-cloud dataset'. The directory tree for the dataset looks like as following:

├──38-Cloud dataset

│------------├──Cloud-Net_trained_on_38-Cloud_training_patches.h5

│------------├──Training

│------------------├──Train blue
. . .

│------------------├──training_patches_38-cloud.csv

│------------├──Test

│------------------├──Test blue
. . .

│------------------├──test_patches_38-cloud.csv

│------------├──Predictions

The training patches are resized to 192 * 192 before each iteration. Then, four corresponding spectral bands are stacked together to create a 192 * 192 * 4 array. A .log file is generated to keep track of the loss values. The loss function used for training is the soft Jaccard loss.

Testing Cloud-Net on 38-Cloud Test Set

Run python main_test.py for getting the predictions. The weights of Cloud-Net, pretrained on 38-Cloud training set, is available here: Cloud-Net_trained_on_38-Cloud_training_patches.h5. Relocate this file in the dataset directory as shown above. The predicted cloud masks will be generated in the "Predictions" folder. Then, use the Evaluation over 38-Cloud Dataset section to get the numerical results and precited cloud masks for the entire scenes.

cloud-net-a-semantic-segmentation-cnn-for-cloud-detection's People

Contributors

Stargazers

Watchers

cloud-net-a-semantic-segmentation-cnn-for-cloud-detection's Issues

Memory error

Using
TensorFlow backend.
100%|██████████|
9201/9201 [00:00<00:00, 211451.44it/s]
2019-08-18
11:18:02.451158: I tensorflow/core/platform/cpu_feature_guard.cc:141]
Your CPU supports instructions that this TensorFlow binary was not
compiled to use: AVX2 FMA
2019-08-18
11:18:02.769927: I
tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful
NUMA node read from SysFS had negative value (-1), but there must be
at least one NUMA node, so returning NUMA node zero
2019-08-18
11:18:02.770403: I
tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0
with properties:
name:GeForce GTX 1060 major: 6 minor: 1 memoryClockRate(GHz): 1.733
pciBusID:0000:01:00.0
totalMemory:
5.94GiB freeMemory: 5.64GiB
2019-08-18
11:18:02.770422: I
tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible
gpu devices: 0
2019-08-18
11:18:09.616736: I
tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device
interconnect StreamExecutor with strength 1 edge matrix:
2019-08-18
11:18:09.616806: I
tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0
2019-08-18
11:18:09.616827: I
tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N
2019-08-18
11:18:09.617285: I
tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created
TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with
5408 MB memory) -> physical GPU (device: 0, name: GeForce GTX
1060, pci bus id: 0000:01:00.0, compute capability: 6.1)
Experiment
name: Cloud-Net_trained_on_38-Cloud_training_patches
Predictionstarted...
Input image size = (384, 384)
Number of input spectral bands = 4
Batchsize = 1
/home/joe/.local/lib/python3.6/site-packages/skimage/transform/_warps.py:110:
UserWarning: Anti-aliasing will be enabled by default in skimage 0.15
to avoid aliasing artifacts when down-sampling images.
warn("Anti-aliasing
will be enabled by default in skimage 0.15 to "
Traceback
(most recent call last):
File
"/home/joe/PycharmProjects/Cloud_Net/Cloud-Net/main_test.py",
line 60, in
prediction()
File"/home/joe/PycharmProjects/Cloud_Net/Cloud-Net/main_test.py",
line 26, in prediction
steps=np.ceil(len(test_img)/ batch_sz))
File
"/usr/local/lib/python3.6/dist-packages/keras/legacy/interfaces.py",
line 91, in wrapper
return
func(*args, **kwargs)
File"/usr/local/lib/python3.6/dist-packages/keras/engine/training.py",
line 1522, in predict_generator verbose=verbose)
File"/usr/local/lib/python3.6/dist-packages/keras/engine/training_generator.py",
line 474, in predict_generator
return np.concatenate(all_outs[0])
MemoryError
Process finished with exit code 1
And my os is ubuntu18.04, CUDA 9.0,nvidia GTX 1060. Is my nvidia memory not enough?

data augmentation

Hello! It is a great job. What data augmentation methods do you use before training? I noticed that some of the clouds in the 38-cloud datasets are too thin to identify as this blew.

What augmentations could we use to make the clouds more clearly outlined?

LICENSE file

Would it be possible to add license information in the repository, so that it can be used in other projects?
Considering the metrics shown on paperswithcode and the fact that it comes with pretrained models,
I think it is a piece of code worth making available for other parties to build on.

Problems about retraining the model

Sorry to bother you @SorourMo. Since I only have datasets consist of RGB images now, I have to modify your source code to make it adapt to inputs of three channles by using only RGB channels of your cloud dataset for training. Further more, I resize the 384x384 images to 256x256 for more applicable to my own task.

Hovewer, I find two issues in the training procedure. Firstly, the code runs very slowly. A single epoch may consumes about 20~30 minutes in my NVIDIA RTX 3090 GPU, while the utilization of GPU is 0% for most of the time. Secondly, after 100 epoches of training, the predictions are all blank images without any semantic. Although the modifications above will make the model suboptimal, the new results are quite unbelievable as complete failure.

I will be very appreciate if you could give me any advice.

Model code optimization

Hello!
In the functions improve_ff_block is it correct to replace code

for ix in range(1):
    if ix == 0:
        x1 = input_tensor1
    x1 = concatenate([x1, input_tensor1], axis=3)
x1 = MaxPooling2D(pool_size=(2, 2))(x1)

x1 = MaxPooling2D(pool_size=(2, 2))(input_tensor1)
x1 = concatenate([x1, x1], axis=3)

and

for ix in range(3):
    if ix == 0:
        x2 = input_tensor2
    x2 = concatenate([x2, input_tensor2], axis=3)
 x2 = MaxPooling2D(pool_size=(4, 4))(x2)

x2 = x = MaxPooling2D(pool_size=(4, 4))(input_tensor2)
 for ix in range(3):
    x2 = concatenate([x2, x], axis=3)

I think it is possible to apply this approach for x3 and x4 calculation.

Issue in main_test.py

I wanted to run direct evaluation with main_test.py if thats possible, since I wanted to adapt the code to run on a custom dataset for later. But im getting the following error on main_test.py -->
Traceback (most recent call last):
File "/isro_data/Cloud-Net-A-semantic-segmentation-CNN-for-cloud-detection/Cloud-Net/main_test.py", line 64, in
prediction()
File "/isro_data/Cloud-Net-A-semantic-segmentation-CNN-for-cloud-detection/Cloud-Net/main_test.py", line 25, in prediction
imgs_mask_test = model.predict_generator(
AttributeError: 'Functional' object has no attribute 'predict_generator'. Did you mean: 'predict_on_batch'?
and if i change the function name as suggested (and then remove the argument keywords generator= and steps= since that gave error), I get error as - TypeError: TensorFlowTrainer.predict_on_batch() takes 2 positional arguments but 3 were given

Each value of output tif from main_test.py is all around 0.5(much bigger than 0.0471) and get all black or white tif by evaluation.m

@SorourMo
First stage(In this project):
1.1 input:
GLOBAL_PATH = 'H:/cloudAndNoncloud/38-Cloud'
TEST_FOLDER = os.path.join(GLOBAL_PATH, '38-Cloud_test')
test_patches_csv_name = 'test_patches_38-cloud.csv'

2 process:
processed by main_tet.py with Cloud-Net_trained_on_38-Cloud_training_patches.h5(417.7MB)
1.3 output:
TIF file（384x384pix， 32bits）in Predictions folder

Q1：(1)Why all values of this TIF file are around 0.5(not around 0.0471)

Second stage(In 38-cloud project, evaluation folder .m files)
2.1 input:
output TIF file from stage1, like patch_248_13_by_8_LC08_L1TP_003052_20160120_20170405_01_T1.TIF
2.2 process:
38-Cloud-A-Cloud-Segmentation-Dataset/tree/master/evaluation/evaluation.m file
refactoring about complete_mask and gt, change complete_pred_mast and gt（clip manually by 248_13 tile location） to 384x384
2.3 output:
All black or white TIF.

Q2:

(2)Is all black or white TIF result of stage2 caused by wrong output of stage1(384x384 tif, values are around 0.5)
(3)Why output of stage1 are all around 0.5 ? what step is wrong from my side? (I feel that all values should be around 0.471, so this predicted_patch = imbinarize(predicted_patch,thresh); could be make sense.
(4) If 0.0471 or 12/255 is settled, why not make evaluation.m file in main_test.py(in python) ?

sorourmo / cloud-net-a-semantic-segmentation-cnn-for-cloud-detection Goto Github PK