This is the code implementation of the paper TSGAN: An Optical-to-SAR Dual Conditional GAN for Optical-based SAR Temporal Shifting by Moien Rangzan, Sara Attarchi, Richard Gloaguen, and Seyed Kazem Alavipanah. The paper is submitted to the ISPRS Journal of Photogrammetry and Remote Sensing.

Please feel free to contact me if you have any questions or problems with the code, and I'll be more than happy to help. You can reach me at [email protected]

Thank you for your interest in our work!

In contrast to the well-investigated field of SAR-to-Optical translation, this study explores the lesser-investigated domain of Optical-to-SAR translation, a challenging field due to the ill-posed nature of this translation. The complexity arises as a single optical data can have multiple SAR representations based on the SAR viewing geometry. We propose a novel approach, termed SAR Temporal Shifting, which inputs an optical data from the desired timestamp along with a SAR data from a different temporal point but with a consistent viewing geometry as the expected SAR data, both complemented with a change map of optical data during the intervening period. This model modifies the SAR data based on the changes observed in optical data to generate the SAR data for the desired timestamp. Our model, a dual conditional Generative Adversarial Network (GAN), named Temporal Shifting GAN (TSGAN), incorporates a siamese encoder in both the Generator and the Discriminator. To prevent the model from overfitting on the input SAR data, we employed a change weighted loss function. Our approach surpasses traditional translation methods by eliminating the GAN's fiction phenomenon, particularly in unchanged regions, resulting in higher SSIM and PSNR in these areas. Additionally, modifications to the Pix2Pix architecture and the inclusion of attention mechanisms have enhanced the model's performance on all regions of the data. This research paves the way for leveraging legacy optical datasets, the most abundant and longstanding source of Earth imagery data, extending their use to SAR domains and temporal analyses.

The architecture of the proposed model is shown below. For detailed information, please refer to the paper.

Generator	Discriminator

To download the dataset you can simply run this notebook in on your local machine or . The notebook will download the dataset and preprocess it.

A detailed description of the dataset and how to add your own data can be found in the dataset folder.

In the paper, we mention that in order to evaluate the model, we have used a hard evaluation method. In this method, around 150 images were selected from the test dataset, and these images showed discernable urban changes. These images are listed in this file.

If you went on and built your own dataset, you can use the same method to evaluate your model. Follow this notebook to create a new .csv file containing the images you want to evaluate your model on.

To train the model, run the train.py file.

> conda activate pytorch_env
> python train.py --num_epoch <number of epochs> --changed_l1_weight <weight of the changed L1 loss> 

Example:

> python train.py --num_epoch 10 --changed_l1_weight 5 --gen_version 1.3 --no_input_change_map 

Help:

> python train.py --help

You can find the results of the model in detail in the paper.

The models you saw in the paper are only a handful of the models that have been tested.

In the temporalgan folder, you can find the code for most of the models that have been tested. I tested multiple attention mechanisms at different levels of the model. CBAM and PAM are the two attention mechanisms that didn't make it to the paper.

Also, GLAM has been used in both 8x8 and 16x16 levels of the model.

Be careful in deleting unwanted models, as subversion of each model inherits from the previous model. For example, if you delete the v1.2 folder, you will corrupt the v1.3, v1.5, and v1.6 models.

So feel free to play around with the code and see what you can come up with.

Here is a beautiful sketch of my initial ideas for the model architecture :)

If you find this work useful, please consider citing:

@misc{rangzan2024tsgan,
      title={TSGAN: An Optical-to-SAR Dual Conditional GAN for Optical based SAR Temporal Shifting}, 
      author={Moien Rangzan and Sara Attarchi and Richard Gloaguen and Seyed Kazem Alavipanah},
      year={2024},
      eprint={2401.00440},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}