Giter Club home page Giter Club logo

depth-completion-gan's Introduction

Depth Completion GAN

This repo is the PyTorch implementation of our ACM Multimedia'21 paper on Sparse to Dense Depth Completion using a Generative Adversarial Network with Intelligent Sampling Strategies

photo not available

Requirements

The base environment(python 3.6) consists of:

pytorch == 1.3.1
torchvision == 0.4.2
tensorboard == 2.4.1
py-opencv == 3.4.2
numpy == 1.17.4
typing == 3.6.4

Dataset

Primarily two datasets were used ShapeNet and NYU_v2

Training

In order to run training:

python train.py --gpus=0,1,2,3 --batch_size=8 --n_epochs=10 --residual_blocks=17 --decay_epoch=5 -n train_test
  • -n --> give a name to the run
  • Modify the val dataloader path with appropriate data directory
  • Typically the directory has the following structure
    .
    ├── data.ShapeNetDepth
        ├── train
        |   ├── image_hr      #contains the ground truth
        |   ├── image_lr      #contains the saprse depth
        |   ├── meta_info.txt #contains the metadata of the directory
        ├── val
        |   ├── image_hr      #contains the ground truth
        |   ├── image_lr      #contains the saprse depth
        |   ├── meta_info.txt #contains the metadata of the directory
        ├── sample
        |   ├── image_hr      #contains the ground truth
        |   ├── image_lr      #contains the saprse depth
        |   ├── meta_info.txt #contains the metadata of the directory
  • The image_hr and image_lr are the folder containing dense and sparse depth respectively
  • The meta_info.txt contains the file names of these folders. Refer to misc/ folder for sample meta_info file
  • The folder "sample" contains a few sparse samples. This is to track the model learning visually.
  • The Dataset path needs to be hard coded within the script look at lines (170, 177, 183) in the train.py and line 264 in validate.py

Resume from checkpoint

In order to resume from a checkpoint, you can use the following command:

python train.py --gpus=0,1,2,3 --batch_size=8 --resume_epoch=1 --checkpoint_model_path=./logdir/train_test/saved_models --n_epochs=11 --residual_blocks=17  --decay_epoch=5 -n train_resume_test
  • The "--resume_epoch" and "--checkpoint_model_path" are required for checkpoint resuming
  • Typically the saved models are names as "generator_{}.pth" "discriminator_{}.pth" where {} contains the epoch to be resumed from

Validation

You can run standalone validation if you have a trained model. For that the checkpoint model path has to have 2 files named generator_best.pth and discriminator_best.pth. You can invoke the validation script by:

python validate.py --gpus=0,1,2,3 --batch_size=8 --residual_blocks=17 --checkpoint_model=./logdir/train_test/saved_models/ -n val_test

Misc

For convenience, some helping scripts have been provided in the misc\ folder

.
├── depthSamplingMP.py  #tutorial script for sparsifying dense depth
├── genMeta.py          #tutorial script for generating the meta data
├── meta_info.txt       #example meta_info file

Citation

If you found the repository helpful, please cite using the following:

@inproceedings{10.1145/3474085.3475688,
author = {Khan, Md Fahim Faysal and Troncoso Aldas, Nelson Daniel and Kumar, Abhishek and Advani, Siddharth and Narayanan, Vijaykrishnan},
title = {Sparse to Dense Depth Completion Using a Generative Adversarial Network with Intelligent Sampling Strategies},
year = {2021},
isbn = {9781450386517},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3474085.3475688},
doi = {10.1145/3474085.3475688},
booktitle = {Proceedings of the 29th ACM International Conference on Multimedia},
pages = {5528–5536},
numpages = {9},
keywords = {depth completion, gan, image compression, sensor sampling},
location = {Virtual Event, China},
series = {MM '21}
}

Acknowledgement

This work was supported in part by National Science Foundation (NSF) SOPHIA (CCF-1822923) and Center for Brain-inspired Computing (C-BRIC) & Center for Research in Intelligent Storage and Processing in Memory (CRISP), two of the six centers in JUMP, a Semiconductor Research Corporation (SRC) program sponsored by DARPA.

depth-completion-gan's People

Contributors

kocchop avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar

depth-completion-gan's Issues

share pre-trained model

hello. looking at the code
Now with this code, I'm trying to speed up kitti depth completion (prototyping). Could you share the pre-trained model?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.