Evaluating your pretrained model about sparse-depth-completion HOT 17 CLOSED

Hi @oconnor127,

save_path should be the directory where the best model is located and not the model itself.
Or you can set the best_file_name variable directly.

Hope this helps.

from sparse-depth-completion.

Thanks for your quick response. I get the same error when typing the path of the model directory "mod"

source Test/test.sh /home/username/Desktop/bryan/Sparse-Depth-Completion/mod/ 5 /home/username/Desktop/bryan/data/ /home/username/Desktop/bryan/groundtruth_depth/

from sparse-depth-completion.

Can you try to set the model path manually here because it clearly fails at that line. Or set a print statement for best_file_name. This will tell you if the path to the model is at least correct.

from sparse-depth-completion.

Ok thanks, this works, but for some reason I the output-depthmap is not correct. It works with the following command:

$source Test/test.sh /home/username/Desktop/bryan/Sparse-Depth-Completion/mod/ 5 /home/username/Desktop/bryan/data/data_depth_selection/

were data_depth_selection is the raw-kitti-dataset containing depth_selection/val_selection_cropped/..
I've downloaded your model "model_best_epoch.pth.tar"

Is it correct to only give the path to the raw-dataset? The created depth map does have the correct names (sequences) and no error occurs. Is there a way to process (obtain dense depth maps) the validation_dataset from kitti in their original structure?

from sparse-depth-completion.

Normally you should get the exact numbers from the paper. What RMSE do you get?

Yes the path seems correct. Just verify the amount of files to make sure.

from sparse-depth-completion.

I just let your approach to create a small amount of depthmaps and abort after some iterations. Thus, I didnt had a look at RMSE or anything. The depthmap (screenshot) the first file (I think) is attached. The contours are visible, but there is still some main issue. (Dont worry about the artifact on the right side, thats just caused by the process of making the depthmap brighter to see it better)

from sparse-depth-completion.

Yes it should look better. Can you set a breakpoint at the start of the loop in test/test.py and check the filenames. Make sure that the RGB and LiDAR filenames match. Maybe let it finish to check what RMSE you get. If you don't get 802mm rmse, something is still wrong.

from sparse-depth-completion.

Hm, unfortunately the paths are correct. For the first iteration it gives:

/home/username/Desktop/bryan/data/data_depth_selection/depth_selection/val_selection_cropped/velodyne_raw/2011_09_26_drive_0002_sync_velodyne_raw_0000000005_image_02.png

/home/username/Desktop/bryan/data/data_depth_selection/depth_selection/val_selection_cropped/image/2011_09_26_drive_0002_sync_image_0000000005_image_02.png

Do you have any other idea?

from sparse-depth-completion.

Hi @oconnor127,

The paths look indeed correct.
I just ran test.sh in this code repository and I get the following
test_log.txt. At the end, you can see that you should get 802 mm RMSE.

Did you already run it completely? It's not easy for me to say how a specific image should look like.

from sparse-depth-completion.

After trying to run it till the end, I get the following error:
Lowest RMSE for selection validation set was 0.7908 in epoch 59
Initializing dataset kitti
0
0
0
0
0
0
1000
1000
===> Start testing
1000it [02:55, 5.69it/s]
average_time: 0.035244497482227514
num imgs: 1000
Test/devkit/cpp/evaluate_depth /home/username/Desktop/bryan/data/data_depth_selection/depth_selection/val_selection_cropped/groundtruth_depth Saved//home/username/Desktop/bryan/Sparse-Depth-Completion/mod//results
Starting depth evaluation..
Number of groundtruth (1000) and prediction files (-1) mismatch!
Segmentation fault (core dumped)

I've used the following command: source Test/test.sh /home/username/Desktop/bryan/Sparse-Depth-Completion/mod/ 1000 /home/username/Desktop/bryan/data/data_depth_selection/ /home/username/Desktop/bryan/data/data_depth_selection/depth_selection/val_selection_cropped/groundtruth_depth

I guess the errors is caused also by the 0,0,0,0,0,0,1000,1000 on the top. I think this might be a structure problem am I right? I just downloaded the dataset and gave the validation_path as argument...Is that correct? Do I need the entire dataset to only evaluate the data_depth_selection (val_selection_cropped)? I just want to evaluate the 1000 validation images.

from sparse-depth-completion.

Hi @oconnor127,

Please check your save directory. Clearly, Saved//home/username/Desktop/bryan/Sparse-Depth-Completion/mod//results is incorrect if you pass /home/username/Desktop/bryan/Sparse-Depth-Completion/mod to test.sh. Remove Saved at the bottom of the file test.sh. The error is quite obvious. It tells you it can not find the 1000 predictions. To solve this issue, the path should be /home/username/Desktop/bryan/Sparse-Depth-Completion/mod//result.

Please read the README. There is mentioned how you should use the pretrained model. However, changing the path should fix your issue.

Hope this helps.
Cheers.

from sparse-depth-completion.

But still this is just an evaluation issue. The main issue regarding the incorrect depth map has nothing to do with that. I got it to run and obtain the mae of 10.3843.

from sparse-depth-completion.

It is hard for me to say what else is incorrect. You should follow the README. Make a directory called Saved and put the directory with the model in there. Then you should only change 2 paths in Test/test.sh. (Please don't change other parameters like num_samples, it should be the default value). I downloaded the repository and the model this morning and I got the expected results (other people also confirmed the numbers).

from sparse-depth-completion.

Hmm weird. I tried it again from scratch and still the same mae (without any error msg).. And the model to download is : model_best_epoch.pth.tar correct?
Nevertheless, thank you for your help.

from sparse-depth-completion.

Hi @oconnor127,

Yes indeed. Are you sure you calculate the mae correctly?
Can you try downloading the dataset again. Maybe the dataset got corrupted. Also not sure if it has something to do with the fact that you're using windows.

To help you further, I will give the exact code and commands I run. You can download the code, model and results as a single zip-file from here. First unzip the files. Then change the path to the dataset in Test/test.sh. Finally, you only need to run sh Test/test.sh best. You can also find my predictions in the directory Saved/best/results/.

Hope this helps.

from sparse-depth-completion.

Hi @oconnor127,

Do you get the expected results or are there still some problems?

from sparse-depth-completion.

If something is still unclear, let me know. Closing this for now.

from sparse-depth-completion.