prclibo / kitti_eval Goto Github PK

The output :
Thank you for participating in our evaluation!
Loading detections...
number of files for evaluation: 1

I would expect similar values for both cases, but I do not get similar values. Is there any difference between these two?

How can I solve this problem?
Thank you for participating in our evaluation!
Loading detections...
number of files for evaluation: 3769
ERROR: Couldn't read: 003323.txt of ground truth. Please write me an email!
An error occured while processing your results.

First, thanks for your work. As a beginner, I don't know what is the output represent.
There are four txt for each class and related '.gp', I don't know the number in each row.
Could you give me some explanation or give me some Tutorials? Thanks.

First, thanks for your amazing work!
While it's really helpful, I really need your help.
Here are the summary I'm currently seeing.

car_detection_AP : 85.031853 81.267220 74.636650
sh: 1: gnuplot: not found
sh: 1: gnuplot: not found
Error: /undefinedfilename in (car_detection_AP.eps)
Operand stack:

Execution stack:
%interp_exit .runexec2 --nostringval-- --nostringval-- --nostringval-- 2 %stopped_push --nostringval-- --nostringval-- --nostringval-- false 1 %stopped_push
Dictionary stack:
--dict:961/1684(ro)(G)-- --dict:0/20(G)-- --dict:78/200(L)--
Current allocation mode is local
Last OS error: No such file or directory
GPL Ghostscript 9.26: Unrecoverable error, exit code 1
sh: 1: pdfcrop: not found
car_orientation_AOS : 84.987823 79.064232 69.322250
sh: 1: gnuplot: not found
sh: 1: gnuplot: not found
Error: /undefinedfilename in (car_orientation_AOS.eps)
Operand stack:

Execution stack:
...
...
...
Execution stack:
%interp_exit .runexec2 --nostringval-- --nostringval-- --nostringval-- 2 %stopped_push --nostringval-- --nostringval-- --nostringval-- false 1 %stopped_push
Dictionary stack:
--dict:961/1684(ro)(G)-- --dict:0/20(G)-- --dict:78/200(L)--
Current allocation mode is local
Last OS error: No such file or directory
GPL Ghostscript 9.26: Unrecoverable error, exit code 1
sh: 1: pdfcrop: not found

When it starts evaluating bird's eye detection, a critical segmentation fault occurs, and it's all stopped.
It has been happening in other similar projects, but I can't figure out what the real issues are.
It will be so helpful if you could give me some piece of hints. Thanks in advance.

Apart from this issue, I have one more question.
How can we decide the 16th value 'score/confidence' in the label? This time, I just put any random number, but I have no idea of calculating the value. Does it come from 2D Detector which usually outputs confidence probability by softmax classification?

Again, thanks for you work, and looking forward to seeing your reply!

Thank you for participating in our evaluation!
mkdir: cannot create directory ‘../../eval_kitti/results/exp1/data12/plot’: File exists
Loading detections...
number of files for evaluation: 0

why 0?

Do the detections get filtered based on the size of 2D box or dimensions of the detected object?

The plot folder is generated, but it is empty，and number of files for evaluation: 0

why???

Hi, thanks for your hard work on implementing the evaluation code. I am looking into the code and have a hard time connecting it to the AP formula in the paper. Most confusion comes from eval_class, which calculates the AP of a single class. Since it's hard for me to organize my questions into one paragraph, I list all of them in the below:

1. Why use detection scores as recall? More specifically, why you use scores as threshold when computing statistics? As far as I know, there's no threshold when computing average precision, which only involves calculating precisions for discretized recalls.

2. Following the previous question, the discretized recalls are 0, 0.1,....,1 as described in the paper. Where do you use this information in eval_class

3.What does N_SAMPLE_PTS mean, and how do you come up with 41?

4. Why we need getThresholds and how does it work?

To sum up, I don't quite understand how it connects to the AP formula, and would appreciate if you can help me out!!!

Hi all,

just wanted to leave a note that thanks to the code provided by @prclibo, we released an official 3D object detection benchmark. Check it out at: http://www.cvlibs.net/datasets/kitti/eval_3dobject.php

Hi, I would expect the 2D AP to be somewhat close to the value I get when I evaluate it with COCO evaluation script.
Kitti 2D (iou=0.5) AP: 35.7
COCO AP(iou=0.5) : 56.5

Any idea why there is a disparity?

I have my result .txt files in my result_dir/, and my validation .txt files in my 'ground_truth_dir/', but every time I will just get the results like this,

Thank you for participating in our evaluation!
Loading detections...
number of files for evaluation: 0
  done.
Your evaluation results are available at:
../Stereo-RCNN/models_stereo/result

I put the training data's label_2 under the folder, but it can't works

// this ground truth is not of the current or a neighboring class and therefore ignored if(ignored_gt[i]==-1) continue;
The case of neighboring class is not included when ignored_gt==-1.
May be some problems happen here

car_detection_AP : 35.000000 79.857147 84.663734
sh: 1: gnuplot: not found
sh: 1: gnuplot: not found
Error: /undefinedfilename in (car_detection_AP.eps)
Operand stack:

Execution stack:
%interp_exit .runexec2 --nostringval-- --nostringval-- --nostringval-- 2 %stopped_push --nostringval-- --nostringval-- --nostringval-- false 1 %stopped_push
Dictionary stack:
--dict:960/1684(ro)(G)-- --dict:0/20(G)-- --dict:77/200(L)--
Current allocation mode is local
Last OS error: No such file or directory
GPL Ghostscript 9.26: Unrecoverable error, exit code 1
sh: 1: pdfcrop: not found
car_orientation_AOS : 18.006918 38.727802 40.937248
sh: 1: gnuplot: not found
sh: 1: gnuplot: not found
Error: /undefinedfilename in (car_orientation_AOS.eps)
Operand stack:

Execution stack:
%interp_exit .runexec2 --nostringval-- --nostringval-- --nostringval-- 2 %stopped_push --nostringval-- --nostringval-- --nostringval-- false 1 %stopped_push
Dictionary stack:
--dict:960/1684(ro)(G)-- --dict:0/20(G)-- --dict:77/200(L)--
Current allocation mode is local
Last OS error: No such file or directory
GPL Ghostscript 9.26: Unrecoverable error, exit code 1
sh: 1: pdfcrop: not found
car_detection_BEV_AP : 35.000000 82.357140 87.229523
sh: 1: gnuplot: not found
sh: 1: gnuplot: not found
Error: /undefinedfilename in (car_detection_BEV_AP.eps)
Operand stack:

I am looking forward to your reply!

When I give the ground truths with some score directly as results but always get 90.90 as the APs. Does anyone achieve AP of 100?

car_detection AP: 0.417685 0.280079 0.230861
PDFCROP 1.38, 2012/11/02 - Copyright (c) 2002-2012 by Heiko Oberdiek.
==> 1 page written on `car_detection.pdf'.
save preds_val/plot/car_orientation.txt
car_orientation AP: 0.195098 0.127194 0.103741
PDFCROP 1.38, 2012/11/02 - Copyright (c) 2002-2012 by Heiko Oberdiek.
==> 1 page written on `car_orientation.pdf'.
Segmentation fault

in this file, evaluate_object_3d_offline.cpp.

Thanks.

Hello,

I see the following:

car_detection_ground AP: 63.954884 73.779037 74.295746

As I understand, the scores are ordered as easy, moderate, hard.
If this this correct then is there something wrong with above results?
How are hard samples having better score than easy ones?

Please help.

Best Regards
Sambit

I'm using the KITTI cpp code (evaluate_object_3d_offline.cpp) to evaluate my birds eye view bounding box detections. The code filters the detections into easy, moderate and hard by the height of the detected object's bounding box in the image space (i.e. pixels). I do not have image detections at all and so all my detections are being ignored (lines 447-452). I'm experimenting solely on BEV data. Any idea how to evaluate ONLY BEV detections (without any corresponding image detections). Is there a reference somewhere for this?

Running the following script,
`
**import numpy as np
from det3d.datasets.utils.eval import calculate_iou_partly

fake_gt_annos = []
anno = {}
anno.update(
{
"name": np.array(['Car']),
"location": np.array([[0.0, 0.0, 0.0]]),
"dimensions": np.array([[1.0, 1.0, 1.0]]),
"rotation_y": np.array([np.pi/8]),
}
)
fake_gt_annos.append(anno)

rets = calculate_iou_partly(
fake_gt_annos, fake_gt_annos, metric=1, num_parts=1, z_axis=2, z_center=0.5
)
print(rets)**
I got:
([array([[7.17528081e-09]])],
[array([[7.17528081e-09]])],
array([1]),
array([1]))
`
Why? I use the same parameters of dt_annos and gt_annos when calling calculate_iou_partly, and I expect to get the overlap result = 1.

Is this a bug? Or I misunderstand something?