Quantitative results on SemanticKITTI dataset about bkisemanticmapping HOT 6 CLOSED

Hi, @stephan000
Thank you for your interest in this project. Class 0 means unoccupied.
I wonder when you visualize the estimated semantic map, do you see any occupied areas?
If so, there might be something incorrect in the evaluation code. I will look into it and let you know.

from bkisemanticmapping.

Thank you for your prompt reply!

The estimated semantic map seems to be somewhat better qualitatively.

However, most estimated classes are unoccupied.
000000.txt

The occupied threshold is 0.7, which is the default value in config/method/semantic_bki.yaml.
So, Ι think this value is fine.

from bkisemanticmapping.

Hi @stephan000,

As the qualitative result looks fine to me. I don't think there's an issue with the code.
The quantitative result given in the paper is obtained for all the semanticKITTI sequences, so I am not sure if the result you got is correct or not.
Also, the 0.7 threshold is not used in this method, the unoccupied voxel is determined by the estimated Categorical distribution.

from bkisemanticmapping.

Hi @ganlumomo ,

To my understanding, the quantitative results listed in the README show the IoU values per scene.
Therefore, I expected that running semantickitti_04, which you provided, would give me the S-BKI results for sequence 04.
Is this incorrect?

It is enough for me to check if semantickitti_04 gives me the right result. However, the results I am getting now seem to be much poorer than the results in the paper.

My environment can be reproduced by docker. If there is a mistake in the procedure, please let me know.

FROM ros:kinetic

ENV DEBIAN_FRONTEND=noninteractive

RUN apt update -y -qq && \
  apt upgrade -y -qq --no-install-recommends && \
  apt install -y -qq \
    wget \
    curl \
    unzip \
    ros-kinetic-tf-conversions \
    ros-kinetic-vision-opencv \
    ros-kinetic-pcl-ros \
    ros-kinetic-rviz

ENV CATKIN_WS=/workspace/catkin_ws
WORKDIR ${CATKIN_WS}/src

RUN git clone https://github.com/ganlumomo/BKISemanticMapping && \
  cd ${CATKIN_WS} && \
  . /opt/ros/kinetic/setup.sh && \
  /opt/ros/kinetic/bin/catkin_make 

ARG FILE_ID=19Dv1jQqf-VGKS2qvbygFlUzQoSvu17E5
ARG FILE_NAME=semantickitti_04.zip

RUN cd ${CATKIN_WS}/src/BKISemanticMapping/data && \
  curl -sc /tmp/cookie "https://drive.google.com/uc?export=download&id=${FILE_ID}" > /dev/null && \
  CODE="$(awk '/_warning_/ {print $NF}' /tmp/cookie)" && \
  curl -Lb /tmp/cookie "https://drive.google.com/uc?export=download&confirm=${CODE}&id=${FILE_ID}" -o ${FILE_NAME} && \
  unzip ${FILE_NAME} && \
  rm ${FILE_NAME}

RUN set -x && \
  cd ${CATKIN_WS}/src/BKISemanticMapping/ && \
  sed -i -e "s/^scan_num: 100/scan_num: 271/g" config/datasets/semantickitti.yaml && \
  curl https://bootstrap.pypa.io/pip/3.5/get-pip.py -o get-pip.py && \
  python3 get-pip.py && \
  rm get-pip.py && \
  python3 -m pip install jupyter numpy scikit-learn && \
  jupyter nbconvert --to python semantickitti_evaluation.ipynb && \
  sed -i -e "s/^evaluation_folder.*/evaluation_folder\ =\ '.\/data\/semantickitti_04\/evaluations\/'/g" semantickitti_evaluation.py

Please run the following commands:

docker build -t bkisemanticmapping .
docker run -dit --name bki_sem_map bkisemanticmapping
docker exec -it bki_sem_map bash
source /workspace/catkin_ws/devel/setup.bash
roslaunch semantic_bki semantickitti_node.launch
cd /workspace/catkin_ws/src/BKISemanticMapping
python3 semantickitti_evaluation.py

from bkisemanticmapping.

Hi @stephan000
I just fixed the bug in the commit: 1e18b1e. Apparently, I added a fixed transformation to the map for a better visualization view, but somehow forgot to add the same transformation in the evaluation code.
Sorry for the inconvenience. Also, the results in the paper are obtained using resolution of 0.1. So you might need to change the resolution here: BKISemanticMapping/config/datasets/semantickitti.yaml.

from bkisemanticmapping.

Hi @ganlumomo

I got almost the same results as in the paper with the default resolution.
Thanks for your quick response!

from bkisemanticmapping.