in utils/config.py ,line 58:
commit = str(subprocess.check_output(["git", "rev-parse", "HEAD"]), 'utf-8').strip()
branch = str(subprocess.check_output(["git", "rev-parse", "--abbrev-ref", "HEAD"]).strip(), 'utf-8').strip()

error like this :
fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git [...] -- [...]'
Traceback (most recent call last):
File "test_crfnet.py", line 117, in
cfg = get_config(args.config)
File "../crfnet/utils/config.py", line 59, in get_config
commit = str(subprocess.check_output(["git", "rev-parse", "HEAD"]), 'utf-8').strip()
File "/home/ZT/anaconda3/envs/Keras-Tensorflow/lib/python3.7/subprocess.py", line 411, in check_output
**kwargs).stdout
File "/home/ZT/anaconda3/envs/Keras-Tensorflow/lib/python3.7/subprocess.py", line 512, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['git', 'rev-parse', 'HEAD']' returned non-zero exit status 128.

how to make my own dataset using images and radar points?

ERROR: Cannot install h5py==2.7.1, keras==2.2.4, numpy==1.16.1 and opencv-python==3.1.0.0 because these package versions have conflicting dependencies.

The conflict is caused by:
The user requested numpy==1.16.1
h5py 2.7.1 depends on numpy>=1.7
keras 2.2.4 depends on numpy>=1.9.1
opencv-python 3.1.0.0 depends on numpy==1.11.1

ImportError: ../crfnet/utils/compute_overlap.cpython-36m-x86_64-linux-gnu.so: undefined symbol: __intel_sse2_strchr

I have already ran
pip install -e .
python setup.py build_ext --inplace

Cuda 10
cudnn 7,4
Linux

Any helpful suggestions?

Thank you sincerely for the code!
Could you please provide more details on how to get the visualized results (akin to the Fig .4 in your paper) at your convenience?

Thank you for your good job!I trained on the nuscenes-mini dataset for 20 epochs, but the output of each epoch is only about 0.0001. Is there something wrong with my configuration? I only modified the dataset path on the default configuration.

Im getting this error, I have no idea what im doing wrong, Any help is much appreciated,
Thanks

setup_requires = ["cython>=0.28", "numpy>=1.16.0"]
before I use pip install -e .
I use conda install numpy 1.18.0 and cython.
it turns out numpy1.18.0 cannot work.
when I run python train_crfnet.py ....
ValueError: numpy.ufunc size changed, may indicate binary incompatibility. Expected 216 from C header,got 192 from PyObject

it says numpy version is too high

and if I uninstall and install numpy =1.16
it cannot work with python = 3.5
" numpy=1.16 -> python[version='>=2.7,<2.8.0a0|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0|>=3.8,<3.9.0a0'] "

could you please add numpy on the top of your requirements.txt ? not in the setup.py ?

maybe it should be changed to  this 

in line 38 in func def download_imagenet(self):
if self.backbone == 'vgg16':
backbone_name = 'vgg16'
resource = keras.applications.vgg16.vgg16.WEIGHTS_PATH_NO_TOP
checksum = '6d6bbae143d832006294945121d1f1fc'
elif 'vgg-max' in self.backbone:
backbone_name = 'vgg16'
resource = keras.applications.vgg16.vgg16.WEIGHTS_PATH_NO_TOP
checksum = '6d6bbae143d832006294945121d1f1fc'
elif self.backbone == 'vgg19':
backbone_name = 'vgg19'
resource = keras.applications.vgg19.vgg19.WEIGHTS_PATH_NO_TOP
checksum = '253f8cb515780f3b799900260a226db6'
else:
raise ValueError("Backbone '{}' not recognized.".format(self.backbone))
return get_file('{}_weights_tf_dim_ordering_tf_kernels_notop.h5'.format(backbone_name))

I got this issue when I tried to train model with distance regression on 8 blobs of nuScenes:

During training, the usage of memory continues to rise until the usage of swap reaches 100%, then the training stops, and no error is reported.

Please help me with this.Thanks!

config:
batchsize=1

Hi,in the CameraRadarFusionNet/crfnet/data_processing/fusion/fusion_projection_lines.py file

from ..tools_nuscenes.nuscenes_helper import get_sensor_sample_data

I haven't find where the repo has the block code of the tools_nuscenes,I need to help.
Thank you for such an excellent work!

I think this file is to detect object in real time, can you share it?

I'm trying to run the code with the distance detection flag enable, but I keep getting this error:
File "test_crfnet.py", line 189, in <module> boxes, scores, labels, dists = prediction_model.predict_on_batch(inputs)[:4] ValueError: not enough values to unpack (expected 4, got 3)

Does anybody used this option?

I have 8 gpus on my machine and they all work well when use nvidia-smi to list
but I got errors below

the keras issue is here
keras-team/keras#11644

it seems like it cannot be fixed unless I downgrade tensorflow to 1.12
but it should use CUDA9.0

I was completely confused why this work used a very high version of CUDA , but use a very low version tf1.13 ?
Do you have other options ?

training_model = multi_gpu_model(model, gpus=multi_gpu)
File "/root/anaconda3/envs/crfnet2/lib/python3.5/site-packages/keras/utils/multi_gpu_utils.py", line 181, in multi_gpu_model
available_devices))
ValueError: To call multi_gpu_model with gpus=8, we expect the following devices to be available: ['/cpu:0', '/gpu:0', '/gpu:1', '/gpu:2', '/gpu:3', '/gpu:4', '/gpu:5', '/gpu:6', '/gpu:7']. However this m
achine only has: ['/cpu:0', '/xla_gpu:0', '/xla_cpu:0', '/gpu:0']. Try reducing gpus.

Hi,

I tried different ways (using conda and docker) to install your module but all failed. Could you please double check the setup.py really works to get the module to run? Really appreciate it!

Hi Author,

I want to convert CRFNet model to some lighter format like onnx, tflite or tensorRT format. Is there any work around you have tried?

Please suggest or share any code.
Since I have tried many libraries like keras2onnx, tf2onnx but not working.

Thanks in Advanced.

While running the setup command "pip install -e .", I get the following error. it seems v1.13 is considered old and 2.2/2.3 is common now. Do you know if substituting the tensorflow-gpu version would work?

ERROR: Could not find a version that satisfies the requirement tensorflow-gpu==1.13.* (from crfnet==0.99.0) (from versions: 2.2.0rc1, 2.2.0rc2, 2.2.0rc3, 2.2.0rc4, 2.2.0, 2.3.0rc0, 2.3.0rc1, 2.3.0rc2, 2.3.0)
ERROR: No matching distribution found for tensorflow-gpu==1.13.* (from crfnet==0.99.0)

The version h5py==2.7.1 leads to an import error on Ubuntu 16.04 Python 3.7.

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/<username>/programs/anaconda3/envs/python3.7/lib/python3.7/site-packages/h5py/__init__.py", line 36, in <module>
    from ._conv import register_converters as _register_converters
  File "h5py/_conv.pyx", line 1, in init h5py._conv
  File "h5py/h5r.pyx", line 1, in init h5py.h5r
  File "h5py/_objects.pyx", line 1, in init h5py._objects
ImportError: /home/<username>/programs/anaconda3/envs/python3.7/lib/python3.7/site-packages/h5py/defs.cpython-37m-x86_64-linux-gnu.so: undefined symbol: H5Ovisit_by_name```


Could it be that Python 3.6. is required?

The latest version of Nuscenes-devkit doesnot support python 3.5.

When I run the test_crf.net on Google Collab using python 3.5 and CUDA 10.0 . The following message is displayed and I dont see anything else happening

Full error message: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:42] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.

Hi，I am learning about sensor fusion.
Can I test this project with a monocular camera and a millimeter wave radar ARS-408?
The sincerity anticipates your reply.Thank you!

Help is requested please.

Hi. I've tried to follow your tutorial.
Every setting I need is installed.

But when I do this command, 'pip install -e .', I got the following error in red color.
Collecting pyarrow
Using cached pyarrow-0.16.0-cp27-cp27mu-manylinux2010_x86_64.whl (20.5 MB)
ERROR: Could not find a version that satisfies the requirement hilbertcurve (from crfnet==0.99.0) (from versions: none)
ERROR: No matching distribution found for hilbertcurve (from crfnet==0.99.0)

what is wrong with it? please answer me.
Specification :
Ubuntu 18.04
RTX 2080Ti

Thanks.

Trying to run the code and first ran into an error when calling from_file_multisweep() in nuscenes_generator.py under data_processing/generator. It was being called with an unknown parameter "merge." When I removed it, it led to the following error-

Traceback (most recent call last):
File "train_crfnet.py", line 434, in
main()
File "train_crfnet.py", line 398, in main
File "/home/iot/anaconda3/lib/python3.6/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File "/home/iot/anaconda3/lib/python3.6/site-packages/keras/engine/training.py", line 1418, in fit_generator
initial_epoch=initial_epoch)
File "/home/iot/anaconda3/lib/python3.6/site-packages/keras/engine/training_generator.py", line 181, in fit_generator
generator_output = next(output_generator)
File "/home/iot/anaconda3/lib/python3.6/site-packages/keras/utils/data_utils.py", line 601, in get
six.reraise(*sys.exc_info())
File "/home/iot/anaconda3/lib/python3.6/site-packages/six.py", line 703, in reraise
raise value
File "/home/iot/anaconda3/lib/python3.6/site-packages/keras/utils/data_utils.py", line 595, in get
got pcs and times
<nuscenes.utils.data_classes.RadarPointCloud object at 0x7fc104a259b0>
inputs = self.queue.get(block=True).get()
File "/home/iot/anaconda3/lib/python3.6/multiprocessing/pool.py", line 644, in get
raise self._value
TypeError: 'RadarPointCloud' object is not iterable

Any help resolving this issue would be greatly appreciated!

Hi, @TUMFTM , thanks for making this pioneer work public.
But unfortunately the checkpoint provided is not working/opening. Can you please check once.

when I change the value of "network" in default.cfg file, I got the following question, how to solve it?
It seems like the code works well when the value of "network" is "vgg16" or "vgg19", but when I modify the value of "network" to "densnet121" or "resnet152", the following problem will occur, how to solve it?

Traceback (most recent call last):
File "train_crfnet.py", line 431, in
main()
File "train_crfnet.py", line 326, in main
model, training_model, prediction_model = create_models(
File "train_crfnet.py", line 145, in create_models
model = model_with_weights(backbone_retinanet(num_classes, num_anchors=num_anchors, modifier=modifier, inputs=inputs, distance=distance, cfg=cfg), weights=weights, skip_mismatch=True, config=copy.deepcopy(cfg), num_classes=num_classes)
File "train_crfnet.py", line 91, in model_with_weights
model_weights = model.get_layer(layer_name).get_weights()
File "/home/ZT/anaconda3/envs/crfnet-tf2/lib/python3.8/site-packages/keras/engine/network.py", line 358, in get_layer
raise ValueError('No such layer: ' + name)
ValueError: No such layer: block1_conv1

fragment of default.cfg:
[CRF-Net]
channels = [0,1,2,5,18]
image_height = 360
image_width = 640

dropout_radar = 0.0
dropout_image = 0.2

network = densenet121
network_width = 1.0
pooling = max

anchor_params = small

Thanks for making your project open-source.

The paper states that that you utilized small anchor sizes for the TUM dataset (Section V A. (b)) but it doesn't specify the anchor sizes for NuScenes. The default.cfg file sets the anchor sizes to small. Did you utilize AnchorParameters.default for anchor sizes for NuScenes? Also, what's the IoU threshold for the mAP results in Table 2 in the paper?

I am trying to replicate results of baseline image network but the hyperparameter configuration (e.g., preprocessing scaling, nms threshold, max detections) is unclear to me.

Hi,
I followed the commands as given in the git repository of TUMFTM /CameraRadarFusionNet.
sudo apt install graphviz
pip install -e . and
python setup.py build_ext --inplace

But when i try executing fusion_projection_lines.py it gives me ModuleNotFoundError which is No module named 'crfnet'
Although it says crfnet is installed. Attached screenshot for the same. Can you please let me know where am i going wrong.

Thanks,

Warning/Error in the "##### Evaluate Test Set #####" in train_crfnet.py

`Running network on 6 workers: N/A% (0 of 6830) | | Elapsed Time: 0:00:00 ETA: --:--:--2020-06-12 04:27:17.081662: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node filtered_detections_2/map/while/strided_slice_4. Error: Pack node (filtered_detections_2/map/while/stack) axis attribute is out of bounds: 1

2020-06-12 04:27:17.081708: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node filtered_detections_2/map/while/strided_slice_2. Error: Pack node (filtered_detections_2/map/while/stack) axis attribute is out of bounds: 1

2020-06-12 04:27:17.081966: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node clipped_boxes_2/strided_slice_2. Error: Pack node (boxes_2/stack) axis attribute is out of bounds: 2

2020-06-12 04:27:17.081986: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node clipped_boxes_2/strided_slice_3. Error: Pack node (boxes_2/stack) axis attribute is out of bounds: 2

2020-06-12 04:27:17.082000: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node clipped_boxes_2/strided_slice_4. Error: Pack node (boxes_2/stack) axis attribute is out of bounds: 2

2020-06-12 04:27:17.082014: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node clipped_boxes_2/strided_slice_5. Error: Pack node (boxes_2/stack) axis attribute is out of bounds: 2

2020-06-12 04:27:17.238923: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node filtered_detections_2/map/while/strided_slice_4. Error: Pack node (filtered_detections_2/map/while/stack) axis attribute is out of bounds: 1

2020-06-12 04:27:17.238966: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node filtered_detections_2/map/while/strided_slice_2. Error: Pack node (filtered_detections_2/map/while/stack) axis attribute is out of bounds: 1

2020-06-12 04:27:17.239223: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node clipped_boxes_2/strided_slice_2. Error: Pack node (boxes_2/stack) axis attribute is out of bounds: 2

2020-06-12 04:27:17.239244: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node clipped_boxes_2/strided_slice_3. Error: Pack node (boxes_2/stack) axis attribute is out of bounds: 2

2020-06-12 04:27:17.239260: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node clipped_boxes_2/strided_slice_4. Error: Pack node (boxes_2/stack) axis attribute is out of bounds: 2

2020-06-12 04:27:17.239275: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node clipped_boxes_2/strided_slice_5. Error: Pack node (boxes_2/stack) axis attribute is out of bounds: 2 `

Note: this warning/error does not stop the evaluation, it continues to evaluations (at Night and at Rain) and finishes the training successfully. But it would be good to know the issue related to the "##### Evaluate Test Set #####" in train_crfnet.py

Error while executing the "python train_crfnet.py --config configs/crf_net.cfg"
Python version: Python 3.6.9
Open-CV version: '4.1.2'

` File "../crfnet/data_processing/generator/nuscenes_generator.py", line 429, in load_image
img_p_full = self.image_plus_creation(self.nusc, image_data=image_sample, radar_data=radar_sample, **kwargs)
File "../crfnet/data_processing/fusion/fusion_projection_lines.py", line 480, in imageplus_creation
cur_img, camera_resize = _resize_image(image_data, image_target_shape)
File "../crfnet/data_processing/fusion/fusion_projection_lines.py", line 46, in _resize_image
resized_image = cv2.resize(image_data, stupid_confusing_cv2_size_because_width_and_height_are_in_wrong_order)

cv2.error: OpenCV(3.4.9) /io/opencv/modules/imgproc/src/resize.cpp:4045: error: (-215:Assertion failed) !ssize.empty() in function 'resize'`

I am not sure why it's showing as OpenCV(3.4.9) in the error statement, the OpenCV version is 4.1.2 inside the virtual environment

Hi. i am using nuscense -mini for initially . But after two epoch giving the error. thankyou

============================================================
##### Start Training #####

Epoch 1/20
Epoch 1/20
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/lib/python3.6/multiprocessing/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "/home/khaula/.local/lib/python3.6/site-packages/keras/utils/data_utils.py", line 401, in get_index
return _SHARED_SEQUENCES[uid][i]
File "../crfnet/data_processing/generator/generator.py", line 361, in getitem
inputs, targets = self.compute_input_output(group, inference=self.inference)
File "../crfnet/data_processing/generator/nuscenes_generator.py", line 588, in compute_input_output
inputs, targets = super(NuscenesGenerator, self).compute_input_output(group)
File "../crfnet/data_processing/generator/generator.py", line 319, in compute_input_output
image_group = self.load_image_group(group)
File "../crfnet/data_processing/generator/generator.py", line 190, in load_image_group
return [self.load_image(image_index) for image_index in group]
File "../crfnet/data_processing/generator/generator.py", line 190, in
return [self.load_image(image_index) for image_index in group]
File "../crfnet/data_processing/generator/nuscenes_generator.py", line 382, in load_image
sensor_channel, nsweeps=required_sweep_count, min_distance=0.0, merge=False)
TypeError: from_file_multisweep() got an unexpected keyword argument 'merge'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "train_crfnet.py", line 421, in
main()
File "train_crfnet.py", line 385, in main
class_weight=class_weights_labels
File "/home/khaula/.local/lib/python3.6/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File "/home/khaula/.local/lib/python3.6/site-packages/keras/engine/training.py", line 1418, in fit_generator
initial_epoch=initial_epoch)
File "/home/khaula/.local/lib/python3.6/site-packages/keras/engine/training_generator.py", line 181, in fit_generator
generator_output = next(output_generator)
File "/home/khaula/.local/lib/python3.6/site-packages/keras/utils/data_utils.py", line 601, in get
six.reraise(*sys.exc_info())
File "/home/khaula/.local/lib/python3.6/site-packages/six.py", line 703, in reraise
raise value
File "/home/khaula/.local/lib/python3.6/site-packages/keras/utils/data_utils.py", line 595, in get
inputs = self.queue.get(block=True).get()
File "/usr/lib/python3.6/multiprocessing/pool.py", line 644, in get
raise self._value
TypeError: from_file_multisweep() got an unexpected keyword argument 'merge'

When multi_gpu=0, the network can be trained smoothly,
but when I change multi_gpu=8, I get the error bellow:
It seems likely the cfg cannot be passed to backbone_retinanet.
it seems like in train_crfnet.py line 131,
backbone_retinanet(num_classes, num_anchors=num_anchors, modifier=modifier, inputs=inputs, distance=distance）
cfg=cfg is missed

File "train_crfnet.py", line 424, in
main()
File "train_crfnet.py", line 329, in main
distance_alpha = cfg.distance_alpha
File "train_crfnet.py", line 131, in create_models
model = model_with_weights(backbone_retinanet(num_classes, num_anchors=num_anchors, modifier=modifier, inputs=inputs, distance=distance), weights=weights, skip_mismatch=True, config=copy.deepcopy(cfg), nu
m_classes=num_classes)
File "../crfnet/model/architectures/vgg.py", line 34, in retinanet
return vgg_retinanet(*args, backbone=self.backbone, **kwargs)
File "../crfnet/model/architectures/vgg.py", line 101, in vgg_retinanet
vgg = vggmax.custom(input_tensor=inputs, include_top=False, weights=None, cfg=cfg)
File "/root/anaconda3/envs/crfnet2/lib/python3.5/site-packages/keras/applications/init.py", line 28, in wrapper
return base_fun(*args, **kwargs)
File "../crfnet/model/architectures/vggmax.py", line 19, in custom
return vggmax(*args, **kwargs)
File "../crfnet/model/architectures/vggmax.py", line 149, in vggmax
fusion_blocks = cfg.fusion_blocks
AttributeError: 'NoneType' object has no attribute 'fusion_blocks'
Exception ignored in: <bound method BaseSession.del of <tensorflow.python.client.session.Session object at 0x7f4762d369e8>>
Traceback (most recent call last):
File "/root/anaconda3/envs/crfnet2/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 738, in del

when i run the file fusion_projection_lines.py the error occured,and i cant find the function named raw_data_fusion in the project.
help please（-w-）

Error while executing the "python train_crfnet.py --config configs/crf_net.cfg"

File "../crfnet/model/layers/_misc.py", line 24, in <module> from crfnet.utils.anchor import generate_anchors File "../crfnet/utils/anchor.py", line 9, in <module> from .anchor_calc import compute_overlap File "../crfnet/utils/anchor_calc.py", line 13, in <module> **_from ..utils.compute_overlap import compute_overlap ModuleNotFoundError: No module named 'crfnet.utils.compute_overlap'_**

Same issue with "python test_crfnet.py --model saved_models/crf_net.h5 --config configs/crf_net.cfg --st 0.5"

I think this file is to detect object in real time, can you share it?

Please share a config file for NuScense mini version 1.0 and also inform the changes required for this version for other programs (if any)

File "evaluate_crfnet.py", line 242, in
eval_data = np.stack(([label_name]*len(recalls[label]), recalls[label], precisions[label]))
TypeError: object of type 'int' has no len()