update 3d segmentation demos about gcam HOT 5 OPEN

Hi,

there should be an updated nnUNet demo in this repo https://github.com/MECLabTUDA/M3d-Cam
It is an updated version of this repo (also from me), but targeted more towards medical CNN (It can be used for non-medical CNNs just the same). I probably will archive this repo or redirect to the new one in the future.

Let me know if the demo is working in the new repo (I updated it 2 weeks ago) or if the nnUNet changed again. If not I will try to update the demo again, but that will probably take a week or two as I am quite busy right now.

Best,
Karol

from gcam.

Hi,

there should be an updated nnUNet demo in this repo https://github.com/MECLabTUDA/M3d-Cam
It is an updated version of this repo (also from me), but targeted more towards medical CNN (It can be used for non-medical CNNs just the same). I probably will archive this repo or redirect to the new one in the future.

Let me know if the demo is working in the new repo (I updated it 2 weeks ago) or if the nnUNet changed again. If not I will try to update the demo again, but that will probably take a week or two as I am quite busy right now.

Best,
Karol

Hi, Karol,

I tried the new repo, the new demo is clearer than before but I still can't run with my dataset. After many times trails, I decided to write my code for feature visualization based on medcam.inject(), and then it works well.
I think a more straightforward demo is good enough for users to understand how does medcam works.

Here is debug info:
１. part of nnUNet model structure(I have five folds trained models):
...
(td): ModuleList()
(tu): ModuleList(
(0): ConvTranspose3d(320, 320, kernel_size=[2, 1, 2], stride=[2, 1, 2], bias=False)
(1): ConvTranspose3d(320, 256, kernel_size=[2, 2, 2], stride=[2, 2, 2], bias=False)
(2): ConvTranspose3d(256, 128, kernel_size=[2, 2, 2], stride=[2, 2, 2], bias=False)
(3): ConvTranspose3d(128, 64, kernel_size=[2, 2, 2], stride=[2, 2, 2], bias=False)
(4): ConvTranspose3d(64, 32, kernel_size=[2, 2, 2], stride=[2, 2, 2], bias=False)
)
(seg_outputs): ModuleList(
(0): Conv3d(320, 2, kernel_size=(1, 1, 1), stride=(1, 1, 1), bias=False)
(1): Conv3d(256, 2, kernel_size=(1, 1, 1), stride=(1, 1, 1), bias=False)
(2): Conv3d(128, 2, kernel_size=(1, 1, 1), stride=(1, 1, 1), bias=False)
(3): Conv3d(64, 2, kernel_size=(1, 1, 1), stride=(1, 1, 1), bias=False)
(4): Conv3d(32, 2, kernel_size=(1, 1, 1), stride=(1, 1, 1), bias=False)
)
)
2. Changes made in nnUNet source code:
trainer.network = medcam.inject(
trainer.network,
output_dir="/home/droplet/Data/nnunet-base/predict/Task104_Fullknee/gcam_results/feature_visualization",
label=1, replace=True, backend="gcam", layer='seg_outputs.4') # I only have one foreground class, but both label=1 and label=2 doesn't work for me. I think the layer of seg_outputs.5 is not defined in my case so I set layer="seg_outputs.4".
Other changes followed the demo.
3. Error log
when label=1:
Traceback (most recent call last):
File "/home/droplet/envs_py/nnunet_py36_env/bin/nnUNet_predict", line 8, in
sys.exit(main())
File "/home/droplet/envs_py/nnunet_py36_env/lib/python3.6/site-packages/nnunet/inference/predict_simple.py", line 221, in main
interp_order_z=interp_order_z)
File "/home/droplet/envs_py/nnunet_py36_env/lib/python3.6/site-packages/nnunet/inference/predict.py", line 645, in predict_from_folder
checkpoint_name=checkpoint_name)
File "/home/droplet/envs_py/nnunet_py36_env/lib/python3.6/site-packages/nnunet/inference/predict.py", line 237, in predict_cases
all_in_gpu=all_in_gpu)[None])
File "/home/droplet/envs_py/nnunet_py36_env/lib/python3.6/site-packages/nnunet/training/network_training/nnUNetTrainerV2.py", line 215, in predict_preprocessed_data_return_softmax
use_gaussian, all_in_gpu)
File "/home/droplet/envs_py/nnunet_py36_env/lib/python3.6/site-packages/nnunet/training/network_training/nnUNetTrainer.py", line 466, in predict_preprocessed_data_return_softmax
pad_kwargs=self.inference_pad_kwargs, all_in_gpu=all_in_gpu)[2]
File "/home/droplet/envs_py/nnunet_py36_env/lib/python3.6/site-packages/nnunet/network_architecture/neural_network.py", line 103, in predict_3D
return_softmax=return_softmax)
File "/home/droplet/envs_py/nnunet_py36_env/lib/python3.6/site-packages/nnunet/network_architecture/neural_network.py", line 290, in _internal_predict_3D_3Dconv_tiled
result[:, lb_x:ub_x, lb_y:ub_y, lb_z:ub_z] += predicted_patch
ValueError: non-broadcastable output operand with shape (1,128,80,224) doesn't match the broadcast shape (2,128,80,224)

when label=2:
Traceback (most recent call last):
File "/home/droplet/envs_py/nnunet_py36_env/bin/nnUNet_predict", line 8, in
sys.exit(main())
File "/home/droplet/envs_py/nnunet_py36_env/lib/python3.6/site-packages/nnunet/inference/predict_simple.py", line 221, in main
before: {'spacing': array([0.62499982, 0.62500012, 0.59999979]), 'spacing_transposed': array([0.59999979, 0.62499982, 0.62500012]), 'data.shape (data is transposed)': (1, 160, 75, 255)}
after: interp_order_z=interp_order_z)
File "/home/droplet/envs_py/nnunet_py36_env/lib/python3.6/site-packages/nnunet/inference/predict.py", line 645, in predict_from_folder
{'spacing': array([0.59999996, 0.58593756, 0.58593768]), 'data.shape (data is resampled)': (1, 160, 80, 272)}

checkpoint_name=checkpoint_name)

File "/home/droplet/envs_py/nnunet_py36_env/lib/python3.6/site-packages/nnunet/inference/predict.py", line 237, in predict_cases
all_in_gpu=all_in_gpu)[None])
File "/home/droplet/envs_py/nnunet_py36_env/lib/python3.6/site-packages/nnunet/training/network_training/nnUNetTrainerV2.py", line 215, in predict_preprocessed_data_return_softmax
use_gaussian, all_in_gpu)
File "/home/droplet/envs_py/nnunet_py36_env/lib/python3.6/site-packages/nnunet/training/network_training/nnUNetTrainer.py", line 466, in predict_preprocessed_data_return_softmax
pad_kwargs=self.inference_pad_kwargs, all_in_gpu=all_in_gpu)[2]
File "/home/droplet/envs_py/nnunet_py36_env/lib/python3.6/site-packages/nnunet/network_architecture/neural_network.py", line 103, in predict_3D
return_softmax=return_softmax)
File "/home/droplet/envs_py/nnunet_py36_env/lib/python3.6/site-packages/nnunet/network_architecture/neural_network.py", line 208, in _internal_predict_3D_3Dconv_tiled
nb_of_classes = self(a).size()[1]
File "/home/droplet/envs_py/nnunet_py36_env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "/home/droplet/envs_py/nnunet_py36_env/lib/python3.6/site-packages/medcam/medcam_inject.py", line 199, in forward
self.test_run(batch, internal=True)
File "/home/droplet/envs_py/nnunet_py36_env/lib/python3.6/site-packages/medcam/medcam_inject.py", line 229, in test_run
_ = self.medcam_dict['model_backend'].generate_attention_map(batch, None)
File "/home/droplet/envs_py/nnunet_py36_env/lib/python3.6/site-packages/medcam/backends/base.py", line 22, in generate_attention_map
self.backward(label=label)
File "/home/droplet/envs_py/nnunet_py36_env/lib/python3.6/site-packages/medcam/backends/base.py", line 41, in backward
self.mask = self._isolate_class(self.logits, label)
File "/home/droplet/envs_py/nnunet_py36_env/lib/python3.6/site-packages/medcam/backends/base.py", line 64, in _isolate_class
mask[:, label] = 1
IndexError: index 2 is out of bounds for dimension 1 with size 2
(1, 160, 80, 272)

By the way, in practice, the output auto-saved by medcam.indect() has wrong dimensions. For example, the input shape is (5,6,7), the output shape is (7,5,6).

Best,
Adot

from gcam.

Hi Adot,

Thanks for the update. I will have a look at it over the weekend. Is the dataset you used public by chance so I could replicate the error myself?

Best,
Karol

from gcam.

Hi Karol,

Sorry, the dataset I used is not public. The input data shape is (160, 256, x), x range from 65 to 89. The mask only has one class. The patch size is (128, 80, 224). In addition, the nnUNet seems to swap axes, the parameters look like: 'transpose_forward': [2, 0, 1], 'transpose_backward': [1, 2, 0].
I forgot to say that I installed nnUNet on April 12. So maybe the error caused by my out of date nnUNet.

Another question: If I want to visualize a middle layer in which output shape does not match to data shape, and I want to use ITK-Snap to observing the result, the interpolation or upsampling operation is needed in implementation?

The source code I implement in my case for reconstruction of entire grad-cam from patches. May be useful. : )

(Sorry for editing many times, I can't upload my code, so I attach a file.)
grad_cam_for_one_class.txt

Best,
Adot

from gcam.

Hi again,

In addition, the nnUNet seems to swap axes, the parameters look like: 'transpose_forward': [2, 0, 1], 'transpose_backward': [1, 2, 0].

The nnUNet transposes the dimensions as Pytorch expects the channel dimension to come first if it is a tensor.

I forgot to say that I installed nnUNet on April 12. So maybe the error caused by my out of date nnUNet.

Can you try to reinstall nnUNet to this specific version with the following commands:

!git clone https://github.com/MIC-DKFZ/nnUNet.git
%cd nnUNet
!git reset --hard b38c69b345b2f60cd0d053039669e8f988b0c0af # Reset repo to a specific commit as nnUNet code changes often. This ensures that the demo will work.
!pip install -e .

If medcam then still has problems with your dataset then the problem originates from medcam and not the nnUnet. This would be good to know. Please also reinstall medcam with pip uninstall medcam and then pip install medcam as I made an unrelated but crucial bugfix.

Another question: If I want to visualize a middle layer in which output shape does not match to data shape, and I want to use ITK-Snap to observing the result, the interpolation or upsampling operation is needed in implementation?

If I understand you correctly you mean if you need to interpolate manually afterwards? It should not be necessary if you set the data_shape parameter in medcam.inject to the output shape. Then medcam will interpolate the attention maps accordingly.

Best
Karol

from gcam.