wayveai / fiery Goto Github PK

Can you specify how to convert our Camera input to required format and get the predictions from model.

Hi,
I'm an undergraduate researcher working with a lab. We would like to use your model as a baseline, but there are a few modifications we need to make. The first is the input - the input to our model is a birds-eye view of a scene, while the input to fiery is a series of camera images. Looking at Figures 2 and 3 of your paper, it seems like it would be possible to cut in after the features are projected to birds-eye view. After working with the model, however, it seems difficult to substitute the vector x with a bird-eye view. Do you have any suggestions for how this may be done?

Hi,

Thank you for sharing this great work!

When I ran the training code, I got NaN for all losses after about 1 epoch of training.
This problem is reproduced whenever I run the training code. (I have tested it three times.)

I followed the same environment setting with anaconda, and also used the same hyper-parameters.
(The only difference is that our PyTorch version is 1.7.1 and yours is 1.7.0, and all other modules are the same as yours.)

Please share your idea about this problem, if you have any. Thanks!

Thanks for your great work. I have a little question about future ego pose during inference? I may find a little tricky because flow prediction is a module before motion planning. In real cases, the flow prediction module has no chance of getting future ego pose. But the code may show future ego pose is irreplacable in inference. When I turn to None, the inference doesn't work.

Hi. Thanks for your great work.
I followed your instructions in README.md to extract nuscenes dataset.
I ran evaluate.py with official pretrained checkpoint (https://github.com/wayveai/fiery/releases/download/v1.0/fiery.ckpt) but got the output as follows:
iou
53.5 & 28.6
pq
39.8 & 18.0
sq
69.4 & 66.3
rq
57.4 & 27.1
Is there something wrong? It seems to be much lower than the results you got.

Hello ,

Thank you for this great project firstly.
Well , is the “get_feature” doing the Orthographic Feature Transform?
Thank you very much

If I want to get the result of Fiery Static of Setting2 in Table I of your paper, should I use the config in "configs/single_timeframe.yml"? When I train the network using this config file from scratch, the IOU is 39.2 when I use the "evaluate.py". However, in the paper, the result is 35.8. Is there another parameter needed to be modified, when I want the network to be one frame as input and the segmentation result of the present frame as output?

Hi, I couldn't understand how the self.downsample parameter was set (why 8 and 16 and how it affects upsampling_in_channels) and why delete_unused_layers is required in the encoder model. I tried to search the efficientnet-pytorch implementation and couldn't find any reference for this operation. Could you explain briefly why this is required? Thank you!

Hello，

is it possible that deploy it in jetson agx Xavier?
Thank you very much.

Dear author,
I just run the code without no change, during the training ,I got the total sum loss with the value <0 .

It looks so weird.
Is that caused by the setting of the "uncertainty" ?
Is that normal?
Really thanks.

Will you release requirement.txt, and I want to know the pytorch-lightning version,thanks!

Hi, I saw that you are also one of the contributors of tom-roddick/mono-semantic-maps. And I got some trouble in generating BEV semantic labels of nuScenes.
The label I generated is like:

Is it right? And how to visualize it in color? Waiting for your answer and thanks a lot!

Thanks for your great work. I'd like to reproduce the training process, but I encountered an error. That is when I use multi-GPU distributed training process, the logging information seems normal, but afterwards the remote server stuck and connection reset and finally the process is killed. My remote server is an independent machine with 4xRTX3090. Is there any issues with the pytorch lightning distributed training that may cause my failure?

No module named 'pytorch_lightning.metrics'

Thanks for sharing the codes. The results are very impressive. I am a little confused about how instance segmentation works, and would you mind give me some suggestions like how to implement it. Many thanks.

First of all, congrats on the great work! Really liked fiery introduced the temporal component to improve the baseline of lift-splat-shoot.

Just curious, how does fiery got its name? FI is future instance I suppose, and I stopped from there :)

I dont see anywhere that the checkpoint is getting saved and while resuming the training, I am getting an error that "size mismatch for model.temporal_model.model.1.aggregation.0.conv.weight"

Hi guys,
I'd like to report a bug.

Otherwise, the center label may miss the last one.
before change

after change

Thanks for your work, I want to know how to export this torch model to onnx, I failed to do this.

Hi, thanks for your great work!
I am very interested in the future distribution visualization in figure 1, could you please tell me how to plot it?

Hi,
when we checked your code, FIERY uses zero vector as a noise in the evaluation. This way, the model uses only the mean vectors for generating samples. In order to generate samples, we changed the this line from:
sample = mu + sigma * noise to sample = mu + sigma * torch.randn_like(noise) to ensure the model uses a gaussian noise vector. However, the results were the same as the previous ones. We thought there might be a problem in the sigma part of the model and print the sigma values. When we examine the sigma values the values were very close to 1 (1 ± 0.05), which means no standard deviation for the distributions.

Is there any other way to generate samples? Can you clarify how you sampled different future predictions with your model?
Thanks in advance

Congratulations on your great work!

I want to follow your work for future research and I have some questions about your released code below:

In the fiery.py file of your code, can you provide more details about the get_geometry function and the projection_to_birds_eye_view function? I'm so confused about how they actually work, especially, the code shown in the red box below.

Thank you very much. Looking forward to your reply!

How can I get the release data for running visualise.py

why the lenth of self.indices is 10? Is this means only train 10 samples data?

Hi, I have a question about the batch size.
In your paper, you trained the model on 4 Tesla V100 with a batch size of 12.
But from you config file, batch size is set to 3 and it seems V100 doesn't have enough gpu memory for a batch size of 12.
So I assume the batch size is batch size per gpu.

In this case, if I use 2 GPUs instead of 4, should I also reduce the learning rate by half?
Thank you very much! Looking forward to your reply!

Best Regards,
Xinchao

AssertionError: Database version not found: /opt1/data/nuscenes/trainval/v1.0-trainval

Hi,

can you give a brief information about how to train and evaluate on the Lyft dataset? How to structure the dataset so that the dataloader can load the samples?

Thanks in advance.

First of all, I really appreciate about your great work.

I tried to run visualise.py

error1)cv2.error: OpenCV(4.7.0) 👎 error: (-5:Bad argument) in function 'line'

Overload resolution failed:

• Can't parse 'pt1'. Sequence item with index 0 has a wrong type

error2)AttributeError: 'FigureCanvasQTAgg' object has no attribute 'renderer'

solution1) Edit 'visualise.py' line37
->cv2.line(trajectory_img, (round(path[t][0]),round(path[t][1])) , (round(path[t+1][0]),round(path[t+1][1])), color, 4)

solution2) Edit 'matplotlib/backends/backend_agg.py' def tostring_rgb(self):

def tostring_rgb(self):
    self.renderer = self.get_renderer() # edit
    return self.renderer.tostring_rgb()

==================
After this two steps, i solved errors but outputs are empty.

Hello, thanks for your great project !. Well , I find that there are road marking(drivable area) on the bev map except the vehicle semantic mask. So I am wondering how could you achieve that ?
Thank you !

How do I use this model for my custom data? My model contains 3 frontal camera videos. It doesn't contain any map data.

Hi,
May I know how long would it take when I train a module on NuScenes
with "baseline.yml"(or other) as the configuration?

Hello and many thanks for your work and sharing your code.

I have a question regarding the way you compute your IoU metric and how it compares against Lift-splat.

You use stat_scores_multiple_classes from PLmetrics to compute the iou. Correct me if I am wrong, but by default the threshold of this method is 0.5

On the other hand, in get_batch_iou of LFS they use a threshold of 0: pred = (preds > 0)
https://github.com/nv-tlabs/lift-splat-shoot/blob/master/src/tools.py

Wouldn't this have an impact on the evaluation results ,and thus, on how you compare to them ?

Hello, recently I found your great work and I want to try the "Visualisation" part locally to check the results, but after I run the command of python visualise.py --checkpoint ${CHECKPOINT_PATH} my terminal pop out an error like the following:

I try to solve it by searching on google but it does not help, could you help me if you know how to solve it. Many thanks.

Hi,

Would you be able to explain how the panoptic_metrics function works?
(Code linked here: https://github.com/wayveai/fiery/blob/master/fiery/metrics.py#L137)
Especially, I wonder why 'void' is included for 'combine_mask', and why 'background' should be changed from 0 to 1.

Also, It is hard to understand the code under the comment "# hack for bincounting 2 arrays together".
(Code linked here: https://github.com/wayveai/fiery/blob/master/fiery/metrics.py#L168)

Thank you!

@anthonyhu Hi, it seems that y-axis should also be inverted, because the y axis of word coord is left.

Hi, thanks for sharing your great work. I saw the predicted segmentation is projected on the RGB images in Figure 1 in your paper. Some are dotted and some are transparent squares. What's the meaning of them? And how to visualize them?

Thank you for your great work and sharing the code!
I'm trying to train a model to overfit on the mini-set for the first step. But it cannot overfit on the mini-set even after 1000 epochs.

According to the code, you have two losses for each task, one is "factor*loss_fn" and the other is for "uncertainty". I'm confused about the part "uncertainty" and I think it may hinder the training.

For example, loss for instance_offset is about 0.2-0.6.

But loss for instance_uncertainty is negative and contribute more (about -1.4 at 14k) to the final loss. So the model may just blindly decrease "uncertainty".

I wonder why you design the loss this why. I'll appreciate it If you have any reference or document to help me understand this part.

Best Regards,
Xinchao

Thanks for your excellent work!
I have some questions about instance_flow.
warped_instance_seg = {} # t0,f01-->t1; t1,f12-->t2; t2,f23-->t3 # t1,f10-->t0; t2,f21-->t1 for t in range(1, seq_len): warped_inst_t = warp_features(instance_img[t].unsqueeze(0).unsqueeze(1).float(), # 1, 1, 200, 200 future_egomotion_inv[t - 1].unsqueeze(0), mode='nearest', spatial_extent=spatial_extent) warped_instance_seg[t] = warped_inst_t[0, 0]
In your paper, "Finally, we obtain feature flow labels by comparing the position of the instance centers of gravity between two consecutive timesteps".I think the code should convert t to t-1, not t-1 to t. How can it get the feature flow? I'm really confuesd about it. I'm looking forward your replying.

In static_pon_setting.yml, config is

_BASE_: 'lift_splat_setting.yml'

TAG: 'pyramid_occupancy_network_setting'

LIFT:
  X_BOUND: [-50.0, 50.0, 0.25]
  Y_BOUND: [-25.0, 25.0, 0.25]

But i think _BASE should be single_timeframe.yml， and FILTER_INVISIBLE_VEHICLES: False

when i predict over given sample data, we are not able to see any black vehicle in the output visualizations, can anyone explain this behaviour ?

Hi, thanks for your great work.
I have a question about loss. When i trained model for my data, the loss < 0 at epoch_0, is this normal?
Config: baseline.yaml in the project

Thanks for your great work.
I ran evaluate.py and got iou, pq, sq and rq. How can I get the VPQ reported in the paper.
Thanks in advance.

Hi,

We have trained your model with the baseline.yml on 4 V100 gpus but the results we got were slightly worse than the ones you reported on the paper. We had to load the weights from static_lift_splat_setting.ckpt because when we didn't, there was a NaN loss every time.

IOU (short | long) | VPQ (short | long)

58.8 | 35.8 | 50.5 | 29.0

59.4 | 36.7 | 50.2 | 29.9

Can you help us to understand why the results are different?
Thanks in advance.

Hi,
In the appendix of your paper, you reported the "Generalised Energy Distance" metric but the code does not have any implementation of that metric. Can you share the code you calculated "Generalised Energy Distance" metrics for your model?

Thanks in advance.

HI,
I train the model based on nuscenes with baseline.yml, but I got the nan loss after one epoch, how to train the model from scratch? Looking forward to your reply！

When I training nuscense dataset，it display issue ：There are a lot of detected instance centers: torch.Size([4377, 2])