[SOLVED] Three potential problems: (A) 1.86 factor (B) initial position needed? (C) ADE/FDE calculation about human-path-prediction HOT 8 CLOSED

Hi,

That's a good point you make. The multimodal predictions made by "charlie" would typically be used downstream by a motion planner to decide what action to take. The planner can reason about multiple predictions before deciding what action is best -- it doesn't necessarily only need the "best" trajectory as input. Because pedestrians and vehicles have multimodal behavior, it is important to reason about many plausible scenarios and consider all reasonable cases before planning what to do. Thus, many prediction systems are multimodal and can generate multiple trajectories. To evaluate how good these methods are doing however, one metric we can use is to see how far off from the ground-truth trajectory is the best trajectory (out of the K trajectories we predicted). While K=20 was rather arbitrarily chosen following prior works, we can evaluate this with different values of K. As you mention, it is computationally expensive to generate and we need a tractable value of K as input to the motion planner, which is why we need a good prediction system that works well with reasonable values of K. The minADE and minFDE metrics are used to evaluate the prediction module, but in a real-world system all trajectories would be used (perhaps after ranking, etc.). Hope this answers your question!

from human-path-prediction.

Hi Zifeng,

Thanks for your interest, hope our code and paper are useful.

1. Please see: #7. We noticed scaling the data helped slightly with better optimization (note, same effect might be possible by tuning lambda1 & lambda2 instead).

2. Please see: #1 (comment). Scaled initial pos is used to allow the network to learn relative positions between pedestrians while matching element magnitudes.

3. When reporting final ADE/FDE numbers for the best of N setting, we need to have the ground truth trajectory as a reference to know which trajectory was the best. Some methods generate all 20 trajectories and then choose the best trajectory. Since our method is goal-based, we generate 20 goals, choose the best goal and only generate the trajectory for this best goal. The best goal is chosen based on the true ground truth destination: https://github.com/HarshayuGirase/Human-Path-Prediction/blob/master/PECNet/scripts/training_loop.py#L107. This is then used to predict the final trajectory given the best goal estimate: https://github.com/HarshayuGirase/Human-Path-Prediction/blob/master/PECNet/scripts/training_loop.py#L117. Note that we could also generate trajectories for all 20 goals and then we can calculate minADE/minFDE metrics. Either way, we will need to compare with the ground truth trajectory for error metrics; the way we have done it just saves some time since we don't need to generate all 20 trajectories. Hope this helps!

from human-path-prediction.

Hi Zifeng,

Thanks for your interest, hope our code and paper are useful.

1. Please see: Question - data scaling (applicable for ETH ?) #7. We noticed scaling the data helped slightly with better optimization (note, same effect might be possible by tuning lambda1 & lambda2 instead).
2. Please see: Model corresponding to Ours-S-TT #1 (comment). Scaled initial pos is used to allow the network to learn relative positions between pedestrians while matching element magnitudes.
3. When reporting final ADE/FDE numbers for the best of N setting, we need to have the ground truth trajectory as a reference to know which trajectory was the best. Some methods generate all 20 trajectories and then choose the best trajectory. Since our method is goal-based, we generate 20 goals, choose the best goal and only generate the trajectory for this best goal. The best goal is chosen based on the true ground truth destination: https://github.com/HarshayuGirase/Human-Path-Prediction/blob/master/PECNet/scripts/training_loop.py#L107. This is then used to predict the final trajectory given the best goal estimate: https://github.com/HarshayuGirase/Human-Path-Prediction/blob/master/PECNet/scripts/training_loop.py#L117. Note that we could also generate trajectories for all 20 goals and then we can calculate minADE/minFDE metrics. Either way, we will need to compare with the ground truth trajectory for error metrics; the way we have done it just saves some time since we don't need to generate all 20 trajectories. Hope this helps!

Hi，

Thank you for the detailed answer! It is really helpful.

I have no problem with the first two questions now. But for the last question, I would like to clarify my opinion further. Let's assume there is a robot Charlie that is running the code in the real world, and it is trying to predict the trajectory of another robot Jenny. Since Charlie doesn't know the goal or intention of Jenny, how should it determine which one from the generated 20 goals is the true destination unless there is communication between them (Jenny tells Charlie where she wants to go)? I don't think this is the original assumption of the trajectory prediction task. The problem here is not whether we generate 20 complete trajectories or 20 goals, it is the robot can not acquire the true positions of the others. Even though 20 complete trajectories are generated, the robot still doesn't know which one is better. We can train the model to acquire the ability to choose the best one somehow, but that is in the training phase. In the test phase, no real data should be offered to the model.

The code is trying to offer to model.predict() the true value of the destination in an indirect way. We can actually leave VAE alone and assume we have a random distribution that covers the whole area, if we sample infinite points, the FDE will go to 0 since there is always a sample that is exactly the true one, and the code is manually choosing it. This is the reason why when K goes to infinity the FDE drops so fast in the graph of the paper. FDE is eventually much much smaller than ADE when K is large (in my experiment, K = 10000, ADE = 7, FDE = 0.98). If we apply this method to all locations, then the ADE will be small too since all points have the true values now, but it is not possible in the real world. K only influences the computational complexity, meaning that if we have enough computing power, the ADE will go to zero. However, it is like a monkey typing randomly on a typewriter endlessly, and a part of the outcome is exactly a work of Shakespeare, it is just a matter of possibility, the model didn't do anything. If you admit that we shouldn't use the method for all points, why is the last one special?

from human-path-prediction.

Hello!

Your answer clarifies my confusion a lot, thank you! But there is one last quick point I want to make, hope you don't mind. Like you said the motion planner doesn't necessarily only need the "best" trajectory as input. If we are only evaluating the performance of the best one, is it persuasive enough to conclude the "Average Performance" of the prediction module is good enough since the motion planner is using all of them? Or is it persuasive enough to illustrate that the same motion planner will work better using your prediction module in the real world? I didn't read all code of the models using the Stanford drone dataset, is it a common metric to measure the performance as your code did? Do you know other works using the same metric? If you do could you please name them? It would be very helpful to me because I might do my research based on yours, thanks in advance!

from human-path-prediction.

This is definitely a good point -- the minADE and minFDE metrics may not capture everything that's important for a prediction module. For example, it may not always ensure that edge/rare scenarios are considered, etc. Incorporating planning metrics to evaluate prediction models could be an interesting direction (I believe https://arxiv.org/pdf/2110.03270.pdf explores this).

But as of now, many works use the minADE/minFDE metrics for evaluation as seen in Tables 1 and 2. Our other paper Y-Net may have some newer works that also use these metrics. Hope this helps!

from human-path-prediction.

This is definitely a good point -- the minADE and minFDE metrics may not capture everything that's important for a prediction module. For example, it may not always ensure that edge/rare scenarios are considered, etc. Incorporating planning metrics to evaluate prediction models could be an interesting direction (I believe https://arxiv.org/pdf/2110.03270.pdf explores this).

But as of now, many works use the minADE/minFDE metrics for evaluation as seen in Tables 1 and 2. Our other paper Y-Net may have some newer works that also use these metrics. Hope this helps!

Now I understand how things go in the code now, thank you! By the way, could you please offer the Standford drone dataset you used for PECNet? It seems that the stuff I download is in quite a different format, or could you please at least tell me where to download the dataset with the exact same format. Thanks again!

from human-path-prediction.

We've provided the pickled files (processed) and pre-processing scripts that we use for Stanford Drone Dataset:

https://github.com/HarshayuGirase/Human-Path-Prediction/blob/master/PECNet/utils/social_utils.py#L66

from human-path-prediction.

Hope this answers all your questions @pengpengjun.

For ease of search, please raise another issue for different question in future. Closing this since all your original queries seem to be solved now.

from human-path-prediction.