Comments (19)
Loftr with the new testing setting as found by @georg-bn :
>>Finished, pairs=540 match_failed=0 matches=2599.4 match_time=0.21s
==== Image Matching ====
#Features: mean=2599 min=94 max=5759
#(Old)Matches: a=2599, i=3204, v=2038
#Matches: a=2599, i=3204, v=2038
MMA@[ 1 3 5 10] px:
a=[0.63 0.91 0.93 0.95]
i=[0.68 0.95 0.96 0.98]
v=[0.59 0.86 0.91 0.92]
==== Homography Estimation ====
Hest solver=cv est_failed=0 ransac_thres=3 inlier_rate=0.86
Hest Correct: a=[0.64 0.87 0.92 0.95]
i=[0.81 0.98 0.99 1. ]
v=[0.49 0.77 0.85 0.91]
Hest AUC: a=[0.36 0.65 0.75 0.84]
i=[0.48 0.79 0.87 0.93]
v=[0.25 0.52 0.64 0.76]
Comments: Not sure whether other methods also got improved in such a setting. And I am also not sure that it is an equal comparison to other methods, since in this setting both the ground truth and data have been changed.
But still!
Thank you all for the discussion and tryings to reproduce the published numbers.
from image-matching-toolbox.
I can't check at the moment, but it sounds like it might be due to Ransac. Perhaps if you increase the number of iterations it becomes more constant?
from image-matching-toolbox.
Hi Truong, this is the uncertainty we got when running HPatches Homography estimation. I would say +-0.5 is reasonable.
from image-matching-toolbox.
Actually I'm now unsure what they mean with AUC... But the OpenCV comment still holds.
Related issue:
zju3dv/LoFTR#136
from image-matching-toolbox.
Hi @Parskatt ,
Thanks for pointing this out. I have updated the HPatches evaluation code to support:
- Report both the cornor error in percentages at specific thresholds and the AUC of the corner errors at those thresholds
- Optional switch between the pydegensac solver and opencv solver, degensac is the default one since it normally leads to better performance for almost all methods in my experience.
The new code example:
python -m immatch.eval_hpatches --gpu 0 \
--config 'superpoint' 'superglue' 'caps_superpoint' \
--task 'both' --h_solver 'degensac' --ransac_thres 2 \
--root_dir . --odir 'outputs/hpatches'
The example output for SuperGlue with degensac is:
>>>> Method=SuperGlue Default config: {'class': 'SuperGlue', 'weights': 'outdoor', 'sinkhorn_iterations': 20, 'nms_radius': 4, 'match_threshold': 0.2, 'max_keypoints': 1024, 'imsize': -1} Thres: [0.2]
Matching thres: 0.2 Save to: None
>>>>Eval hpatches: task=matching+homography method=SuperGlue_r4 rthres=2.0 thres=[1, 3, 5, 10]
==== Image Matching ====
#Features: mean=921 min=212 max=1024
#(Old)Matches: a=491, i=522, v=462
#Matches: a=491, i=522, v=462
MMA@[ 1 3 5 10] px:
a=[0.44 0.85 0.94 0.98]
i=[0.5 0.86 0.96 0.99]
v=[0.39 0.85 0.93 0.97]
==== Homography Estimation ====
Hest solver=degensac est_failed=2 ransac_thres=2.0 inlier_rate=0.86
Hest Correct: a=[0.5 0.82 0.9 0.96]
i=[0.58 0.93 0.99 1. ]
v=[0.43 0.72 0.82 0.92]
Hest AUC: a=[0.24 0.56 0.68 0.81]
i=[0.28 0.64 0.77 0.88]
v=[0.21 0.48 0.6 0.74]
The AUC results of the several methods that I quickly tested:
# Degensac with ransac threshold = 2 at error thresholds [3, 5, 10] px
SuperPoint: 0.53 0.65 0.78
SuperPoint+SuperGlue: 0.56 0.68 0.81
D2Net: 0.45 0.58 0.73
R2D2: 0.53 0.64 0.76
Patch2Pix (w.NCNet): 0.53 0.65 0.78
CAPS (w.SuperPoint): 0.53 0.66 0.78
LoFTR (all matches): 0.59 0.69 0.8
# Opencv with ransac threshold = 2 at error thresholds [3, 5, 10] px
SuperPoint: 0.37 0.51 0.68
SuperPoint+SuperGlue: 0.39 0.53 0.71
CAPS (w.SuperPoint): 0.33 0.49 0.67
LoFTR (all matches): 0.48 0.6 0.74
Feel free try out other settings. While I manage to produce similar or better AUC results of SuperPoint+SuperGlue, D2Net and R2D2 to those reported in Tab.1 of LoFTR paper, I didn't manage to reproduce it with LoFTR using either all matches or top1k matches.
If you get reply from zju3dv/LoFTR#136, I can then figure out how to update the code for LoFTR.
from image-matching-toolbox.
Awesome! Thanks.
I am very confused regarding how AUC was actually implemented in either SuperGlue or LoFTR, hopefully they will respond soon.
I know that they used ransac threshold=3, but I find it difficult to believe that that would increase the AUC with such a high amount.
I'll give an update if they respond.
from image-matching-toolbox.
@GrumpyZhou LoFTR additionally scales their homographies to fit images resized so that their shorter side is 480. I'm not sure how much this matters, but I know that some images are very large, so it could make a difference.
from image-matching-toolbox.
Hi @Parskatt ,
I followed the modification:
- Resize the image shorter side to 480
- Set ransac threshold to 3
AUC@ 1 3 5 10 px:
LoFTR (all matches, opencv): [0.23 0.49 0.61 0.74]
LoFTR (all matches, degensac) [0.25 0.55 0.67 0.79]
LoFTR (1k matches, opencv): [0.27 0.5 0.61 0.74]
LoFTR (1k matches, degensac) [0.3 0.55 0.67 0.79]
Hm...I don't think this is an issue with the AUC metric implementation. Also considering I have used this repo to reproduce LoFTR results on Aachen benchmark, it is most likely to be a setting difference on HPatches. Directly using their reported number or asking the authors to provide the code to reproduce those numbers is the simplest solution if you want to compare your method to LoFTR.
I am not sure I will have time to dig into this and try to reproduce that table 😅.
from image-matching-toolbox.
Thanks for all your work :)
from image-matching-toolbox.
@Parskatt , I also want to reproduce the results of LoFTR for HPatches. Do you find the solution for this problem?
from image-matching-toolbox.
@TruongKhang Sorry, no, I was not able to reproduce their results. I think it's better to compare to their method on MegaDepth and Scannet, which is reproducible.
from image-matching-toolbox.
@Parskatt I wonder which settings they used in order to make a huge difference of performance like that. I'll try to ask them in more detail. Most methods were trained on Megadepth and then evaluated on HPatches. I think an evaluation on HPatches is necessary
from image-matching-toolbox.
I have managed to get quite close to the numbers reported in the paper, see #22 . @Parskatt & @TruongKhang, you were probably right when you suggested that they rescale the homography and compute the errors in the downscaled images.
from image-matching-toolbox.
wow, awesome @georg-bn @GrumpyZhou! I tried to do something like that I couldn't get good results. I might do wrong somewhere. Thank you for fixing the code!
from image-matching-toolbox.
I think it makes sense to calculate the errors in the downscaled images since the aim is to make all images of approximately equal importance in the final score. As long as the other methods are evaluated in the same way I believe this to be fair.
It would make even more sense to make all the images the exact same size, e.g. in SuperPoint they are resized to 640x480 (or 480x640 depending on which side is largest). I tried this and obtained:
>>Finished, pairs=540 match_failed=0 matches=2547.3 match_time=0.12s
==== Image Matching ====
#Features: mean=2547 min=94 max=4235
#(Old)Matches: a=2547, i=3168, v=1971
#Matches: a=2547, i=3168, v=1971
MMA@[ 1 3 5 10] px:
a=[0.63 0.91 0.93 0.95]
i=[0.68 0.96 0.96 0.98]
v=[0.59 0.86 0.91 0.92]
==== Homography Estimation ====
Hest solver=cv est_failed=0 ransac_thres=3.0 inlier_rate=0.87
Hest Correct: a=[0.64 0.87 0.92 0.95]
i=[0.82 0.98 0.99 1. ]
v=[0.47 0.77 0.85 0.9 ]
Hest AUC: a=[0.36 0.65 0.75 0.84]
i=[0.5 0.79 0.87 0.93]
v=[0.24 0.52 0.64 0.76]
Furthermore, by increasing the matching threshold from 0.2
to 0.5
I got with the "short side=480"-setting:
>>Finished, pairs=540 match_failed=0 matches=2238.2 match_time=0.12s
==== Image Matching ====
#Features: mean=2238 min=7 max=5571
#(Old)Matches: a=2238, i=2873, v=1649
#Matches: a=2238, i=2873, v=1649
MMA@[ 1 3 5 10] px:
a=[0.67 0.94 0.96 0.97]
i=[0.71 0.98 0.99 1. ]
v=[0.63 0.9 0.93 0.94]
==== Homography Estimation ====
Hest solver=cv est_failed=0 ransac_thres=3.0 inlier_rate=0.91
Hest Correct: a=[0.64 0.87 0.92 0.95]
i=[0.83 0.98 0.99 1. ]
v=[0.47 0.76 0.86 0.91]
Hest AUC: a=[0.37 0.65 0.75 0.84]
i=[0.51 0.79 0.87 0.93]
v=[0.25 0.52 0.64 0.76]
And with matching threshold 0.5
and all images scaled to 640x480 or 480x640:
>>Finished, pairs=540 match_failed=0 matches=2194.5 match_time=0.12s
==== Image Matching ====
#Features: mean=2195 min=7 max=4199
#(Old)Matches: a=2195, i=2845, v=1590
#Matches: a=2195, i=2845, v=1590
MMA@[ 1 3 5 10] px:
a=[0.67 0.94 0.96 0.97]
i=[0.71 0.98 0.99 1. ]
v=[0.63 0.9 0.93 0.94]
==== Homography Estimation ====
Hest solver=cv est_failed=0 ransac_thres=3.0 inlier_rate=0.90
Hest Correct: a=[0.64 0.87 0.92 0.95]
i=[0.83 0.98 0.99 1. ]
v=[0.46 0.77 0.86 0.9 ]
Hest AUC: a=[0.38 0.66 0.76 0.85]
i=[0.51 0.8 0.88 0.94]
v=[0.25 0.52 0.64 0.76]
This last run does match the numbers reported in the paper, but I have no idea if the settings actually are "correct" - probably not as I don't see why they would have changed the matching threshold. I will try asking the authors again in zju3dv/LoFTR#136
from image-matching-toolbox.
@georg-bn, if we can reproduce the results as approximately the same as in the paper, I think we can directly get the reported results in the paper for comparison. The matching threshold is a tuning hyper-parameter and changing it in the evaluation on other datasets is normal, I think
from image-matching-toolbox.
@TruongKhang Yes, I suppose you are correct.
from image-matching-toolbox.
Some final(?) observations: Changing to only using top 1000 matches worsened the performance somewhat, but this can be compensated by setting the confidence
in cv2.findHomography
to 0.99999
, which is the RANSAC-confidence used in the LoFTR pose estimation experiments.
from image-matching-toolbox.
@georg-bn, I find that the results are changed a little every time I re-run the HPatches evaluation. Do you have this problem?
from image-matching-toolbox.
Related Issues (20)
- LoFTR has a bug in position_encoding.py
- COTR no matches found in hpatches test HOT 3
- Test loftr in inloc HOT 12
- Where is the alignments in InLoc? HOT 1
- Missing RobotCar/queries HOT 9
- robotcar-season eval run very slowly HOT 3
- Test loftr in aachen HOT 2
- python setup.py develop failed HOT 1
- what is the meaning of $colmap path ? HOT 1
- The possibilities of python to C++ HOT 1
- Feature point annotation tool HOT 10
- Errors in loading state_dict for patch2pix HOT 1
- FileNotFoundError: no 'cameras.bin' when test loftr on aachen
- Images illumination change evaluation
- I have a question
- Installation Problems, HOT 1
- Pycolmap Error not found during install HOT 1
- IMMATCH ModuleNotFoundError HOT 1
- Download.sh for pretrained weights DO NOT WORK HOT 1
- Patch2Pix AttributeError: 'Conv4d' object has no attribute 'weight'
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from image-matching-toolbox.