Giter Club home page Giter Club logo

mr2nerf's Introduction

Meshroom camera location to NERF conversion tool with per camera intrinsics

This tool is for use with https://github.com/NVlabs/instant-ngp and allows the use of Meshroom camera locations.

Usage

In Meshroom, add a ConvertSfMFormat node and change the SfM Format to json. af32538a-bd34-4c62-aa1d-9706002ea0a0

Optional: Meshroom does not align or constrain solved cameras, you may want to add a SfMTransform after the StructureFromMotion node, set the Transformation Method to Manual, and adjust the location and rotation of the solved cameras.

When you Start the Meshroom processing, it will generate a folder for the output of the ConvertSfMFormat node, which you can easily find by right clicking on the node and selecting Open Folder. The file sfm.json will be generated when Meshroom is done processing. This is the file you need for this script's --input function. 0eec4a33-97a7-4c38-91dd-5261500aae68

Run the mr2nerf.py on this JSON file using the following command, replacing the text in brackets [โ€ฆ] with the file names and paths on your machine:

Commands

Example:

  • Replace [PATH_TO_JSON_FILE] with the path to the JSON that was exported from Meshroom
  • Replace [PATH_TO_IMAGES] with the path to your images folder
python mr2nerf.py --input "[PATH_TO_JSON_FILE].json" --imgfolder "[PATH_TO_IMAGES]"

The quotes are only required if you have spaces in any of the folder or file names.

Additional command examples

Scale the scene up by 100

python mr2nerf.py --input "[NAME_OF_JSON_FILE].json" --imgfolder "[PATH_TO_IMAGES]" --scale 100

Display the cameras in 3d and set the camera size (for debugging only, requires installing matplotlib and pytransform3d)

python mr2nerf.py --input "[NAME_OF_JSON_FILE].json" --imgfolder "[PATH_TO_IMAGES]" --plot --camera_size 1

Arguments:

Argument Default Value Description
--input None specify json file location
--out transforms.json specify output file path
--imgfolder .\images location of image folder
--imgtype jpg ex.: jpg, png, ...
--aabb_scale 16 sets the aabb scale
--no_scene_orientation False disable the Reality Capture orientation
--no_scale False disable the Reality Capture scale
--no_center False disable the scene centering
--plot False display the camera positions
--camera_size 0.1 the size of the displayed cameras
--debug_ignore_images False ignores the input images, for debugging only
--threads 8 number of threads to use when reading images

mr2nerf's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

mr2nerf's Issues

I tried the script on a few 100 shots from my phone and got this:

Hello,
first time trying this so bare with me please and sorry if I don't really get the instructions..
I am not really sure what to do with the manual transform though is there something I need to enter in the SfMTransform (manual) Translation x y z or Euler Rotation? I left everything to 0 in doubt.

2023-07-30 00:39:07 | INFO | font_manager.py : 1544 | >>> generated new fontManager
2023-07-30 00:39:10 | WARNING | mr2nerf.py : 109 | >>> W/H ratio does not match sensor ratio, this is likely a bug from Meshroom. Will use fl_x to set fl_y.
Traceback (most recent call last):
File "C:\Users\Shady\Documents\mr2nerf\mr2nerf.py", line 226, in
camera['transform_matrix'] = transforms[poseId]
~~~~~~~~~~^^^^^^^^
KeyError: '137075643'

image

Script test doesn't yield expected results. Could there be a bug?

Dear joreeves

I tried to use your script but did not arrive at the expected result. I generated camera postions from 17 pictures in Meshroom via the StructureFromMotion node. I exported this to a json file via the ConvertSfMFormat node per your instructions. I then ran your script to generate the transforms.json file and arrived at following results.
As example 1 of 17 source pictures:
IMG_3299

Results:
MeshroomCameras_ToInstantNGP_Result01
MeshroomCameras_ToInstantNGP_Result02
MeshroomCameras_ToInstantNGP_Result03

I got these messages when I ran the script:
image

The dense point cloud I generated from Meshroom some nodes down the line however seems perfectly fine so I do not really doubt the correctness of the generated camerapositions, not in Meshroom and also not in your script as can be seen above in the InstantNGP screenshots.
MeshroomDensePointcloud_Result01

Then I tested to generate the transforms.json file anew via the instantngp-batch procedure described over here:
https://github.com/jonstephens85/instantngp-batch
The procedure first runs colmap to detect camera postions anew and then uses the script colmap2nerf.py to generate the transfroms.json file. This procedure yields significantly better results. See below.
ColmapCamerasViaColmap2Nerf_ToInstantNGP_Result01
ColmapCamerasViaColmap2Nerf_ToInstantNGP_Result02
ColmapCamerasViaColmap2Nerf_ToInstantNGP_Result03

However, I have needed to calculate the camerapositions twice now, once in Meshroom and once via Colmap. I would prefer to only do this once and see promise for your script if it would yield the same results as the Colmap procedure.
I was wondering if there maybe could be a bug somewhere in your script? For example, I see that the considered volume that the neural network calculates is considerably larger when using the transforms.json file generated by your script compared to the volume used by the colmap generated transforms.json file.
Looking forward to any answer you can give. Please let me know if you need any of my testfiles. I am more than happy to provide those.

For your convenience I have already included both transforms.json files generated by Colmap and by Meshroom and your script:
transforms_FromColmap.json.txt
transforms_FromMeshroom.json.txt

Kind regards
Paulus

Asymmetric Focal Length

Focal Lengths in the output transforms.json came out like so:

"fl_x": 3282.2516711791563,
"fl_y": 5842.794435803687,

For the dataset I was testing "fl_x" looked right, but usually "fl_y" is the same value so it seemed strange. When testing in intant-ngp in never converged. When I manually set "fl_y" to the same value it converged.

Looking at the sfm.json I generated from meshroom it looks like there's only a single value for focal length (I assume in millimeters). Is the script converting based on this value and scaling based on image dimensions?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.