erlerphilipp / pix2model Goto Github PK

New develop environment without certificates and custom port which is reachable inside the TU Netz,

send_stop_job_command responds with an job-not-found exception, which is due to a different naming scheme for the jobs compared to the folders that are created for a job (i think)

Currently, user get this message after starting a job:

When they close the tab, the UID is gone for good and the job wastes resources. Here are a few ideas to prevent this:

1. Write explicitly that users should keep the link / UID saved somewhere.
2. Open the UID-link so that it is stored in the browser history.
3. Switch directly to the editor and wait for the results to show up.

I would prefer a combination of 2. and 3. (1. would be obsolete). But this requires:

• The editor must always be available, or at least a simpler version.
• The editor needs to differentiate between "file not found" and "in progress". The "in progress" should be more visible then.

You can click on the drop area to upload more images while the upload of one job runs. However, the additional images are not enqueued with the other ones. If you click again on submit, the second ones are submitted to a another job.

Looks like it works but it's a strange mixup of separate jobs. Also, only one result link is show. The easiest and best solution is probably to disable additional file drops while the upload runs.

https://www.cg.tuwien.ac.at/content/Datenschutzerkl%C3%A4rung
https://www.cg.tuwien.ac.at/impressum
Unter Team einfügen?

my brother uploaded images, probably with the "convert to pointcloud" option. the pointcloud appears correctly in the editor but i get an error when i click on "reconstruct mesh". no info is given in the error-log. manual reconstruction with meshlab works, so i suspect an interface error.

http://netidee.cg.tuwien.ac.at/?id=e6f3d1ee-5eb2-4492-b8a0-0ba2d98d718f

"Programmiere einen Prozess, der

• n Fotos (als PNG, JPEG, NPY etc.) bekommt
• Structure-From-Motion darauf anwendet und
• eine Punktwolke erzeugt. PW als npy array speichern.
  Die genutzte Variante von SFM soll schnell zu implementieren sein, vorhandene Bibliotheken checken, wie z.B. OpenCV. Lizenz checken!
  Etwas Recherche und mit Stefan absprechen, welche Variante gut geeignet ist."

"Programmiere einen Webservice

• mit minimalem Web-Interface
• Upload für Fotos (Datenmenge limitieren, um DOS zu verhindern)
• Lasse die Fotos durch die Pipeline (1.1-1.3) laufen
• Gefärbtes Mesh zum Download anbieten
• Webservice Set-up dokumentieren, damit alle im Team testen können
• Überlastung verhindern durch z.B. eine einfache Warteliste
  Checke Django, Heroku, (Nvidia) Docker"

Before cropping:

After cropping:

Looks like the normals are not updated when deleting points (which invalidates their IDs).

Check if the automatic removal of old data works reliable

Add a small delete button in the upload area on each image so they can be removed before uploading without re-selecting all images.

Seems like the automatic SSL certificate renewal with certbot fails:

"Failed to renew certificate netidee.cg.tuwien.ac.at with error: Some challenges have failed."

Server umbauen, damit die einzelnen Pipeline-Schritte getrennt laufen können. Z.B. je ein Microservice (Docker-Container) pro Schritt. Warteschlange mit Schätzung der Wartezeit. Mehrere Prozesse gleichzeitig erlauben, aber nur 1 Rekonstruktion pro Grafikkarte. Upload von Punktwolken erlauben (SFM überspringen). Fortschritt und Position in Wartenschlange auf Website anzeigen.

In the default pipeline with our test data (100 images), most of the computation time is spent with Meshing and Texturing. We can expect a significant speed-up by replacing some of the steps, roughly 25 min -> 10min. Also, there are newer methods available that should improve the quality.

This will be the improved pipeline:

Changes are only in Step 2 (after editor):

1. Normal Estimation (low priority): The new meshing can easily be extended to use rough normals. AliceVision likely doesn't output a point cloud with normals. We can make them ourselves from the camera poses. The hemisphere is known and will help the reconstruction a lot.
2. Meshing: Use POCO or similar. Retrain with the P2S ABC var-noise dataset. Add data augmentation (outliers, scene-based). Fix POCO bugs (marching cubes indexing, global layer ignored, kd-tree workers).
3. Mesh Filtering: Since we're not working with AliceVision's Tetrahedral reconstruction anymore, we need a different filtering method. The best choice is probably Quadric Mesh Simplification (PyMeshlab).
4. UV Unwrap (low priority): AliceVision's Meshing includes an unwrapping step, which is quite slow and produces inefficient textures. An alternative could be an automatic unwrapping by PyMeshlab or Blender. AliceVision's Texturing node can stay but the parameters must be set so that it uses the existing UV coordinates.

Implement the default Meshroom Pipeline with only minor changes. Minor changes mean the editor must still work and some parameters should be changed.

AliceVision binaries need to be called like from Meshroom. This will require adapting the lib path before the CLI calls:
cmdBase = '**export LD_LIBRARY_PATH**=$LD_LIBRARY_PATH:{}/aliceVision/lib'.format(absComputePath) + ' && {}/aliceVision/bin/aliceVision_convertSfMFormat'.format(absComputePath)

Adapting the default pipeline to the editor:

• Transforms can be applied to the data with a SfM Transform node.
• The handling of the cropping area in the editor needs to change. The meshing node has a parameter for a custom bounding box, which needs to come from the editor. If none is provided, the meshing will find its own.

Requires Meshroom Docker #162

Read this: https://sketchfab.com/blogs/community/tutorial-meshroom-for-beginners/

Here are some rough measurements and parameter suggestions:

Upload images one after the other to avoid multi-processing overhead? Is this a relevant issue for the server?

Add link to Meshroom test data: https://github.com/alicevision/dataset_monstree
Add elephant test data

1. pix2model-backend-1 | [2023-08-01T14:43:57Z][INFO] WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
2. babel-preset-react-app is part of the create-react-app project, which is not maintianed anymore. It is thus unlikely that this bug will ever be fixed. Add "@babel/plugin-proposal-private-property-in-object" to your devDependencies to work around this error. This will make this message go away.
3. material ui outdated

Meshroom / AliceVision has many parameters for each step. We need to find out which ones are crucial and find some good settings for our application. The focus is on speed, and quality second. Let's try to get the best possible quality within ~20 min compute time.

The most relevant steps to speed up are meshing and texturing.

Read this: https://sketchfab.com/blogs/community/tutorial-meshroom-for-beginners/

add a link to this repo's issues to the website so that users can notify us when the server is broken.

maybe next to privacy/impressum?

Clarify:

• is ssh_key in secrets really the public key?
• have both workflows in both branches?
• check in deploy script? vars file?
• point by default to deploy.sh instead of start.sh
• single command in readme to create default env file with contents?
• extend certificate section

@akiwa please comment on these ideas / questions

Check dependabot updates

Try on dev branch

There is a new release of PyMeshlab: cnr-isti-vclab/meshlab#1218

possibility to select number of threads in filter screened poisson

@thefloff Is this worth a try? Was it single threaded before or used all available cores?

Be careful with the update, they apparently changed the filter names.

Austria / EU apparently requires a Accessibility Statement.

1. Check if our website fulfills any of the requirements: https://www.ffg.at/digitale-barrierefreiheit/digitales-zugaenglich-machen/kriterienkatalog-web
2. Fix simple issues.
3. Write a declaration
   1. Info: https://www.ffg.at/barrierefreiheit/barrierefreiheitserklaerung
   2. Template DE: https://www.ffg.at/digitale-barrierefreiheit/wissenswertes-und-fakten/mustererklaerung-de
   3. Template EN: https://www.ffg.at/en/digital-accessibility/facts/model-accessibility-statement

DOS verhindern, indem häufige IP-Addressen blockiert werden. Admin-Zugang, um einzelne Prozesse beenden zu können und Parameter (z.B. Größen-Limits und Grid-Resolution) für einzelne Nutzer ändern zu können. Login mit temporären Login-Daten. Login-Daten und alle anderen Daten auf Knopfdruck sofort von Server löschen, sonst nach 1 Tag. Auf Wunsch Email senden, wenn das Ergebnis bereit zum Download ist. Server-Telemetrie und Email, wenn der Server nicht mehr erreichbar ist (status-checker Prozess auf anderen CG Server?).

"Programmiere einen Prozess, der

• eine Punktwolke (als NPY, LAS etc.) bekommt
• Screened Poisson Surface Reconstruction (PyMeshlab) darauf anwendet
• ein Mesh erzeugt. Mesh als STL oder PLY speichern.
  Lizenz von PyMeshLab checken!"

We want to test changes on the server using the latest commit of the develop branch.

This is partially implemented but not ready yet. A key problem is dealing with two different servers on separate GPUs but they need to share parts of the system.

"Basic editing von Punktwolke und/oder Mesh:

• cropping, verschieben, skalieren, rotieren, zurücksetzen
• in Web-Interface using WebGL
• Bounding-Box längen anzeigen"

Possible speed-up with minor quality impact -> low priority

Check if this is even necessary. MVS in AliceVision (Prepare Dense Scene, Depth Map, Depth Map Filter) doesn't take very long with ~1min.

Meshing should be faster too, which could make it more relevant. AliceVision's Meshing will be impacted a lot by using only the sparse point cloud. So we should do this only after switching to our own meshing.

probably related to #133

"Oberflächenrekonstruktion durch Points2Surf ersetzen.
Kleinere Änderungen sind nötig:

• Punktwolke vor Verarbeitung auf -0.5…+0.5 skalieren
• das erzeugte Mesh auf die ursprüngliche Größe der Punktwolke skalieren
• sehr große Punkwolken vorher down-samplen auf ca. 50k Punkte
• Grid-resolution für besser Geschwindigkeit optimieren
• Neu trainieren mit ABC+Famous+Thingi10k datasets?"

"Wenn der Server im Testbetrieb mit den einfachen Algorithmen läuft, erste Tests mit Bekannten und Verwandten machen.
Klären, ob die Ergebnisse die Erwartungen erfüllen und warum. Tests mit verschiedenen Mengen von Fotos, verschiedene Qualität, Lichtverhältnisse, Objekte etc. Richtlinien für nötige Anzahl Fotos suchen. Gefundene Probleme entweder gleich beheben oder in den Workplan eintragen.
Kontakt zu Künstlern, Museen etc. herstellen und testen lassen.

Zwischenbericht schreiben und 2. Rate beantragen

"Genauere Recherche zusammen mit Stefan, um die beste Variante von SFM für die in den Tests gefundenen Bedingungen auszuwählen.
Implementierungsaufwand abwägen.
Implementieren, evtl. in C++, sonst wahrscheinlich in Python."

Upload elephant test data and Meshroom project

the upload field is deaktivated after klicking on submit, even when submitting no images. don't do the deactivation if zero images are selected?

Minimize interruptions caused by CI deploying a new commit.
Best idea so far: make CI wait until the tasks queue is done.
Open problem: this will take a while if the server is busy.

"Nearest Neighbor Vertex Colors (NNVC) ist sehr einfach und wird bei kleineren Grid-Resolutions schlechte Ergebnisse liefern. Mit Philipp, Stefan und/oder Michi absprechen, wie man das verbessern kann. Mögliche Lösungen:

• Sub-division des Meshes und dann NNVC
• automatisches UV-Mapping und Erzeugung von Texturen aus den Punktwolken.
• Texturen aus den Fotos?"
• Paper: "Texture Defragmentation for Photo-Reconstructed 3D Models" will come to Meshlab
• Interessantes Paper: "Point2color: 3D Point Cloud Colorization Using a Conditional Generative
  Network and Differentiable Rendering for Airborne LiDAR"

"Programmiere einen Prozess, der

• eine Punktwolke (als NPY, LAS etc.) bekommt
• ein Mesh (als STL oder PLY) bekommt
• für jeden Vertex im Mesh den nächsten Punkt in der Punktwolke sucht
• die Farbe vom Punkt zum Vertex überträgt und
• das gefärbte Mesh speichert.
  Nearest-neighbor search mit spatial acceleration structure, z.B. scipy.ckdtree."

Got the same error for all uploaded images:

Any idea what the reason might be?
Display error without HTML tags?

"Hochgeladene Fotos anzeigen, Punktwolken und Meshes (mit WebGL) anzeigen.
Extra Viewer, mit dem man bereits runtergeladene Meshes anzeigen kann."

Save error log to textfile in output folder

There is currently an ugly error message in the upload field for images that are too large. A specific human readable error message should be implemented.

Image for reference

I moved browser tab with active editor from one screen to another. The gizmo doesn't match the cursor position anymore.
Seems like some cursor offset is not updated. DPI scaling and different resolutions might play a role.

Taskmanager shows that the editor keeps the GPU busy:

although there is not yet anything to display:

Stop render loop until the object is ready?

i uploaded several images. during upload, i canceled one image upload. all uploads are broken afterwards:

"Zusammen mit den Technikern des CG Instituts die Hardware besprechen. Ca. Ryzen 7, 2 x RTX 3070, 64 GB RAM.
Domain für den Webservice absprechen und von den Technikern einrichten lassen. Es wird vielleicht eine Sub-Domain wie
images2mesh.cg.tuwien.ac.at
Bei Lieferschwierigkeiten evtl. einen alten Rechner temporär verwenden.
SSH-Zugang für Projektmitarbeiter?"

• Points2Surf ist relativ langsam. Mit iterativem Verfeinern wie MISE in Occupancy Networks müsste es schneller und genauer werden.
• Grid auf Bounding Box limitieren.