Explore Street View depth maps about kdb HOT 29 CLOSED

The Elevation API is not yet open to the public/research. May happen in a few months. Can't really count on it.

from kdb.

In the mean time, this is still open: https://maps.google.com/cbk?output=xml&ll=45.508457,-73.532738&dm=1

And we can generate such an image using https://github.com/cc-ai/floods-gans/blob/master/various_scripts/depth_map.py

Now open questions:

• can we map an exact StreetView position to such a query?
  • can we get the original image from which this is computed?
  • can we specify pitch, camera view, angle and so on
• is the code fast enough + scalable?
• how to compute a water plane from this data?
• how to embed said plane into original image?

GitHub

cc-ai/floods-gans

Using GANs to simulate how a house would look like, had it suffered from a flood - cc-ai/floods-gans

from kdb.

Before going into matching depth-map and street view images automatically, I would like to know if you found GPS coordinates where you managed to match it qualitatively playing on streetview? If so could you please communicate them? I can't find a single place where I can match it visually.

from kdb.

I agree I have not and it is an open question since the API is not documented (or is it?)

from kdb.

@DatCorno on the backend we are using the same API or it's different?

from kdb.

Reconstructing the spherical img
There are some steps to automatically reconstruct the spherical image relative to a pano_id that should be developped but in the meantime this App do the work for us.

Reconstructing 3D point cloud
Here is a notebook colab to explore depth-map where I tested if we can superpose depth map to the panoramic img without success and where I translated some medium advices to generate the 3D point cloud (there is work to do before having a height in meter)

from kdb.

Ok so this is not good enough for us to work with right? This API is still open but basically it's not as informative as we expected?

from kdb.

I would not say that, I believe we should write our own code following the steps in this paper explaining how to reconstruct the panorama ourself from the tiles and see if it match the depth map: There is a lot of potential in this approach.

from kdb.

@gcosne how long it will take in your opinion, if we write our own code?

from kdb.

Somehow I find a panorama that almost match the depth map :

Remark : I discover one thing: Height of the Lidar is about 2.5 meters. Using the formula Normalized/minimum(height) * Lidar-Height allow to translate the normalized coordinates in meters.

Then I tried to" flood "the image until a certain height but I think the lidar doesn't have a lot of points below 1.5m. Hence the segmentation map generated is good when height >2.m but get sparse when we go below 1.7m.
Flood up to 1.7m

Flood up to 1.3m

from kdb.

Pas mal du tout. La question qui nous reste sur les bras c'est: est-ce qu'on peut travailler avec ces panoramiques? i.e. est-ce que c'est suffisamment "relatable" pour en faire qqch et est-ce qu'on peut l'avoir pour n'importe quel endroit / si non lesquels?

Il faut en parler à Alice et mettre au point un premier groupe de testing utilisateur pour se faire un avis, imho

from kdb.

oups French, I meant: this is not bad at all! we ought to research that with users to understand if that's usable / relatable for our purpose + figure out if there are restrictions on localizations for which we can get this depth map + original panorama

from kdb.

from kdb.

very interesting, good job, so you found the magic number I was asking :)
for the flooding part maybe we can find another technick, like reconstructing the scene in 3D and flood afterwards ... but not sure for now, maybe too heavy, we can think more about it

from kdb.

Testing with depth estimation : Using very light single image depth estimator rather than 'google api', we almost have usable masks.

To Be Done:

• Identify if we have region where google api works for us.
• Calibrating depth estimator with google api.

Qualitative Results flooding up to 1.5m :

from kdb.

well, the flood part doesn't work ...
btw, what about people on street view images? should we "drown" them?

from kdb.

from kdb.

Update :
I translated DownloadPanorama C++ function from this incredible project in python.

We can now automatically extract xml information and generate both depthmap and panoramic images within a script. I think I will explore more of PaulWagener project, I sent him an email.

Cropping the image so they are the same size rather than resizing them actually work better. Not perfect yet but I will fix that soon.

from kdb.

from kdb.

Not yet but I do trust it will, I have to play with yaw heading.

from kdb.

from kdb.

Update : We can almost call it a success.

I said almost because the size of the tiles are not consistent, sometimes the 360° equirectangular image is already built (no overlapping between tiles), sometimes it's not. The good news are that we can identify it by the amount of black padding in the image.

          360° panorama without overlapping                                     More than 360° panoramic image

from kdb.

from kdb.

I assume so, nothing of an issue here but just a thing to be aware.

Next steps :

• Finish the work in low resolution.
• Create the stitching function for already built panoramic images in higher resolution.

from kdb.

from kdb.

cool, good job ;)

from kdb.

Done: Low and High resolution are working. I will clean and commit the code soon.

Next steps : Make it work with the 3D.

• Flood the ground.
• Flood a certain height.

from kdb.

from kdb.

3D reconstruction: There is an echo around the car that makes the ground plane estimation tricky

from kdb.