"send_images" command surprisingly slow about tdw HOT 4 CLOSED

Thanks both for your answers!

For context, I am evaluating threedworld to use for learning object manipulation from raw images using CV + RL, and from the feature matrix on your website it seemed like the perfect environment (good graphics + physics). So a live stream of images is exactly what I need, although 720p might be slightly overkill.

I understand all of your points, I already excluded everything except (1) reading pixels from Unity and (2) compressing the image as only likely reasons for what I observe based on experiments I ran.

From the benchmark you pointed out (thanks!), I was able to find that indeed you do have a flag for the point (2) that I asked for: {"$type": "set_img_pass_encoding", "value": False}, and by setting this I indeed get a considerable speed-up, further confirming my suspicion. That's great, as I don't care about transfer costs: everything runs on localhost.

(Also, the benchmark on my machine is similar to the expected values, confirming my setup is fine.)

Regarding (1), I have never done this in Unity, but in OpenGL, it should not lead to that dramatic slowdown. In fact, that's exactly what primusrun and optirun commands for linux laptops do: render into a pixel buffer (PBO) on the discrete GPU, and move this onto the CPU's integrated GPU for rendering, and they do so in real-time.

One more comment:

The _img passes defaults to a jpg. All other passes need to be lossless png to be accurate.

This does not seem to be the case. I'm looking at the raw bytes I'm getting, and the RGB image definitely has the typical \x89PNG header, so I'm sure it's png encoded, not jpg.

Anyways, {"$type": "set_img_pass_encoding", "value": False} is what I was looking for, thanks.

from tdw.

Hi Lucas --
You can send the "send_images" command once, with frequency of "always"; sending the command every frame is going to cause a slowdown. The objects_and_images.py example controller shows a slightly different way of structuring image saving. Have you tried that controller, to see what difference in performance that makes on your system?

While getting pixel data out of Unity is always going to be a bit of a bottleneck, you should be able to get more like 35-40 fps at that resolution. We have published benchmarks on the repo that you can refer to.

from tdw.

@lucasb-eyer

For benchmarks: https://github.com/threedworld-mit/tdw/blob/master/Documentation/benchmark/image_capture.md
For information re: performance optimizations: https://github.com/threedworld-mit/tdw/blob/master/Documentation/benchmark/performance_optimizations.md

Known causes of image capture slowdown:

• Reading the pixel buffer is slow (there is no way to avoid this)
• The amount of data being sent over the wire is high (see this document for how the size of the byte array affects performance)
• Encoding to jpg or png is slow (there is no way to avoid this in Unity) But the raw data is so larger that it's better to encode than not.
• Running in Unity Editor is very slow
• Saving images to disk (or converting them into PIL) can be slow (though this is probably not as big a factor as the other factors, and can be mitigated with multithreading)

It seems that the image is sent to the controller as a png encoded image. I believe this is where the slowdown comes from. It is not possible to encode an image into png format in real-time for higher resolutions in my experience.

The _img passes defaults to a jpg. All other passes need to be lossless png to be accurate.

from tdw.

@lucasb-eyer Closing this Issue. I ran your controller and you're not doing anything incorrect... Unfortunately, this an issue with how Unity reads the pixels off the GPU.

A few ways to speed up the simulation:

1. Reduce screen size
2. Only send {"$type": "send_images", "frequency": "once", "avatar_id": avatar_id} when you actually need the image; rendering without this command is much faster.
3. If all you need is a video and not the frames themselves, video capture with an external program might be faster.

from tdw.