A procedural blender pipeline to generate images for deep learning

The corresponding arxiv paper: https://arxiv.org/abs/1911.01911

In general, one run of the pipeline first loads or constructs a 3D scene, then sets some camera positions inside this scene and in the end renders different types of image (rgb, depth, normals etc.) for each of them. The blender pipeline consists of different modules, each of them performing one step in the described process. The modules are selected, ordered and configured via a yaml file.

To run the blender pipeline one just has to call the run.py script in the main directory together with the desired config file.

python run.py config.yaml <additional arguments>

This will now run all modules specified in the config file step-by-step in the configured order.

The following modules are already implemented and ready to use:

• Load *.obj files and SunCG scenes
• Automatic lighting of SunCG scenes
• Loading camera positions from file
• Sampling camera positions inside SunCG rooms
• Rendering of rgb, depth, normal and segmentation images
• Merging data into .hdf5 files

For advanced usage which is not covered by these modules, own modules can easily be implemented (see Writing modules)

• Basic scene: A small example loading an .obj file and camera positions before rendering normal and color images.
• Simple SUNCG scene: Loads a suncg scene and camera positions from file before rendering color, normal, segmentation and a depth images.
• SUNCG scene with camera sampling: Loads a suncg scene and automatically samples camera poses in every room before rendering color, normal, segmentation and a depth images.

A very small config file could look like this:

{
  "setup": {
    "blender_install_path": "/home_local/<env:USER>/blender/",
    "blender_version": "blender-2.80-linux-glibc217-x86_64",
    "pip": [
      "h5py"
    ]
  },
  "global": {
    "all": {
      "output_dir": "<args:0>"
    },
    "renderer": {
      "pixel_aspect_x": 1.333333333
    }
  },
  "modules": [
    {
      "name": "renderer.NormalRenderer",
      "config": {
        "samples": 255
      }
    }
  ]
}

To prevent the hardcoding of e.q. paths, placeholder are allowed inside the configuration:

When starting the pipeline, the blender version and python packages required for the given config are automatically installed. Such software related options are specified inside the setup section of a config.

The section modules consists of a list of dict objects which all specify a module. Every of these module specifications has the following properties:

The modules are executed in the exact same order as they are configured inside the modules section.

This section contains configuration parameters that are relevant for multiple or all modules. The configuration specified inside all is inherited by all modules, while the config specified inside <dir> (e.q. renderer) is inherited by all modules with the prefix <dir>. (e.q. renderer.NormalRenderer)

A module is a class executing one step in the pipeline.

Here is the basic structure of such a module:

from src.main.Module import Module

class CameraLoader(Module):

    def __init__(self, config):
        Module.__init__(self, config)
        [...]

    def run(self):
        [...]

The constructor of all modules is called before running any module, also in the order specified in the config file. Nevertheless it should only be used for small preparation work, while most of the module`s work should be done inside run()

The module`s configuration can be accessed via self.config. This configuration object has the methods get_int, get_float, get_bool, get_string, get_list, get_raw_dict, each working in the same way.

• The first parameter specifies the key/name of the parameter to get. By using / it is also possible access values nested inside additional dicts (see example below).
• The second parameter specifies the default value which is returned, if the requested parameter has not been specified inside the config file. If None is given, an error is thrown instead.

Example:

Config file:

{
  "global": {
    "all": {
      "output_dir": "/tmp/",
      "auto_tile_size": false
    },
    "renderer": {
      "pixel_aspect_x": 1.333333333
    }
  },
  "modules": [
    {
      "name": "renderer.NormalRenderer",
      "config": {
        "auto_tile_size": true,
        "cycles": {
          "samples": 255
        }
      }
    }
  ]
}

Inside the renderer.NormalRenderer module:

self.get_int("cycles/samples", 42)  
# -> 255

self.get_float("pixel_aspect_x") 
# -> 1.333333333

self.get_string("output_dir", "output/") 
# -> /tmp/

self.get_bool("auto_tile_size") 
# -> True

self.config.get_int("resolution_x", 512)
# ->  512

self.config.get_int("tile_x") 
# -> throws an error

In some modules it makes sense to revert changes made inside the module to not disturb modules coming afterwards (For example renderer modules should not change the state of the scene).

This often requried funcitonality can be easily done via the Utility.UndoAfterExecution() with-statement:

Example:

def run(self):
    bpy.context.scene.cycles.samples = 50
    
    with Utility.UndoAfterExecution():
        bpy.context.scene.cycles.samples = 320
        
        print(bpy.context.scene.cycles.samples)
        # Outputs: 320

    print(bpy.context.scene.cycles.samples)
    # Outputs: 50

All changes inside the with-block are undone which could also be undone via CTRL+Z inside blender.

To exchange information between modules the blender's custom properties are used (Blender allows to assign arbitrary information to scenes and objects). So modules can read out custom properties set by earlier modules and change their behaviour accordingly.

Example

• The module loader.SuncgLoader adds the custom property category_id to every object
• The module renderer.SegMapRenderer reads out this property and sets the segmentation color of every object correspondingly

In this way the renderer.SegMapRenderer can also be used without using the loader.SuncgLoader. The loader used instead just has to also set the category_id.