mappy / mapnik Goto Github PK

The branch https://github.com/mappy/mapnik/tree/parallel_queries contains our work for adding parallel queries to Posgtis, which can serve as a basis for general parallelisation of datasources.

Scheduling the rendering for parallelisation

You might notice that we had to make important changes on the rendering loop :
we've split fonction apply_to_layer in two : prepare_datasource_query and render_styles, so that we could split the main rendering loop in apply in two :
first loop calls prepare_datasource_query which collects iterators on features, and gives it back to render_styles throug a rendering_material structure.

Thus, we'll be able to launch asynchronous fetching of the features on the first loop. For the moment, it has to be done inside the plugin itself, but this should be the place to launch a thread. For the moment, we've only developped a change in Postgis plugin that takes avdantage of that by sending an asynchronous query when calling method features(), and waiting for database response when iterating on the features begin.

To do this, we took the liberty to replace the memory_datasource by a featureset_buffer wich is more suitable for our use.

Launching asynchronous queries on Postgres

We use native postgres function PQsendQuery when the rendering engine requires a features iterator through plugin->features() method, and when the rendering engine tries to read the first feature, we use PQgetResult which waits until some features are available.

By the way, we had to introduce the notion of "rendering context" on a plugin to schedule the several database queries for the same map, without concurrency issues. This induces changes on the plugin interface that are transparent for the current implementations. This option was far more pragmatic than creating a pool of layers to render.