A GeoNode app that implements a flow to upload/import data files.
The modular logic adapts to different file types, and can be extended by implementing custom handlers.

geonode-importer has been created as a GeoNode 4.0 app to handle some formats that at the time were unsupported by GeoNode. It has then been extended to include all previously handled formats.
In GeoNode 4.1 geonode-importer replaced the previous importer logic.

• ESRI Shapefile - Vector
• GeoPackage - Vector
• GeoJSON - Vector
• KML - Vector
• CSV - Vector
• GeoTiff - Raster

IMPORTANT: At the moment the importer doesn't support overwriting/skipping existing layers from the UI. Every upload will create a new dataset. Overwriting a layer (overwrite_existing_layer) and skipping an already existing layer (skip_existing_layers) is supported through the API. Refer to the API documentation for more details and exmplaes.

• Features in the same table must have the same geometry type. Mixed geometry tpyes are not supported, therefore GEOMETRY columns are not accepted
• The XML file and the SLD file uploaded along with the GPKG are ignored
• The number of layers in a GPKG must be lower than the max_parallel_upload configuration value

• The filename should not contain dots, for example "invalid.file.name.geojson" -> "valid_file_name.geojson"

• The CSV colum accepted for lat/long CSVs (POINTS) are the followings:
  • lat, latitude, y
  • long, longitude, x
• For any other geometry type the following columns are accepted:
  • geom, geometry, the_geom, wkt_geom

Starting from GeoNode 4.1.0 the new importer is installed and configured by default.

The following documentation is only meant to report what is automatically done under the hood.

The importer relies on the gdal utilities to perform format conversions and manipulations.

You need to install the gdal-bin package in your system, be it a base system or a docker environment; in the latter case, make sure it is installed in the celery and in the django services.
In a ubuntu/debian system you can install gdal-bin with the command:

apt install gdal-bin

You need at least version 3.2.2 (this is the version that has been tested).
To check your version please run either:

$ gdalinfo --version
GDAL 3.3.2, released 2021/09/01   

or

$ ogrinfo --version
GDAL 3.3.2, released 2021/09/01

To install gdal-bin on other platforms please refer to https://gdal.org/download.html.

Make sure you activated the virtualenv if you are using one.

pip install -e git+https://github.com/cesar-benjamin/geonode-importer.git@master#egg=geonode_importer

The following settings in GeoNode's settings.py drive the importer functionality:

INSTALLED_APPS += ('dynamic_models', 'importer', 'importer.handlers',)

CELERY_TASK_QUEUES += (
    Queue('importer.import_orchestrator', GEONODE_EXCHANGE, routing_key='importer.import_orchestrator'),
    Queue('importer.import_resource', GEONODE_EXCHANGE, routing_key='importer.import_resource', max_priority=8),
    Queue('importer.publish_resource', GEONODE_EXCHANGE, routing_key='importer.publish_resource', max_priority=8),
    Queue('importer.create_geonode_resource', GEONODE_EXCHANGE, routing_key='importer.create_geonode_resource', max_priority=8),
    Queue('importer.import_with_ogr2ogr', GEONODE_EXCHANGE, routing_key='importer.import_with_ogr2ogr', max_priority=10),
    Queue('importer.import_next_step', GEONODE_EXCHANGE, routing_key='importer.import_next_step', max_priority=3),
    Queue('importer.create_dynamic_structure', GEONODE_EXCHANGE, routing_key='importer.create_dynamic_structure', max_priority=10),
    Queue('importer.copy_geonode_resource', GEONODE_EXCHANGE, routing_key='importer.copy_geonode_resource', max_priority=0),
    Queue('importer.copy_dynamic_model', GEONODE_EXCHANGE, routing_key='importer.copy_dynamic_model'),
    Queue('importer.copy_geonode_data_table', GEONODE_EXCHANGE, routing_key='importer.copy_geonode_data_table'),
    Queue('importer.copy_raster_file', GEONODE_EXCHANGE, routing_key='importer.copy_raster_file'),
    Queue('importer.rollback', GEONODE_EXCHANGE, routing_key='importer.rollback'),
)

DATABASE_ROUTERS = ["importer.db_router.DatastoreRouter"]

SIZE_RESTRICTED_FILE_UPLOAD_ELEGIBLE_URL_NAMES += ('importer_upload',)

IMPORTER_HANDLERS = os.getenv('IMPORTER_HANDLERS', [
    'importer.handlers.gpkg.handler.GPKGFileHandler',
    'importer.handlers.geojson.handler.GeoJsonFileHandler',
    'importer.handlers.shapefile.handler.ShapeFileHandler',
    'importer.handlers.kml.handler.KMLFileHandler',
    'importer.handlers.csv.handler.CSVFileHandler',
    'importer.handlers.geotiff.handler.GeoTiffFileHandler'
])

In case of a local environment, Geoserver and Geonode should be able to reach the default MEDIA_ROOT.

If some permission is missing, please change the FILE_UPLOAD_DIRECTORY_PERMISSIONS to make the folder accessible to both

Run migrations:

python manage.py migrate
python manage.py migrate --database datastore

To change the task rate limit, please update the following env variables:

IMPORTER_GLOBAL_RATE_LIMIT= # default 5
IMPORTER_PUBLISHING_RATE_LIMIT= # default 5
IMPORTER_RESOURCE_CREATION_RATE_LIMIT= # default 10
IMPORTER_RESOURCE_COPY_RATE_LIMIT = # default 10

# https://gdal.org/drivers/vector/pg.html#configuration-options
DISABLE_PG_COPY_OGR2OGR= # If TRUE disable the usage of COPY during the import in postgres. Disable it can affect the performance 

The importer will return different error codes according to the encountered error.
Here a description of the various codes:

By default, we use the LAUNDER option in ogr2ogr to convert a list of special chars to make them compatible with PostgreSQL.

Along with this, also the layer name during the dataset import is converted to make it aligned with ogr2ogr.

In detail the list is the following: