- Consumes a Kafka message for each observation to be imported.
- Query the dp-import-api for the dimension option database id's
- Insert the observation into the DB
- Run
brew install kafka
- Run
brew install neo4j
- Configure neo4j, edit /usr/local/Cellar/neo4j/3.2.0/libexec/conf/neo4j.conf
- Set
dbms.security.auth_enabled=false
- Run
brew services restart neo4j
Scripts for updating and debugging Kafka can be found here(dp-data-tools)
Environment variable | Default | Description |
---|---|---|
BIND_ADDR | :21700 | The port to bind to |
KAFKA_ADDR | http://localhost:9092 | The address of the Kafka instance |
OBSERVATION_CONSUMER_GROUP | dp-observation-importer | The Kafka consumer group to consume observation extracted events from |
OBSERVATION_CONSUMER_TOPIC | observation-extracted | The Kafka topic to consume observation extracted events from |
DATASET_API_URL | http://localhost:21800 | The URL of the dataset API |
BATCH_SIZE | 1000 | The number of messages to process in each batch if the time out has not been reached |
BATCH_WAIT_TIME | 200ms | The duration to wait before processing a partially full batch of messages (time.Duration) |
ERROR_PRODUCER_TOPIC | report-events | The Kafka topic to send the error messages to |
RESULT_PRODUCER_TOPIC | import-observations-inserted | The Kafka topic to send the observations inserted messages to |
KAFKA_MAX_BYTES | 0 | |
CACHE_TTL | 60m | The amount of time to wait before clearing the cache (time.Duration) |
GRACEFUL_SHUTDOWN_TIMEOUT | 10s | The shutdown timeout (time.Duration) |
SERVICE_AUTH_TOKEN | AA78C45F-DD64-4631-BED9-FEAE29200620 | The service authorization token |
ZEBEDEE_URL | http://localhost:8082 | The host name for Zebedee |
HEALTHCHECK_INTERVAL | 30s | The period of time between health checks |
HEALTHCHECK_CRITICAL_TIMEOUT | 90s | The period of time after which failing checks will result in critical global check status |
GRAPH_DRIVER_TYPE | neo4j | String identifier for the implementation to be used (e.g. 'neo4j', 'neptune' or 'mock') |
ENABLE_GET_GRAPH_DIMENSION_ID | true | Use store ID's for Neptune search |
See CONTRIBUTING for details.
Copyright © 2016-2021, Office for National Statistics (https://www.ons.gov.uk)
Released under MIT license, see LICENSE for details.