Adds the ability to send Kafka message from nginx, using librdkafka. The code of this module was based on nginx-json-log

To link statically against nginx, cd to nginx source directory and execute:

./configure --add-module=/path/to/nginx-kafka-log-module

To compile as a dynamic module (nginx 1.9.11+), use:

./configure --add-dynamic-module=/path/to/nginx-kafka-log-module

In this case, the load_module directive should be used in nginx.conf to load the module.

• syntax: kafka_log kafka:topic body message_id
• default: none
• context: location

Enables Kafka logging on the specified location. topic, body and message_id can contain nginx variables.

• syntax: kafka_log_kafka_client_id client_id
• default: nginx
• context: main

Sets the Kafka client id.

• syntax: kafka_log_kafka_debug context_list
• default: nginx
• context: main

Sets the list of debug contexts, use comma to separate multiple values.

• syntax: kafka_log_kafka_brokers broker_list
• default: nginx
• context: main

Sets the list of bootstrap Kafka brokers, use comma to separate multiple values.

• syntax: kafka_log_kafka_compression type
• default: snappy
• context: main

Sets the message compression format.

• syntax: kafka_log_kafka_partition id
• default: auto
• context: main

Sets the topic partition id, the default is to automatically assign a partition according to the message id.

• syntax: kafka_log_kafka_log_level level
• default: 6
• context: main

Sets the logging level of librdkafka (syslog(3) levels)

• syntax: kafka_log_kafka_max_retries retries
• default: 0
• context: main

Defines how many times to retry sending a failed MessageSet. Note: retrying may cause reordering.

• syntax: kafka_log_kafka_buffer_max_messages count
• default: 100000
• context: main

Maximum number of messages allowed on the producer queue.

• syntax: kafka_log_kafka_backoff_ms millis
• default: 10
• context: main

The backoff time in milliseconds before retrying a message send.

http {
	kafka_log_kafka_brokers 192.168.0.1:9092,192.168.0.2:9092;
	
	server {
		location /log/ {
			kafka_log kafka:sometopic $args $arg_msgId;
			return 200;
		}
	}

Hitting http://domain/log/ will send a message with all query parameters to the Kafka topic sometopic.

All code in this project is released under the BSD license unless a different license for a particular library is specified in the applicable library path.

Copyright © 2016 Paulo Pacheco. Copyright © Kaltura Inc. All rights reserved.