Adapter used to convert OTEL batches to/from OTEL Arrow batches in both directions.

This package is still experimental and subject to change. It is currently used by an experimental OTLP/Arrow gRPC exporter and receiver.

Other important links:

• Project Roadmap.
• Project Board describing the current state of the project.
• Arrow schemas used by this package.
• The underlying OTEP describing the rationale, specifications and different phases of this project.

To (re)generate the ArrowStreamService gRPC service, you need to install the protoc compiler and the protoc-gen-grpc plugin.

go install google.golang.org/protobuf/cmd/[email protected]
go install google.golang.org/grpc/cmd/[email protected]
export PATH="$PATH:$(go env GOPATH)/bin"
cd ./proto
./generate.sh

Once the *.pb.go files are generated, you need to replace the content of the api/collector/arrow/v1 directory by the generated files present in the ./proto/api/collector/arrow/v1 directory.

This tool generates synthetic metrics and store them as protobuf binary files.

This tool generates synthetic metrics and store them as protobuf binary files.

This tool generates synthetic metrics and store them as protobuf binary files.

These command line tools are used to compare the performance of OTLP and OTLP Arrow entity representations. Each tool takes as input a set of OTLP entities encoded as protobuf binary files and produces an evaluation report and a set of data files (CSV format).

This tool compares the performance of OTLP and OTLP Arrow representations for metrics.

$ go run tools/trace_benchmark/main.go <INPUT FILES>

The input files must be one or more OTLP traces (ExportTraceServiceRequest) encoded as protobuf binary files.

This tool compares the performance of OTLP and OTLP Arrow representations for metrics.

This tool compares the performance of OTLP and OTLP Arrow representations for metrics.

The integration of this package with the OpenTelemetry Collector is done in the following experimental repository:

• experimental-arrow-collector

Below some utility functions in package pkg/otel/traces than can be used to convert OTLP traces to/from OTLP Arrow traces:

• OtlpArrowProducer takes OTLP Traces and produces OTLP Arrow Traces.
• OtlpProducer takes OTLP Arrow Traces and produces OTLP Traces.

Arrow Intermediate Representation (framework to convert row-oriented structured data to Arrow columnar data)

• Values (supported types: bool, i[8|16|32|64], u[8|16|32|64], f[32|64], string, binary, list, struct)
• Fields
• Record
• Record Builder
• Record Repository
• Generate Arrow records
  • Scalar values
  • Struct values
  • List values (except list of list)
• Optimizations
  • Dictionary encoding for string fields (uint8 and uint16 indices)
  • Dictionary encoding for binary fields (uint8 and uint16 indices)
  • Multi-field sorting (string field)
  • Multi-field sorting (binary field)

Transform OTLP entities into OTLP Arrow entities This capability will be used to implement a receiver into the existing collector (phase 1)

• General.
  • Complex attributes
  • Complex body
  • Schema URL
  • Description
  • Unit
• OTLP metrics --> OTLP_ARROW events
  • Gauge
  • Sum
  • Summary
  • Histogram
  • Exponential histogram
  • Univariate metrics to multivariate metrics
  • Aggregation temporality
  • Exemplar
• OTLP logs --> OTLP_ARROW events
  • Logs
• OTLP trace --> OTLP_ARROW events
  • Trace
  • Links
  • Events

Transform OTLP Arrow entities into OTLP entities This capability will be used to implement an exporter into the existing collector (phase 1).

• General
  • Complex attributes
  • Complex body
  • Schema URL
  • Description
  • Unit
• OTLP_ARROW events --> OTLP metrics
  • Gauge
  • Sum
  • Summary
  • Histogram
  • Exponential histogram
  • Univariate metrics to multivariate metrics
  • Aggregation temporality
  • Exemplar
• OTLP_ARROW events --> OTLP logs
  • Logs
• OTLP_ARROW events --> OTLP trace
  • Trace
  • Links
  • Events

Protocol

• BatchArrowRecords proto
• Arrow Stream service
• BatchArrowRecords producer
• BatchArrowRecords consumer

Benchmarking tools

• Synthetic data generator
  • ExportMetricsServiceRequest (except for histograms and summary)
  • ExportLogsServiceRequest
  • ExportTraceServiceRequest
• Framework to compare OTLP and OTLP_ARROW performances (i.e. size and time)
  • General framework
  • Compression algorithms (lz4 and zstd)
  • Console output
  • Export CSV
• OTLP batch creation + serialization + compression + decompression + deserialization
• OTLP_ARROW batch creation + serialization + compression + decompression + deserialization
• Synthetic data generators
  • logs_gen
  • metrics_gen
  • traces_gen
• Benchmarking tools
  • logs_benchmark
  • metrics_benchmark [WIP define a way to specify the multivariate metrics configuration]
  • traces_benchmark
• Assertions to check the correctness of the OTLP_ARROW representation
  • Metrics
  • Logs
  • Trace

Performance

• Performance and memory optimizations
• Check memory leaks (e.g. Arrow related memory leaks)

CI

• GitHub Actions to build, test, check at every commit.

Integration

• Integration with Open Telemetry Collector.

Documentation

• Update OTEP 0156.

Feedback to implement

• @jmacd's feedback
• @atoulme's feedback

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change. For more information, please read CONTRIBUTING.

OTEL Arrow Adapter is licensed under Apache 2.0.