It'd be lovely to know things like the length of the libhoney queue. Consider stealing the metrics implementation from samproxy (https://github.com/honeycombio/samproxy/tree/master/metrics) and use it to give people running the OT proxy some insight into how it's functioning.

When processing large numbers of spans, it is possible to overflow the libhoney queue. The default batch size of 50 is fine for low traffic, but it would be nice to be able to tune that and other libhoney parameters (number of concurrent batches, batch timeout, queue size, etc.).

Consider adding a method of configuring the underlying libhoney with all the knobs it has.

Hi All,

I am trying to get up and running sending traces from an existing Spring Cloud Sleuth instrumented Spring Boot application to honeycomb-opentracing-proxy and it is failing with the following error in the proxy console:

time="2019-10-25T02:51:55Z" level=info msg="error unmarshaling spans" error="json: cannot unmarshal bool into Go struct field binaryAnnotation.value of type string" type=application/json

Spring Boot Version: 2.1.3.RELEASE
Spring Cloud Sleuth Version: 2.1.1.RELEASE

I am running the open tracing proxy with the following docker command:

docker run -p 9411:9411 honeycombio/honeycomb-opentracing-proxy -k $KEY -d $DATA_SET

From reading the documentation the honeycomb-opentracing-proxy only supports v1 of the JSON API so I have explicitly set that in spring cloud config as this appears to default to v2.

application.properties

spring.zipkin.sender.type=web
spring.zipkin.base-url=http://localhost:9411
spring.zipkin.enabled=true
spring.zipkin.encoder=json_v1

Any help would be greatly appreciated. Also raised on Stack Overflow here

Hi there,

I'm adding tracing support to a react static site generator called Gatsby. We were going to instrument the codebase with open tracing so that we could plugin many tracing backends. I was happy to see that honeycomb supports open tracing via this proxy.

I'm a little confused though. This proxy is a zipkin proxy, not an open tracing proxy (since open tracing is simply a language level spec). And Zipkin is no longer an open tracing supported platform. In addition, there are no official javascript zipkin open tracing libraries.

So, what is the recommendation on how to send traces to honeycomb from javascript? Is it to ignore open tracing and just use zipkin-js? If not, are there plans for a future honeycomb open tracing js library?

We'd really love to enable honeycomb as a tracing backend :)

Currently the proxy handles OT log events by adding the event's attributes to the parent span. This is semantically incorrect as a log event represents a thing that happened at a specific instance (an error, say), vs a span which represents something with a duration. This implementation loses the timestamp of the log event(s)

Further, if a single span has more than one log event with the same attributes, only the values from the last such event will be set on the span.

Ideally log events would show up as distinct events in HC, with a parent span and timestamp, but no duration. These events should appear in the trace UI, although it's not clear to me how they should be displayed, as they have no duration...

In some cases it might be useful to send events to the proxy from a browser-based application. The browser would check CORS headers before sending, so the proxy would need to set Access-Control-Allow-Origin to the customer's expected origin (or something permissive like *). Would be nice to allow proxy users to configure the value of this header to support this use case.

Then at least theoretically you could intermingle Beelines and OT Proxy in the same dataset.

The current way of specifying a sample rate is a bit confusing (to me), and imprecise.

My understanding is --samplerate=n means 1/n traces will be forwarded, and I think n must be an integer. This means, expressed as a probability, the least aggressive sample rate possible is 50% (1/2). Going from 100% to 50% is a big jump. I'd like to be able to specify the sample rate as a probability between 0 and 100 (or a float between 0 and 1). This would let me have a sample rate of, say, 75%.

If there is actual demand for it.

Could this agent be extended to allow dynamic sampling of traces, maybe by adding a sample rate as meta-data to root spans? Would this be considered, feature-wise?

Currently, we simply discard any annotations and client addr / server addr binary annotations on spans. In the future we could do something smarter.

In PR #42, support for Zipkin's V2 API was added – thanks again for that, @fbcbarbosa and @nathanleclaire!

However, I noticed that the Jaeger instance that we're currently running downstream of the honeycomb proxy started storing my services' traces under unknown-service-name as soon as I switched our Opentracing reporters to use the Zipkin V2 API endpoint and JSON format.

After some digging, I found out that the reason for this seems to be that the Honeycomb proxy sends these V2 JSON payloads to the V1 API endpoint, because that's what's hardcoded in main.go. That totally confuses Jaeger.

I'm happy to send in a PR for this myself, I've mainly created this issue to keep track of the problem.

In particular, if you're using a jaeger client with the zipkin wire format, the tracer configuration is somewhat obtuse. Need to document this clearly. Also need to document how to send a test span, debug whether ingest is working, etc.

If an incoming JSON span omits the timestamp field, it gets set to 0 when getting serialized into the ZipkinJSONSpan type. The time conversion from the expected microseconds since unix epoch then happens on that zero value, which returns 1970 as a time.Time object on the Span that will be sent to Honeycomb.

It would be nice if there was a zero check against the timestamp to cancel the whole thing and, in the face of no incoming timestamp, either assign now or pass it on to Honeycomb with no timestamp (where the API will assign now).

I'm setting up the opentracing proxy for a little personal project, ingesting spans from envoy; unfortunately, the most interesting spans get dropped, because they can't be decoded due to this envoy bug: envoyproxy/envoy#9341 (fixed in envoyproxy/envoy#9432):

time="2020-04-16T14:35:01Z" level=info msg="error unmarshaling spans" error="json: cannot unmarshal number 1.58704770013901e+15 into Go struct field ZipkinJSONSpan.timestamp of type int64" type=application/json

This could ~easily be avoided by accepting a data encoding that has an unmistakable numeric representation (protobuf!) - and envoy supports sending v2 zipkin spans as protobuf in newer versions.

Support for that would be really neat, as I imagine it'll also ~guarantee forwards-compatibility of the schema...