Telemetry filters on span events about applicationinsights-java HOT 19 CLOSED

@ivan-zaitsev I'm going to work on the sample app using OTEL extensions. if it doesn't work, i will discuss it with my team and then get back to you.

from applicationinsights-java.

Happy new year! Will get back to you soon. Sorry was out of office during the holidays. Thank you for trying :)

from applicationinsights-java.

@ivan-zaitsev yes, that's the part I need to fix in our java agent. But addTracerProviderCustomizer works.

from applicationinsights-java.

@ivan-zaitsev no, we're working on it.. It's a bit tricky though. I will make sure i test it and then confirm it's working in my sample.. then once this is released next week, you can use it. you have been super helpful assisting me getting to the bottom of this. I appreciate your patience.

from applicationinsights-java.

@ivan-zaitsev good news! it seems to be working. Azure-Samples/ApplicationInsights-Java-Samples#272 will generate this payload:

{
  "ver": 1,
  "name": "RemoteDependency",
  "time": "2024-01-16T22:55:00.503Z",
  "iKey": "<REDACTED>",
  "tags": {
    "ai.internal.sdkVersion": "java:3.5-SNAPSHOT",
    "ai.operation.id": "d1d540898b5bc6697637561c9dea25db",
    "ai.cloud.roleInstance": "<REDACTED>"
  },
  "data": {
    "baseType": "RemoteDependencyData",
    "baseData": {
      "ver": 2,
      "id": "018c905b75ea7b3f",
      "name": "mySpan",
      "type": "InProc",
      "duration": "00:00:10.297740",
      "success": true,
      "properties": {
        "controlledError": "true", **// this is added via the Telemetry Processor**
        "myCustomAttributeKey2": "myCustomAttributeValue2", **// added via the extension via the Sample above**
        "myCustomAttributeKey": "myCustomAttributeValue"
      }
    }
  }
}

You can reproduce it if you follow my readme .

It will be part of 3.5 GA release probably next week if no delay. I'll give you an update.

from applicationinsights-java.

@ivan-zaitsev just a heads up: 3.5 GA release is delayed to the week of Feb 21. We are waiting for a couple of fixes in upstream to be released in OpenTelemetry 2.1. Thank you for your patience.

from applicationinsights-java.

3.5.0 is live today.

Thanks, it means we can close this issue?

i can close it for you. i will update the sample too. thank you!

from applicationinsights-java.

@ivan-zaitsev span #2 above, current telemetry processor supports it.

{
    "type": "attribute",
    "include": {
        "matchType": "strict",
        "attributes": [
            {
                "key": "http.status_code",
                "value": 404
            }
        ]
    },
    "actions": [
        {
            "action": "insert",
            "key": "controlledError",
            "value": true
        }
    ],
    "id": "mark as not an error for status codes"
}

Are you asking for supporting filtering span.events via telemetry processors?

from applicationinsights-java.

Are you asking for supporting filtering span.events via telemetry processors?

Yes. Second span is there just as an example. The first one has actual events.

from applicationinsights-java.

From different thread @heyams said that it won`t be supported.

It's sad that this won't be supported, because currently it is not possible to filter on events or exceptions (they added as span events). If I would submit merge request, would you consider adding this feature?

I think it is pretty easy to add, because EventData already contains getAttributes() method, and everything is already in place to handle attributes, only new processor similar to SpanExporterWithAttributeProcessor should be added to process span.getEvents().stream().forEach(event -> event.getAttributes()) instead of span.getAttributes().

How i think it can be added:

1. New ProcessorType value = EVENT_ATTRIBUTE should be added. Here:

   ApplicationInsights-Java/agent/agent-tooling/src/main/java/com/microsoft/applicationinsights/agent/internal/configuration/Configuration.java

   Line 127 in e3eccab

   ATTRIBUTE("an attribute"),

2. New EventSpanExporterWithAttributeProcessor class should be added. Similar to this: SpanExporterWithAttributeProcessor to process attributes for each event from span.getEvents().

Then it would be possible to filter on event attributes like this (only new type is added, everything else is the same):

{
  "sampling" : {
    "percentage" : 100
  },
  "preview" : {
    "processors": [
      {
        "id" : "processors/attribute/strict",
        "type": "event-attribute",                  --- new type
        "include": {
          "matchType": "regexp",
          "attributes": [
            {
              "type": "string",
              "key": "exception.type",
              "value": "com.azure.storage.blob.models.BlobStorageException"
            },
            {
              "type": "string",
              "key": "exception.message",
              "value": "Status code 404"
            }
          ]
        },
        "actions": [
          {
            "action": "insert",
            "key": "strictKey",
            "value": "strictValue"
          }
        ]
      }
    ]
  }
}

It is possible to filter on "exception.type" and "exception.message" event attributes because they are added automatically by ImmutableExceptionEventData.

from applicationinsights-java.

Hello @heyams

Tried to create OTEL extension, but it didn't work. I can see that my custom addTracerProviderCustomizer and addSpanExporterCustomizer are executed before SecondEntryPoint, and tracer exporter processor AgentSpanExporter.export is executed after my custom SimpleSpanProcessor. But by some reason tracer exporter processor does not export those changes.

Code example:

resources/META-INF/services/io.opentelemetry.sdk.autoconfigure.spi.AutoConfigurationCustomizerProvider

otel.OpentelemetryExporterCustomizer

package otel;

import java.util.Collection;

import io.opentelemetry.api.common.Attributes;
import io.opentelemetry.api.common.AttributesBuilder;
import io.opentelemetry.sdk.autoconfigure.spi.AutoConfigurationCustomizer;
import io.opentelemetry.sdk.autoconfigure.spi.AutoConfigurationCustomizerProvider;
import io.opentelemetry.sdk.common.CompletableResultCode;
import io.opentelemetry.sdk.trace.SdkTracerProviderBuilder;
import io.opentelemetry.sdk.trace.data.DelegatingSpanData;
import io.opentelemetry.sdk.trace.data.SpanData;
import io.opentelemetry.sdk.trace.export.SimpleSpanProcessor;
import io.opentelemetry.sdk.trace.export.SpanExporter;

public class OpentelemetryExporterCustomizer implements AutoConfigurationCustomizerProvider {

    @Override
    public int order() {
        return -1;
    }

    @Override
    public void customize(AutoConfigurationCustomizer autoConfiguration) {
        autoConfiguration.addTracerProviderCustomizer((tracer, otelConfig) -> wrapTracerProvider(tracer));
        autoConfiguration.addSpanExporterCustomizer((exporter, otelConfig) -> wrapSpanExporter(exporter));
    }

    private SdkTracerProviderBuilder wrapTracerProvider(SdkTracerProviderBuilder tracerProviderBuilder) {
        SpanExporter spanExporter = wrapSpanExporter(SpanExporter.composite());
        tracerProviderBuilder.addSpanProcessor(SimpleSpanProcessor.create(spanExporter));
        return tracerProviderBuilder;
    }

    private SpanExporter wrapSpanExporter(SpanExporter spanExporter) {
        return new ExtendedSpanExporter(spanExporter);
    }

    private static class ExtendedSpanExporter implements SpanExporter {

        private final SpanExporter delegate;

        private ExtendedSpanExporter(SpanExporter delegate) {
            this.delegate = delegate;
        }

        @Override
        public CompletableResultCode export(Collection<SpanData> spans) {
            return delegate.export(spans.stream().map(this::process).toList());
        }

        @Override
        public CompletableResultCode flush() {
            return delegate.flush();
        }

        @Override
        public CompletableResultCode shutdown() {
            return delegate.shutdown();
        }

        private SpanData process(SpanData span) {
            AttributesBuilder builder = Attributes.builder();
            builder.putAll(span.getAttributes());
            builder.put("test-key", "test-value");

            SpanData result = new ExtendedSpanData(span, builder.build());
            System.out.println(result);
            return result;
        }

    }

    public static class ExtendedSpanData extends DelegatingSpanData {

        private final Attributes attributes;

        public ExtendedSpanData(SpanData delegate, Attributes attributes) {
            super(delegate);
            this.attributes = attributes;
        }

        @Override
        public Attributes getAttributes() {
            return attributes;
        }

    }

}

Run app

java -jar app.jar -Dotel.javaagent.extensions=otel-extension.jar

from applicationinsights-java.

Figured it out. Usual otel extension creation guidelines with span exporters are not applicable here. ApplicationInsights-Java uses span processor as last sink for data exporting. Exporters from addSpanExporterCustomizer are combined together and being used in addTracerProviderCustomizer as separate processor. Processors don't change span data by chaining and delegating like exporters do, so one processor is not affected by other one, because of that any modification of span data in exporters from addSpanExporterCustomizer will not simply work for other processors. As last export of data is performed by different processor the exporters from addSpanExporterCustomizer will not be applied. Therefore, to make it work the modification in span processors should be made.

But this way it is not possible to do any conditions on spans, because:

1. onStart is executed right away when span is started, so some data is missing as it can be added later after that. onEnd has only ReadableSpan which is not possible to change.
2. SimpleSpanProcessor is executed before ApplicationInsights-Java processors and exporters, therefore spans are missing some data from there.

Code example:

package otel;

import io.opentelemetry.context.Context;
import io.opentelemetry.sdk.autoconfigure.spi.AutoConfigurationCustomizer;
import io.opentelemetry.sdk.autoconfigure.spi.AutoConfigurationCustomizerProvider;
import io.opentelemetry.sdk.trace.ReadWriteSpan;
import io.opentelemetry.sdk.trace.ReadableSpan;
import io.opentelemetry.sdk.trace.SdkTracerProviderBuilder;
import io.opentelemetry.sdk.trace.SpanProcessor;

public class OpentelemetryExporterCustomizer implements AutoConfigurationCustomizerProvider {

    @Override
    public int order() {
        return -1;
    }

    @Override
    public void customize(AutoConfigurationCustomizer autoConfiguration) {
        autoConfiguration.addTracerProviderCustomizer((tracer, otelConfig) -> wrapTracerProvider(tracer));
    }

    private SdkTracerProviderBuilder wrapTracerProvider(SdkTracerProviderBuilder tracerProviderBuilder) {
        tracerProviderBuilder.addSpanProcessor(new SimpleSpanProcessor());
        return tracerProviderBuilder;
    }

    private static class SimpleSpanProcessor implements SpanProcessor {

        @Override
        public boolean isStartRequired() {
            return true;
        }

        @Override
        public void onStart(Context parentContext, ReadWriteSpan span) {
            span.setAttribute("test-key", "test-value");
        }

        @Override
        public boolean isEndRequired() {
            return false;
        }

        @Override
        public void onEnd(ReadableSpan span) {
        }

    }

}

Either I missed something, or it seems pretty limited that it is not possible to add span processors/exporters which would modify span for the ApplicationInsights-Java processing/exporting flow.

I think if it a custom exporter as a delegate for existing one would be added here then it should be possible to check/modify spans right away before exporting.

from applicationinsights-java.

@ivan-zaitsev I tried something similar here.

My extension code gets applied, but it requires some code change in the Java agent in order to make it full-functional.
Please stay tune and I will try to work on the java agent to make this sample works for your use case.

from applicationinsights-java.

Hello @heyams

Thank you for your help.
Interesting, everything seems similar to my first example. But in my case the exporters were executed but it was not applied when I checked logs in applications insights. I think that all SpanExporters which are added by addSpanExporterCustomizer have no effect on the actual http exporter which is configured as span processor - here.

from applicationinsights-java.

Hello @heyams, any update on this? Seems that there no other way at the moment to add any custom span processing on top of this java agent.

from applicationinsights-java.

@ivan-zaitsev yes, #3493 is intended to support OpenTelemetry Extension. I will test it in my sample and get back to you soon.

from applicationinsights-java.

@heyams Thanks! Sorry for pinging you too often, I just thought it was abandoned :)

from applicationinsights-java.

3.5.0 is live today.

from applicationinsights-java.

3.5.0 is live today.

Thanks, it means we can close this issue?

from applicationinsights-java.