Broken trace when recovering persistent actor in sharded cluster about phobos-issues HOT 6 CLOSED

Hi Gareth,

I have an open issue for looking into Akka.Persistence in general with Phobos - it looks to us like we don't tie the Persist before and after in together correctly. Do you have a timeframe you need this done by? We were going to work on that once we have our OTel tracing beta all the way out the door this month.

from phobos-issues.

Hi Aaron,

I wasn't sure if it was the same issue or a different one (albeit related), so I raised it separately. While we were waiting for an update, we put in a workaround for the other persistence issue which is to wrap the call to Persist and short circuit the trace manually as follows:

protected void PersistWithTracing<TEvent>(TEvent @event, Action<TEvent> handler)
{
    if (@event is null)
        throw new ArgumentNullException(nameof(@event));

    var parentSpan = Context.GetInstrumentation().ActiveSpan;

    Persist(@event, _ =>
    {
        if (handler is null)
            return;

        var phobosActorContext = Context.GetInstrumentation();

        var spanBuilder = phobosActorContext.Tracer
            .BuildSpan($"custom.akka.persistence.handler_{@event.GetType().Name}")
            .AsChildOf(parentSpan);

        // Get tags using reflection
        var tags = GetTags(phobosActorContext.ActiveSpan);

        foreach (var tag in tags)
            spanBuilder.WithTag(tag.Key, tag.Value);

        using (spanBuilder.StartActive())
            handler(@event);
    });
}

In some cases this is working, but in other it's not and the trace in the description is one example where it isn't. If this is all related though I'm happy for you to roll the two issues up and close one; however you best want to track it.

As to a timeframe, I'll discuss it with the team and get back to you as soon as I have some info on that.

Many thanks!

from phobos-issues.

Progress update on our end: we've added some prototype instrumentation to capture all actor startup / restart activity prior to processing messages. This is a necessary ingredient for being able to trace recoveries from the point of view of the PersistentActor. The next stage is making sure that the persistence plugins themselves can properly correlate the activity on their end - this might require an update to Akka.Persistence itself to make the plugins more traceable, but we'll see.

from phobos-issues.

Thanks Aaron; appreciate the update!

from phobos-issues.

Whoops, not closed yet. Just merged in an Akka.NET PR that is needed to implement this in Phobos.

from phobos-issues.

Resolved via Phobos 1.4.0

from phobos-issues.