Additional Memory Consumption of 15-20GB 30-90% due to Performance Monitoring about sentry-dotnet HOT 5 CLOSED

I recommend measuring this (taking a memory heap dump) since guessing what might be the culprit can be a risky/time consuming game

from sentry-dotnet.

One thought about this is that we clone the scope every time a new span or transaction is created. It would be interesting to know how much of the memory that Sentry is allocating is in the form of Scopes (vs TransactionTracers/SpanTracers).

Assuming all the cloned scopes are significant though, there might be a way to track the differences between child scopes and parent scopes rather than creating entirely new long lived clones.

So one solution, for example, rather than setting properties directly on the scopes of child spans, we could associate one or more actions with them that could be used to determine the effective scope: List<Action<Scope>> _scopeModifications. The actual/effective scope for each span could be retrieved by taking a clone of the root scope and applying any scope modifications (similar to playing back the events in an EventSourcing architecture)... so you'd grab an ephemeral instance of the Span.Scope only when you needed to access it (probably it would have the lifetime of an individual method call... rather than staying in memory for as long as the transaction was running).

This would reduce memory consumption... it would come at the cost of speed though (as every time you needed to access the scope, you'd have to take a fresh clone of the root scope and apply the scope modifications).

Potentially we could give users the ability to choose whether they wantMemoryOptimizedTracing or CPUOptimizedTracing... depending on which was the more important constraint in their particular business context.

from sentry-dotnet.

• BenchmarkDotNet (which we're using for benchmarking)
  • BenchmarkSwitcher

from sentry-dotnet.

When profiling applications I've noticed that, unsurprisingly, where memory is being consumed depends very much on what th application is doing. So if, for example, I get artillery to call this endpoint a few hundred times, most of the memory is consumed by allocations of breadcrumb (and it's various composite properties):

                    endpoints.MapGet("/breadcrumbs", async context =>
                    {
                        void BreadcrumbCountdown(int countdown)
                        {
                            SentrySdk.AddBreadcrumb(countdown.ToString());
                            if (--countdown == 0)
                            {
                                return;
                            }

                            BreadcrumbCountdown(countdown);
                        }

                        BreadcrumbCountdown(100);
                        await context.Response.WriteAsync("Hello Breadcrumbs!");
                    });

On the other hand, if I have artillery call this, it's all about the scope:

                    endpoints.MapGet("/scopes", async context =>
                    {
                        void ScopeCountdown(int countdown)
                        {
                            using (SentrySdk.PushScope())
                            {
                                SentrySdk.ConfigureScope(scope =>
                                {
                                    scope.SetExtra("countdown", countdown);
                                });

                                if (--countdown == 0)
                                {
                                    return;
                                }

                                ScopeCountdown(countdown);
                            }
                        }

                        ScopeCountdown(100);
                        await context.Response.WriteAsync("Hello Scopes!");
                    });

I'm wondering if it's possible to get some memory snapshots from real world applications that aren't so biased. It'd be really illuminating to know what was taking up the most memory in the customer application that triggered this issue, for example... and that will depend entirely on what their application is doing and how they've instrumented it with Sentry. Have they used lots of breadcrumbs? Do they have deep scope trees? Are they heavily using Tags? It could be any number of things, but it feels like we're stabbing in the dark a bit without data from real world applications.

from sentry-dotnet.

Considering this closed until proven otherwise.
Optimizations that happened:

• #2642
• #2641
• #2636
• #2632
• #2619

from sentry-dotnet.