Comments (10)
alexeyzimarev
commented on September 21, 2024
1
The last alpha will double-check that the options argument is not null, just to confirm (or not) that it is an issue.
from eventuous.
diegosasw
commented on September 21, 2024
I've made each test to use its own MongoDb Database and EventStoreDb Streamname to see if the problem had anything to do with shared resources (e.g: the checkpoint collection in a database, or similar), but the problem remains (see the repo that reproduces the problem)
I still get the same Object reference not set to an instance of an object. at Eventuous.EventStore.Subscriptions.EventStoreCatchUpSubscriptionBase1..ctor(EventStoreClient eventStoreClient, T options, ICheckpointStore checkpointStore, ConsumePipe consumePipe, ILoggerFactory loggerFactory)`
at zero, one or more tests randomly.
from eventuous.
diegosasw
commented on September 21, 2024
It's definitely something related to the services.AddSubscription<AllStreamSubscription, AllStreamSubscriptionOptions>(...), because the problem is not reproduced when not adding any subscription.
Also, assigning a different subscription Id to each subscription (i.e: to each TestServer instantiation/launching) doesn't make any difference.
If I de-register all the IHostedService immediately after AddSubscription, then the errors don't happen.
So, probably it's something related to the fact that the SubscriptionRegistrationExtensions has a delayed instantiation of SubscriptionHostedService?
If I look at Eventuous implementation, the IHostedService is not registered as a specific implementation but as a delegate. Could that be the cause of a race condition?
//
return services
.AddSubscriptionBuilder(builder)
.AddSingleton(sp => GetBuilder(sp).ResolveSubscription(sp))
.AddSingleton<IHostedService>(
sp =>
new SubscriptionHostedService(
GetBuilder(sp).ResolveSubscription(sp),
sp.GetService<ISubscriptionHealth>(),
sp.GetService<ILoggerFactory>()
)
);
Is the error due to the ILoggerFactory being null in Eventuous.EventStore.Subscriptions.AllStreamSubscription..ctor(EventStoreClient eventStoreClient, AllStreamSubscriptionOptions options, ICheckpointStore checkpointStore, ConsumePipe consumePipe, ILoggerFactory loggerFactory)?
Not sure which one is the parameter causing the NullReferenceException because the logs don't say.
from eventuous.
alexeyzimarev
commented on September 21, 2024
Subscriptions should tolerate the logger factory being null, it's a nullable argument, and there are coalescing everywhere (supposedly).
from eventuous.
alexeyzimarev
commented on September 21, 2024
Can't you set a breakpoint in the constructor base call to see where it crashes?
from eventuous.
diegosasw
commented on September 21, 2024
It never crashes when debugging all so it won't hit breakpoints. But I found that the problem happens when resolving the IHostedService, which is registered as a delegate in SubscriptionRegistrationExtensions.cs
It's the messageSubscrition resolving what's causing it. Something to do with reflection.
var messageSubscription = GetBuilder(sp).ResolveSubscription(sp);
I could capture the exception making this changes at AddSubscription<T, TOptions>(...)
return services
.AddSubscriptionBuilder(builder)
.AddSingleton(sp => GetBuilder(sp).ResolveSubscription(sp))
.AddSingleton<IHostedService>(
sp => {
try {
var messageSubscription = GetBuilder(sp).ResolveSubscription(sp); // here it crashes SOMETIMES when running in parallel
var subscriptionHealth = sp.GetService<ISubscriptionHealth>();
var loggerFactory = sp.GetService<ILoggerFactory>();
var hostedService =
new SubscriptionHostedService(
messageSubscription,
subscriptionHealth,
loggerFactory);
return hostedService;
}
catch (Exception exception) {
throw new Exception("Gotcha", exception);
}
}
);
now gives more details on what's going on. It seems the root cause is losing the exception trace because it's happening when the DependencyInjection tries to resolve the IHostedService.
Here's the full log
System.Exception: Gotcha
System.Exception
Gotcha
at Microsoft.Extensions.DependencyInjection.SubscriptionRegistrationExtensions.<>c__DisplayClass0_0`2.<AddSubscription>b__1(IServiceProvider sp) in /media/diegosasw/data/src/sandbox/eventuous/src/Core/src/Eventuous.Subscriptions/Registrations/SubscriptionRegistrationExtensions.cs:line 56
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite callSite, RuntimeResolverContext context)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitIEnumerable(IEnumerableCallSite enumerableCallSite, RuntimeResolverContext context)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite callSite, RuntimeResolverContext context)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.Resolve(ServiceCallSite callSite, ServiceProviderEngineScope scope)
at Microsoft.Extensions.DependencyInjection.ServiceProvider.CreateServiceAccessor(Type serviceType)
at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(Type serviceType, ServiceProviderEngineScope serviceProviderEngineScope)
at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
at Program.<Main>$(String[] args) in /media/diegosasw/data/src/sandbox/eventuous/src/Sample/src/Modular.Api/Program.cs:line 60
at InvokeStub_Program.<Main>$(Object, Object, IntPtr*)
at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
--- End of stack trace from previous location ---
at Microsoft.AspNetCore.Mvc.Testing.DeferredHostBuilder.DeferredHost.StartAsync(CancellationToken cancellationToken)
at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Start(IHost host)
at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.CreateHost(IHostBuilder builder)
at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.DelegatedWebApplicationFactory.CreateHost(IHostBuilder builder)
at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.ConfigureHostBuilder(IHostBuilder hostBuilder)
at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.EnsureServer()
at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.CreateDefaultClient(DelegatingHandler[] handlers)
at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.CreateDefaultClient(Uri baseAddress, DelegatingHandler[] handlers)
at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.CreateClient(WebApplicationFactoryClientOptions options)
at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.CreateClient()
at Modular.Api.FunctionalTests.TestSupport.FunctionalTest..ctor() in /media/diegosasw/data/src/sandbox/eventuous/src/Sample/test/Modular.Api.FunctionalTests/TestSupport/FunctionalTest.cs:line 26
at Modular.Api.FunctionalTests.AnotherTests..ctor()
at System.RuntimeType.CreateInstanceDefaultCtor(Boolean publicOnly, Boolean wrapExceptions)
System.Reflection.TargetInvocationException
Exception has been thrown by the target of an invocation.
at System.Reflection.ConstructorInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
at System.Reflection.RuntimeConstructorInfo.InvokeWithManyArguments(RuntimeConstructorInfo ci, Int32 argCount, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at System.Reflection.RuntimeConstructorInfo.Invoke(BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at Eventuous.Subscriptions.Registrations.SubscriptionBuilder`2.ResolveSubscription(IServiceProvider sp) in /media/diegosasw/data/src/sandbox/eventuous/src/Core/src/Eventuous.Subscriptions/Registrations/SubscriptionBuilder.cs:line 218
at Microsoft.Extensions.DependencyInjection.SubscriptionRegistrationExtensions.<>c__DisplayClass0_0`2.<AddSubscription>b__1(IServiceProvider sp) in /media/diegosasw/data/src/sandbox/eventuous/src/Core/src/Eventuous.Subscriptions/Registrations/SubscriptionRegistrationExtensions.cs:line 43
System.NullReferenceException
Object reference not set to an instance of an object.
at Eventuous.EventStore.Subscriptions.EventStoreCatchUpSubscriptionBase`1..ctor(EventStoreClient eventStoreClient, T options, ICheckpointStore checkpointStore, ConsumePipe consumePipe, ILoggerFactory loggerFactory) in /media/diegosasw/data/src/sandbox/eventuous/src/EventStore/src/Eventuous.EventStore/Subscriptions/EventStoreCatchUpSubscriptionBase.cs:line 19
at Eventuous.EventStore.Subscriptions.AllStreamSubscription..ctor(EventStoreClient eventStoreClient, AllStreamSubscriptionOptions options, ICheckpointStore checkpointStore, ConsumePipe consumePipe, ILoggerFactory loggerFactory) in /media/diegosasw/data/src/sandbox/eventuous/src/EventStore/src/Eventuous.EventStore/Subscriptions/AllStreamSubscription.cs:line 67
at InvokeStub_AllStreamSubscription..ctor(Object, Object, IntPtr*)
at System.Reflection.ConstructorInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
Hopefully this makes more sense to you than to me 😃
PS: I forked eventuous, and added my own branch with eventuous + Sample where I can reproduce the issue
https://github.com/diegosasw/eventuous/tree/issues/181
from eventuous.
alexeyzimarev
commented on September 21, 2024
The only reason it can crash is that the options argument is null:
protected EventStoreCatchUpSubscriptionBase(
EventStoreClient eventStoreClient,
T options,
ICheckpointStore checkpointStore,
ConsumePipe consumePipe,
ILoggerFactory? loggerFactory
) : base(options, checkpointStore, consumePipe, options.ConcurrencyLimit, loggerFactory)
=> EventStoreClient = eventStoreClient;
Here's the only place that accesses anything inside the argument: options.ConcurrencyLimit.
I don't think it's related to the hosted service or the builder as it crashes in the EventStoreCatchUpSubscriptionBase constructor according to the stack trace.
from eventuous.
alexeyzimarev
commented on September 21, 2024
I will add a check if the options object is empty before doing anything else.
from eventuous.
alexeyzimarev
commented on September 21, 2024
There's no such thing as race condition when dependencies are being resolved. You don't see it crashing when you remove hosted services from the container as the subscription doesn't start, and, therefore, it's not being resolved, so all the DI-related code doesn't execute.
Honestly, I don't have any idea why it would not work.
from eventuous.
diegosasw
commented on September 21, 2024
Tried with 0.14.1-alpha.0.7. The issue must come from somewhere else but it manifests there.
Running tests separately or with parallelization disabled with
[assembly: CollectionBehavior(CollectionBehavior.CollectionPerClass, DisableTestParallelization = true)]
avoids the problem, but unfortunately when running in parallel it still fails randomly with this exception.
System.ArgumentNullException: Value cannot be null. (Parameter 'options')
System.ArgumentNullException
Value cannot be null. (Parameter 'options')
at Eventuous.EventStore.Subscriptions.EventStoreCatchUpSubscriptionBase`1..ctor(EventStoreClient eventStoreClient, T options, ICheckpointStore checkpointStore, ConsumePipe consumePipe, ILoggerFactory loggerFactory)
at Eventuous.EventStore.Subscriptions.AllStreamSubscription..ctor(EventStoreClient eventStoreClient, AllStreamSubscriptionOptions options, ICheckpointStore checkpointStore, ConsumePipe consumePipe, ILoggerFactory loggerFactory)
at InvokeStub_AllStreamSubscription..ctor(Object, Object, IntPtr*)
at System.Reflection.ConstructorInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
EDIT: Alejandro-SB changes solves the mysterious issue!
from eventuous.
Related Issues (20)
- Ability to return a custom return type for a command API reponse HOT 5
- SQL Server __schema__.Streams.StreamName is too big to be indexed
- Unable to create isolated Aggregates with State and Id HOT 5
- SQL Server Subscription skips an event on non-initial startup
- Provide StreamContext in UpdateBuilder<TBuilder>.Configure() for MondoDb HOT 1
- With the SQL Server event store, event handlers are sometimes called twice for each event HOT 4
- Health check stays "healthy" even if Event Handler throws an exception HOT 5
- Transient error drops the subscription. (Connection reset by peer) HOT 6
- Add subscription drop/resubscribe tests HOT 1
- Add subscription measure for RDBMS subs HOT 1
- Error at schema initialization (PostgresException: '42704: type public.stream_message[] does not exist) HOT 1
- Eventuous should always return `Result<TState>` HOT 2
- Does it make sense to keep `Aggregate` without state? HOT 9
- Preserve original exception stacktrace in command service HOT 1
- Enable custom configuration of NpgsqlDataSourceBuilder when using AddEventuousPostgres HOT 1
- DuplicateTypeException in CommandService HOT 11
- "Can not start an Activity that was already started" in BaseProducer
- Memory leak (CancellationTokenSource) in AsyncHandlingFilter HOT 1
- Use source generator for type mapping HOT 1
- Use source generator for mapping HTTP API to command handlers HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from eventuous.