[QUERY] Monitoring ServiceBusProcessor about azure-sdk-for-net HOT 7 CLOSED

Hi @cjablonski76. Thanks for reaching out and for your question. The Service Bus processor is designed to be highly resilient to errors and will automatically recover from transient errors and continue processing. It does not automatically stop under any normal circumstances unless your process crashes. So long as IsProcessing returns true, the processor is running and actively attempting to receive messages. Any errors that it encounters, whether part of the internal infrastructure or in your processing handler, are surfaced to the error handler. If the flag is set and your error handler is not being pinged, then the processor is running, healthy, and looking for messages.

As it is working, the processor emits tracing data that follows the Open Telemetry Messaging specification, which can be collected using any OTel tooling, such as the Azure Monitor Open Telemetry Exporter package. If you're looking for low-level instrumentation, the Azure SDK logs emit data for every service operation and its result as well as client state. As you appear familiar with those, I'll leave out the details.

To summarize all of that:

• There's really no need to monitor service traffic to understand processor health. So long as it has been started, the processor is self-healing and will continue to attempt receives. Your error handler will be notified of errors as they're encountered. If your error handler is not seeing activity, the processor is not experiencing problems.

• To monitor activities related to processing, the OTel traces are the best approach and follow a well-documented standard.

• To watch individual service operations and client state, the Azure SDK logs that you're referring to are the only option.

is there any similar metric emitted so that we can effectively monitor our receivers to ensure that there's always one or more receivers available?

That is not something that the Azure SDK clients have insight into. Each client is independent and tracks its own activity and state. It sounds like you're asking about service-level metrics, which the Azure SDK maintainers can't assist with. Your best path forward for would be to open an Azure support request or inquire on the Microsoft Q&A site.

from azure-sdk-for-net.

Hi @cjablonski76. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text "/unresolve" to remove the "issue-addressed" label and continue the conversation.

from azure-sdk-for-net.

Ok, @jsquire, that all makes sense. I think what is throwing me then is that I keep seeing some exception bubble to my error handle method.
It appears that when our containers are being shutdown (as part of natural events), we're getting hit with exceptions like:

ErrorSource: Receive.
Exception: System.ObjectDisposedException:
The message processor is unable to continue and will stop processing;
the host application has closed the connection to the Service Bus service.
Object name: 'ServiceBusConnection'.
System.ObjectDisposedException:
The message processor is unable to continue and will stop processing; the host application has closed the connection to the Service Bus service.
Object name: 'ServiceBusConnection'.

I assume this isn't expected behavior, right?

from azure-sdk-for-net.

whoops, forgot to add the /unresolve

from azure-sdk-for-net.

@cjablonski76 : The TLDR is that that is normal when you don't stop your processor.

That exception is raised when the processor isn't shut down gracefully and the ServiceBusClient that spawned the processor is closed/disposed while the processor is running. Since the underlying connection is owned by the client, the processor loses access to the connection while still running. Because this only happens when an application intentionally calls CloseAsync or DisposeAsync on the client (directly or via DI cleanup), the processor knows that it is not allowed to recover the connection and, instead, allows its internal receivers to crash.

You can prevent this by stopping the processor before cleaning up the client.

from azure-sdk-for-net.

Hi @cjablonski76. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text "/unresolve" to remove the "issue-addressed" label and continue the conversation.

from azure-sdk-for-net.

@cjablonski76 : The TLDR is that that is normal when you don't stop your processor.

That exception is raised when the processor isn't shut down gracefully and the ServiceBusClient that spawned the processor is closed/disposed while the processor is running. Since the underlying connection is owned by the client, the processor loses access to the connection while still running. Because this only happens when an application intentionally calls CloseAsync or DisposeAsync on the client (directly or via DI cleanup), the processor knows that it is not allowed to recover the connection and, instead, allows its internal receivers to crash.

You can prevent this by stopping the processor before cleaning up the client.

Ok, that's what I was afraid of. I'll have to figure out how a way to stop the processor gracefully when the app shuts down.
Thanks!

from azure-sdk-for-net.