This package enables sending data gathered by NServiceBus.Metrics to an instance of ServiceControl.Monitoring
License: Other
NServiceBus.Metrics.ServiceControl enables sending monitoring data gathered with NServiceBus.Metrics to a ServiceControl.Monitoring service.
It is part of the Particular Service Platform, which includes NServiceBus and tools to build, monitor, and debug distributed systems.
See the Send Metrics data to ServiceControl documentation for more details on how to use it.
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
![dependabot-preview[bot] avatar](https://avatars.githubusercontent.com/in/2141?v=4 "dependabot-preview[bot]")
![internalautomation[bot] avatar](https://avatars.githubusercontent.com/in/120307?v=4 "internalautomation[bot]")
This issue about preparing the release of v1 of this package that shorly will be followed up with next two versions: 2.0 (targeting NSB.Metrics 2.0) and 3.0 (targeting NSB.Metrics 3.0).
PoA:
SC.Monitoring to the date of the official release| NSB Version | NSB.Metrics version | Component location |
|---|---|---|
| 5 | - | NSB.Metrics.ServiceControl 1.* |
| 6 | 1.* | NSB.Metrics 1.* |
| 6 | 2.* | NSB.Metrics.ServiceControl 2.* |
| 7 | 3.* | NSB.Metrics.ServiceControl 3.* |
Current versions are not strong named.
When using the ASQ transport, user is required to manually escape characters in the monitoring queue name. i.e. if the monitoring queue name is Particular.Monitoring then it must be changed to Particular-Monitoring as the period character (.) is not allowed.
For transports that support non-durable messages (which at this point is only RabbitMQ), there can be performance advantages for sending messages as non-durable.
Metrics data seems like a good candidate for this kind of optimization because each individual data point isn't business critical.
For RabbitMQ, if we sent metrics data as non-durable, it gives the broker permission to skip disk I/O.
Based on Particular/docs.particular.net#4264, is there a specific reason the metrics plugin throws at startup for send-only endpoints?
I understand that as of now, the captured metrics aren't that interesting for send-only endpoints however, throwing exceptions seems to be an unexpected and unusual behavior at fist glance.
Dependabot couldn't reach https://www.myget.org/F/particular/api/v3/index.json as it timed out.
Is https://www.myget.org/F/particular/api/v3/index.json accessible over the internet? If it is then this may be a transitory issue and can be ignored - Dependabot will close it on its next successful update run.
Raised based on https://discuss.particular.net/t/shutdown-of-services-takes-a-long-time-with-servicecontrol-metrics-plugin-enabled/
All customers using SendMetricDataToServiceControl
When stopping an endpoint, the operation can be delayed until ServiceControlReportingInterval is reached.
The loop sending data to ServiceControl does not take the cancellation token into account, see
https://github.com/Particular/NServiceBus.Metrics.ServiceControl/blob/master/src/NServiceBus.Metrics.ServiceControl/ReportingFeature.cs#L188-L189
The reporting task sends messages to servicecontrol. The sending of these messages can fail if there is a connectivity issue with the transport. For example the following error with the Azure Service Bus transport:
System.TimeoutException at ServiceBus.Metrics.ServiceControl.ServiceControlReporting.NServiceBusMetadataReport+<RunReportAsync>d__1.MoveNext
Details:
"Microsoft.Azure.ServiceBus.ServiceBusTimeoutException","message":"The operation did not complete within the allocated time 00:00:59.9953352 for object sender165."
This results in a log entry with log level ERROR but without any information for operators that analyze the logs for problems.
The log entry should show information about the error and its impact. For example, we could catch exceptions and wrap them in a new exception that provides information or write a log entry with info like
Failed to dispatch metrics message for ServiceControl monitoring to transport
We could also consider to not have the log level set to ERROR but to WARN as metric messages are not critical for operation.
No response
The ASQ transport uses a serializer to create message envelopes as ASQ does not support headers.
The monitoring instance is expecting json-serialized message envelopes.
If a serializer other than json is used in the endpoint then the monitoring instance cannot deserialize any monitoring messages it is sent.
Hello,
I've upgraded to v3.0.2 but now I get the following Exception
System.IO.FileLoadException: 'Could not load file or assembly 'NServiceBus.Metrics.ServiceControl, Version=3.0.0.0, Culture=neutral, PublicKeyToken=null' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)'
When I downgrade to 3.0.1 it's working again.
<package id="Castle.Core" version="4.3.1" targetFramework="net461" />
<package id="Microsoft.ApplicationInsights" version="2.7.2" targetFramework="net461" />
<package id="Microsoft.ApplicationInsights.NLogTarget" version="2.7.2" targetFramework="net461" />
<package id="Microsoft.Azure.KeyVault.Core" version="3.0.0" targetFramework="net461" />
<package id="Microsoft.Azure.WebJobs" version="2.2.0" targetFramework="net461" />
<package id="Microsoft.Azure.WebJobs.Core" version="2.2.0" targetFramework="net461" />
<package id="Microsoft.Data.Edm" version="5.8.4" targetFramework="net461" />
<package id="Microsoft.Data.OData" version="5.8.4" targetFramework="net461" />
<package id="Microsoft.Data.Services.Client" version="5.8.4" targetFramework="net461" />
<package id="Microsoft.Extensions.Logging.Abstractions" version="2.1.1" targetFramework="net461" />
<package id="Microsoft.NETCore.Platforms" version="2.1.1" targetFramework="net461" />
<package id="Microsoft.VisualStudio.SlowCheetah" version="3.1.66" targetFramework="net461" developmentDependency="true" />
<package id="Microsoft.Web.WebJobs.Publish" version="2.0.0" targetFramework="net461" />
<package id="Microsoft.Win32.Primitives" version="4.3.0" targetFramework="net461" />
<package id="Microsoft.WindowsAzure.ConfigurationManager" version="3.2.3" targetFramework="net461" />
<package id="NETStandard.Library" version="2.0.3" targetFramework="net461" />
<package id="Newtonsoft.Json" version="11.0.2" targetFramework="net461" />
<package id="Ninject" version="3.3.4" targetFramework="net461" />
<package id="Ninject.Extensions.ContextPreservation" version="3.3.1" targetFramework="net461" />
<package id="Ninject.Extensions.Factory" version="3.3.2" targetFramework="net461" />
<package id="Ninject.Extensions.NamedScope" version="3.3.0" targetFramework="net461" />
<package id="NLog" version="4.5.10" targetFramework="net461" />
<package id="NServiceBus" version="7.1.4" targetFramework="net461" />
<package id="NServiceBus.Azure.Transports.WindowsAzureServiceBus" version="8.0.3" targetFramework="net461" />
<package id="NServiceBus.DataBus.AzureBlobStorage" version="2.0.0" targetFramework="net461" />
<package id="NServiceBus.Heartbeat" version="3.0.0" targetFramework="net461" />
<package id="NServiceBus.Metrics" version="3.0.0" targetFramework="net461" />
<package id="NServiceBus.Metrics.ServiceControl" version="3.0.2" targetFramework="net461" />
<package id="NServiceBus.Newtonsoft.Json" version="2.1.0" targetFramework="net461" />
<package id="NServiceBus.Ninject" version="7.0.0" targetFramework="net461" />
<package id="NServiceBus.NLog" version="3.0.0" targetFramework="net461" />
<package id="System.AppContext" version="4.3.0" targetFramework="net461" />
<package id="System.Collections" version="4.3.0" targetFramework="net461" />
<package id="System.Collections.Concurrent" version="4.3.0" targetFramework="net461" />
<package id="System.ComponentModel.EventBasedAsync" version="4.3.0" targetFramework="net461" />
<package id="System.Console" version="4.3.1" targetFramework="net461" />
<package id="System.Diagnostics.Debug" version="4.3.0" targetFramework="net461" />
<package id="System.Diagnostics.DiagnosticSource" version="4.5.0" targetFramework="net461" />
<package id="System.Diagnostics.Tools" version="4.3.0" targetFramework="net461" />
<package id="System.Diagnostics.Tracing" version="4.3.0" targetFramework="net461" />
<package id="System.Dynamic.Runtime" version="4.3.0" targetFramework="net461" />
<package id="System.Globalization" version="4.3.0" targetFramework="net461" />
<package id="System.Globalization.Calendars" version="4.3.0" targetFramework="net461" />
<package id="System.IO" version="4.3.0" targetFramework="net461" />
<package id="System.IO.Compression" version="4.3.0" targetFramework="net461" />
<package id="System.IO.Compression.ZipFile" version="4.3.0" targetFramework="net461" />
<package id="System.IO.FileSystem" version="4.3.0" targetFramework="net461" />
<package id="System.IO.FileSystem.Primitives" version="4.3.0" targetFramework="net461" />
<package id="System.Linq" version="4.3.0" targetFramework="net461" />
<package id="System.Linq.Expressions" version="4.3.0" targetFramework="net461" />
<package id="System.Linq.Queryable" version="4.3.0" targetFramework="net461" />
<package id="System.Net.Http" version="4.3.3" targetFramework="net461" />
<package id="System.Net.Primitives" version="4.3.0" targetFramework="net461" />
<package id="System.Net.Requests" version="4.3.0" targetFramework="net461" />
<package id="System.Net.Sockets" version="4.3.0" targetFramework="net461" />
<package id="System.ObjectModel" version="4.3.0" targetFramework="net461" />
<package id="System.Reflection" version="4.3.0" targetFramework="net461" />
<package id="System.Reflection.Extensions" version="4.3.0" targetFramework="net461" />
<package id="System.Reflection.Primitives" version="4.3.0" targetFramework="net461" />
<package id="System.Resources.ResourceManager" version="4.3.0" targetFramework="net461" />
<package id="System.Runtime" version="4.3.0" targetFramework="net461" />
<package id="System.Runtime.Extensions" version="4.3.0" targetFramework="net461" />
<package id="System.Runtime.Handles" version="4.3.0" targetFramework="net461" />
<package id="System.Runtime.InteropServices" version="4.3.0" targetFramework="net461" />
<package id="System.Runtime.InteropServices.RuntimeInformation" version="4.3.0" targetFramework="net461" />
<package id="System.Runtime.Numerics" version="4.3.0" targetFramework="net461" />
<package id="System.Security.Cryptography.Algorithms" version="4.3.1" targetFramework="net461" />
<package id="System.Security.Cryptography.Encoding" version="4.3.0" targetFramework="net461" />
<package id="System.Security.Cryptography.Primitives" version="4.3.0" targetFramework="net461" />
<package id="System.Security.Cryptography.X509Certificates" version="4.3.2" targetFramework="net461" />
<package id="System.Spatial" version="5.8.4" targetFramework="net461" />
<package id="System.Text.Encoding" version="4.3.0" targetFramework="net461" />
<package id="System.Text.Encoding.Extensions" version="4.3.0" targetFramework="net461" />
<package id="System.Text.RegularExpressions" version="4.3.0" targetFramework="net461" />
<package id="System.Threading" version="4.3.0" targetFramework="net461" />
<package id="System.Threading.Tasks" version="4.3.0" targetFramework="net461" />
<package id="System.Threading.Timer" version="4.3.0" targetFramework="net461" />
<package id="System.Xml.ReaderWriter" version="4.3.1" targetFramework="net461" />
<package id="System.Xml.XDocument" version="4.3.0" targetFramework="net461" />
<package id="WindowsAzure.ServiceBus" version="4.1.11" targetFramework="net461" />
<package id="WindowsAzure.Storage" version="8.7.0" targetFramework="net461" />
Currently, the RawDataReporter component accepts the minimal flushSize that is required to send a message to a SC.Monitoring instance. This value is used to provide a lower boundary for the size of flushed data and ensure that reports are sent in a reasonable batches, not keeping too much data for too long. Unfortunately, this value is used only for the lower boundary. The upper limit is set by taking the end of the current chunk of data, which effectively is unbounded (or bounded by the ring itself, which is a lot).
I propose to introduce an upper boundary for the length of a reported chunk. This would highly likely address this problem.
The following error was observed with Azure Storage Queues transport. It's worth to notice that it happens only under a big load (when number of reported values in a time of sending one reporting message is big enough).
Azure Storage Queue messages cannot exceed 64kb, this is exceeded under load.
2018-01-26 14:19:39,663|ERROR|99|NServiceBus.Metrics.ServiceControl.ReportingFeature+ServiceControlRawDataReporting|Error while reporting raw data to particular.monitoring.asq.
System.ArgumentException: Messages cannot be larger than 65536 bytes.
at Microsoft.WindowsAzure.Storage.Queue.CloudQueueMessage.GetMessageContentForTransfer(Boolean shouldEncodeMessage, QueueRequestOptions options) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\Common\Queue\CloudQueueMessage.Common.cs:line 285
at Microsoft.WindowsAzure.Storage.Queue.CloudQueue.AddMessageImpl(CloudQueueMessage message, Nullable`1 timeToLive, Nullable`1 initialVisibilityDelay, QueueRequestOptions options) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Queue\CloudQueue.cs:line 2541
at Microsoft.WindowsAzure.Storage.Queue.CloudQueue.BeginAddMessage(CloudQueueMessage message, Nullable`1 timeToLive, Nullable`1 initialVisibilityDelay, QueueRequestOptions options, OperationContext operationContext, AsyncCallback callback, Object state) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Queue\CloudQueue.cs:line 1324
at Microsoft.WindowsAzure.Storage.Core.Util.AsyncExtensions.TaskFromVoidApm[T1,T2,T3,T4,T5](Func`8 beginMethod, Action`1 endMethod, T1 arg1, T2 arg2, T3 arg3, T4 arg4, T5 arg5, CancellationToken cancellationToken) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Util\AsyncExtensions.cs:line 246
at Microsoft.WindowsAzure.Storage.Queue.CloudQueue.AddMessageAsync(CloudQueueMessage message, Nullable`1 timeToLive, Nullable`1 initialVisibilityDelay, QueueRequestOptions options, OperationContext operationContext, CancellationToken cancellationToken) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Queue\CloudQueue.cs:line 1400
at Microsoft.WindowsAzure.Storage.Queue.CloudQueue.AddMessageAsync(CloudQueueMessage message, Nullable`1 timeToLive, Nullable`1 initialVisibilityDelay, QueueRequestOptions options, OperationContext operationContext) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Queue\CloudQueue.cs:line 1382
at NServiceBus.AzureStorageQueues.Dispatcher.Send(MessageWrapper wrapper, CloudQueue sendQueue) in C:\BuildAgent\work\956312761da410d2\src\Transport\Dispatcher.cs:line 105
at NServiceBus.AzureStorageQueues.Dispatcher.<Send>d__2.MoveNext() in C:\BuildAgent\work\956312761da410d2\src\Transport\Dispatcher.cs:line 95
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.Metrics.ServiceControl.ReportingFeature.ServiceControlRawDataReporting.<>c__DisplayClass3_0.<<CreateReporter>g__Sender0>d.MoveNext() in C:\BuildAgent\work\5769f7f8f0ebec65\src\NServiceBus.Metrics.ServiceControl\ReportingFeature.cs:line 250
A customer that targets .net framework 4.5 with NServiceBus 5 cannot use NServiceBus.Metrics.ServiceControl as it targets 4.5.2
Failed to buffer metrics data for ${metricType} after ${attempts} attempts.Some customers experience the symptoms listed above due to a non-deterministic execution order due to an unknown root cause. It is possible for some endpoints to be affected while others are not.
The ReportingFeature feature has an implicit dependency on the MetricsFeature provided by the NServiceBus.Metrics package. However, this dependency is not expressed via the DependsOn API which can potentially cause these features to be used in a different orders.
While the Setup code is not directly dependent, the FeatureStartupTasks of the features are. The expected order of the feature setup and task start is the following:
SetupRegisteredObservers startup taskoptions.RegisterObservers API and registers the following (simplified) callback logic: metrics = registerCallback.metrics. It then passes metrics to the ServiceControlRawDataReporting startup task as a reference.options.RegisterObservers, including the logic mentioned in step 2.Because the order isn't actually guaranteed, this order can also happen:
options.RegisterObservers API and registers the following (simplified) callback logic: metrics = registerCallback.metrics. It then passes metrics to the ServiceControlRawDataReporting startup task as a reference.SetupRegisteredObservers startup taskmetric collection is empty here because the ServiceControlRawDataReporting.Start hasn't called the callback yetoptions.RegisterObservers.in this case, no reporters have been setup for the metrics. In the long run, this also causes the metrics buffer to be continuously filled without any consumer draining it, which also causes the following warnings in the log: Failed to buffer metrics data for ${metricType} after ${attempts} attempts.. At first glance, the logic which logs this warning in RingBufferExtensions also causes and infinite-loop. I did not further investigate this, but it looks like this could cause additional issues.
additional notes:
MetricsFeature to be prioritized due to a DependsOn dependency from a custom feature. This does not solve the problem as this causes the Setup phase to execute in the right order (this is not a reliable approach anyway but I'm not going deeper on this right now) but it has no impact on the order of FeatureStartupTasks, this order is completely dependant on the orde of the scanned typed by the AssemblyScanner.endpointConfiguration.AssemblyScanner().ScanAppDomainAssemblies = false; as it has a chance of using a different order for the assembly-scanning process but it's by no means a reliable workaround.
I am upgrading NSB 6 to 7 and I use the Metrics 2.* . So I had to update the metrics to 3.0 in order to work with NSB 7 (not compatible with version 6). Now the issue is that there is not a version compatible for NServiceBus.Metrics.ServiceControl (current only NServiceBus.Metrics (>= 2.0.0 && < 3.0.0))
When is planed a compatible version?
The SimpleJson codebase is not maintained and is fairly old. This constitutes a maintenance risk, and while there are no known vulnerabilities, if there ever were any detected, it's unlikely there would be a fix.
It should be possible to replace SimpleJson, perhaps with System.Text.Json, although some migration may be required.
We're running multiple endpoints in a docker environment with similar settings. Regularly one of those goes on fritz and uses 100% available CPU. When checking the logs via docker container logs --tail 100 <name> the log is completely filled with thousands of these messages:
2020-10-08 09:25:54.725 WARN Failed to buffer metrics data for $CriticalTime after $10 attempts.
2020-10-08 09:25:54.725 WARN Failed to buffer metrics data for $CriticalTime after $10 attempts.
2020-10-08 09:25:54.725 WARN Failed to buffer metrics data for $CriticalTime after $10 attempts.
2020-10-08 09:25:54.725 WARN Failed to buffer metrics data for $CriticalTime after $10 attempts.
2020-10-08 09:25:54.726 WARN Failed to buffer metrics data for $CriticalTime after $10 attempts.
2020-10-08 09:25:54.726 WARN Failed to buffer metrics data for $CriticalTime after $10 attempts.
2020-10-08 09:25:54.726 WARN Failed to buffer metrics data for $CriticalTime after $10 attempts.
2020-10-08 09:25:54.726 WARN Failed to buffer metrics data for $CriticalTime after $10 attempts.
2020-10-08 09:25:54.726 WARN Failed to buffer metrics data for $CriticalTime after $10 attempts.
2020-10-08 09:25:54.726 WARN Failed to buffer metrics data for $CriticalTime after $10 attempts.
2020-10-08 09:25:54.726 WARN Failed to buffer metrics data for $CriticalTime after $10 attempts.
2020-10-08 09:25:54.726 WARN Failed to buffer metrics data for $CriticalTime after $10 attempts.
2020-10-08 09:25:54.726 WARN Failed to buffer metrics data for $CriticalTime after $10 attempts.
2020-10-08 09:25:54.727 WARN Failed to buffer metrics data for $CriticalTime after $10 attempts.
2020-10-08 09:25:54.727 WARN Failed to buffer metrics data for $CriticalTime after $10 attempts.
2020-10-08 09:25:54.727 WARN Failed to buffer metrics data for $CriticalTime after $10 attempts.
2020-10-08 09:25:54.727 WARN Failed to buffer metrics data for $CriticalTime after $10 attempts.
2020-10-08 09:25:54.727 WARN Failed to buffer metrics data for $CriticalTime after $10 attempts.
2020-10-08 09:25:54.727 WARN Failed to buffer metrics data for $CriticalTime after $10 attempts.
2020-10-08 09:25:54.727 WARN Failed to buffer metrics data for $CriticalTime after $10 attempts.
2020-10-08 09:25:54.727 WARN Failed to buffer metrics data for $CriticalTime after $10 attempts.
2020-10-08 09:25:54.728 WARN Failed to buffer metrics data for $CriticalTime after $10 attempts.
2020-10-08 09:25:54.728 WARN Failed to buffer metrics data for $CriticalTime after $10 attempts.
2020-10-08 09:25:54.728 WARN Failed to buffer metrics data for $CriticalTime after $10 attempts.
2020-10-08 09:25:54.728 WARN Failed to buffer metrics data for $CriticalTime after $10 attempts.
2020-10-08 09:25:54.728 WARN Failed to buffer metrics data for $CriticalTime after $10 attempts.
2020-10-08 09:25:54.728 WARN Failed to buffer metrics data for $CriticalTime after $10 attempts.
2020-10-08 09:25:54.728 WARN Failed to buffer metrics data for $CriticalTime after $10 attempts.
2020-10-08 09:25:54.728 WARN Failed to buffer metrics data for $CriticalTime after $10 attempts.
2020-10-08 09:25:54.728 WARN Failed to buffer metrics data for $CriticalTime after $10 attempts.
2020-10-08 09:25:54.729 WARN Failed to buffer metrics data for $CriticalTime after $10 attempts.
2020-10-08 09:25:54.729 WARN Failed to buffer metrics data for $CriticalTime after $10 attempts.
2020-10-08 09:25:54.729 WARN Failed to buffer metrics data for $CriticalTime after $10 attempts.
2020-10-08 09:25:54.729 WARN Failed to buffer metrics data for $CriticalTime after $10 attempts.
What process is responsible for generating this message (so I can perhaps deduct what's different between this particular service and the others that are seemingly similarly configured)? Any help resolving this would be appreciated.
Thanks
I just created a test endpoint, using the SQL transport. The monitoring queue didn't exist yet and I'm getting queue does not exist exceptions. The endpoint instance remains running, no critical error is raised. However, the log file gets flooded with ERROR log entries.
Maybe this exception needs to be handled gracefully, demoted to a single line WARN entry instead of a full stacktrace ERROR?
Problem:
When all the endpoints are run the first time, the ServicePulse Monitoring dashboard reports queue length correctly.
Restarting the sending and receiving endpoint, and running the same load test, the dashboard shows all of the other metrics, such as Processing time, Critical time, etc increasing, but queue length remains at 0.
@mikeminutillo was able to repro this on the monitoring demo after updating to RC bits.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.