Connects to Particular/PlatformDevelopment#729

The readme could include some helpful information about how to run acceptance tests:

• Setting AzureStoragePersistence.ConnectionString env variable (and the ASQ connection string if necessary)
• How to swap out the default transport with, say, ASQ
• Maybe instructions on how to set up ASP in Azure and how to get the connection string?

Issue by danielHalan
8/3/2014 12:50:19 PM +00:00

Originally opened as Particular/NServiceBus.Azure#156

Would be nice if the Exceptions thrown by the AzureSagaPersister infrastucture would appear in the NSB Output as full messages, as they appear today as "Message has failed".

The specific problem was when changing from Raven to Azure Saga Storage, and Azure does't support some data types such as string[] etc. The Initiating Command would process successfully, and no Exception raised by the Debugged, but the Message would fail, and for an short instance it would have saved the correct error message in the header, but quickly there-after the command is re-run, then producing an Exception that the Saga has already started, and overwriting the 'real' error message... and no trace of the source error in the output logs...

Think otherwise it would be a good practice to have all infrastructural errors logged with descriptive "ERROR" messages, especially when dealing with Azure which has some of its own quirks.

Connects to Particular/NServiceBus.Azure#309

When a new subscription arrives for an endpoint already having a subscription persisted, the new subscription is discarded. With v6, the new subscription contains additional information about the logical endpoint, required for sender-side distribution which is then not available. This will lead to duplicate events dispatched to scaled out instances.

Failing test available here: cecde17

ping @Particular/azure-storage-persistence-maintainers

The nuspec file currently points to NServiceBus v5, this should be updated to v6.

https://github.com/Particular/NServiceBus.AzureStoragePersistence/blob/develop/packaging/nuget/nservicebus.azurestoragepersistence.nuspec#L21

Connects to https://github.com/Particular/PlatformDevelopment/issues/729

With version 7.0.0 of the WindowsAzure.Storage library, we should package ASQ with it to allow customers to use version 7+, which they will be asking to allow them to use the latest MSFT library.

this is in SecondaryIndexPersister. i assume a rethrow (outside the if) is missing?

                    try
                    {
                        await persist(deserializeSagaData).ConfigureAwait(false);
                    }
                    catch (StorageException e)
                    {
                        if (IsConflict(e))
                        {
                            // swallow ex as another worker created the primary under this key
                        }
                    }

running this
http://docs.particular.net/samples/azure/storage-persistence/#Nuget-v6-pre

with no emulator and break on all exceptions results in many exceptions on the server project

for example

Microsoft.WindowsAzure.Storage.StorageException occurred
  HResult=-2146233088
  Message=Unable to connect to the remote server
  Source=Microsoft.WindowsAzure.Storage
  StackTrace:
       at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteSync[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Executor\Executor.cs:line 604
  InnerException: 
       HResult=-2146233079
       Message=Unable to connect to the remote server
       Source=System
       StackTrace:
            at System.Net.HttpWebRequest.GetResponse()
            at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteSync[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Executor\Executor.cs:line 695
       InnerException: 
            ErrorCode=10061
            HResult=-2147467259
            Message=No connection could be made because the target machine actively refused it 127.0.0.1:10002
            NativeErrorCode=10061
            Source=System
            StackTrace:
                 at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
                 at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
            InnerException: 

But none of them seem to bubble up to the console or log

Issue by yvesgoeleven
12/15/2014 9:54:14 AM +00:00

Originally opened as Particular/NServiceBus.Azure#225

As an implementation for: Particular/NServiceBus#2602

• Change ATs from the current package to a suite of Persistence specific tests
• Update Component Tests to cover Sagas and Pub/Sub

so that the message doesn't get lost

There is some legacy there from when the codebase use to exist in the azure project

Connects to Particular/PlatformDevelopment#182

This is the AzureStoragePersistence adapter epic which incorporates the work for making the AzureStoragePersistence adapter async/await enabled. Please see this blog post about the reasoning behind it.

instead locking everywhere?

To be consistent with core v6 & the rest of the transports & persistences, we should switch to a code only configuration model on top of SettingsHolder.

Todo: should we provide an overlay for backward compat of xml based config model as well?

Includes following subtasks:

• Remove default connectionstring #3
• Add ability to configure connection string for all persister in a single command #15
• Support NServiceBus/Persistence setting #8

Connects to https://github.com/Particular/PlatformDevelopment/issues/729

When converting the subscription code I noticed that we only persisted the transport address and the message type. Once upgraded to Core v6, ASP will need to store and retrieve both the transport address and the endpoint name. This is not a problem for subscriptions that are created after the new ASP assemblies are released, but it is a problem for the subscriptions that were created with Core v5 and retrieved using the Core v6 version of ASP.

A subscription created against the Core v5 assemblies will have no specific Endpoint Name to return when requested by the Core v6 assemblies. As such, we need to figure out how to construct that Endpoint Name from the data we have (transport address and message type).

In discussion with @SeanFeldman we were looking at message type and that value, once decoded, looks like:

Samples.Azure.StoragePersistence.Client@SOME-MACHINE

We can parse the Endpoint Name from that value, but it raised the question of on-premise vs Azure hosted endpoints. On-premise endpoints will have a message type that includes the '@' symbol followed by the machine name. My understanding is that those hosted in Azure will not have the '@' and machine name in them.

The question we ended up on was to if having two endpoints registering as subscribers, each with an endpoint name that only consists of the type, would cause issues?

@yvesgoeleven @Scooletz @Particular/azure-maintainers

Connects to https://github.com/Particular/PlatformDevelopment/issues/836

Who's affected

• Any user using NServiceBus.Azure persistence for sagas

Symptoms

When saga completes, secondary index record is left behind, filling up storage table with records.

Steps to reproduce

Start and complete a saga. Once saga completed, inspect storage table to find secondary index record. Persistence sample can be used to reproduce the issue.

Currently Azure saga persister is not performant at all.
When saga is persisted, partition/row keys are assigned an internal GUID generated by NServiceBus which has nothing to do with the message data. When customers provide mappings to find saga instances, partition/row keys are not involved, causing azure storage full table scan. That's a performance killer. From Azure documentation:

As with keys in a relational database table, the PartitionKey and RowKey values are indexed to create a clustered index that enables fast look-ups; however, the Table service does not create any secondary indexes so these are the only two indexed properties.

The level of performance you are guaranteed is explicitly set at the partition/row key level. therefore, when you run a full table scan, its not efficient and doesn't have any guarantee of performance.

Performance (top to bottom) based on query types:

1. PartitionKey + RowKey
2. PartitionKey + RowKey + OtherProperty
3. PartitionKey + OtherProperty
4. OtherProperty <== NSB today

Azure Storage changed from returning a WebException to using a StorageException. This needs to be changed in code to make sure that any logic in exception handlers is executed properly

After dealing with multiple errors during bump up of the recent core6 beta7, I think, that we should reconsider the fact of copying acceptance tests to the storage solution. Should we use ATT project as a whole suite? I'm not sure, but using a copied set of tests (without applying fixes and changes from the core) doesn't look like a good idea. What's your opinion @Particular/azure-storage-persistence-maintainers ?

Issue by kseniya-terekhova 1/24/2016 1:28:44 PM +00:00 Originally opened as Particular/NServiceBus#3352

We've encountered an issue in our environment when running several instances of NserviceBus with timeouts persisted in AzureStorage. Sometimes message's retries are prematurely stoped after such an exception have been logged:
System.Data.Services.Client.DataServiceClientException: {"odata.error":{"code":"UpdateConditionNotSatisfied","message":{"lang":"en-US","value":"The update condition specified in the request was not satisfied...

This situation seems to be caused by concurrently working NServiceBus.Azure.TimeoutPersister's copies, that perform following actions:

1. Instance1 calls TryRemove method that loads TimeoutDataEntity with PartionKey==timeoutId, just before Instance2 have removed this entity. Method goes further and conceivably slows down when downloading timeout state from AzureBlob.
2. At that time Instance2 removes all TimeoutDataEntities, performs unsuccessful message handling and schedules next retry by saving TimeoutDataEntity with PartitionKey=='Time in PartitionKeyScope format' and RowKey==timeoutId, and also new TimeoutDataEntity with PartionKey==timeoutId
3. TryRemove method of Instance1 loads "fresh" TimeoutDataEntity with with PartitionKey=='Time in PartitionKeyScope format' (due to insufficient format granularity ("yyyyMMddHH")) and marks it for deletion in ServiceContext. Old TimeoutDataEntity with PartionKey==timeoutId is market for deletion only after that.
4. Instance1 calls non-transactional context.SaveChangesWithRetries(), that removes "fresh" TimeoutDataEntity with PartitionKey=='Time in PartitionKeyScope format', but fails to remove old, non-existent TimeoutDataEntity with PartionKey==timeoutId, throwing UpdateConditionNotSatisfied exception.
5. TimeoutDataEntity with with PartitionKey=='Time in PartitionKeyScope format' is not found by GetNextChunk method any more, so message retries are stopped. In addition, timeout state from AzureBlob is lost as well.

We found this problem in NserviceBus 4.7.0.0, as in later versions TryRemove method silently swallows DataServiceRequestException and no indication of retries' problem is visible.

The suggestion is to move deletion of TimeoutDataEntity with PartionKey==timeoutId just after it has been loaded and remove AzureBlob with timeout state only when all TimeoutDataEntities were successfully erased.

Issue by SeanFeldman 10/28/2014 12:33:03 PM +00:00 Originally opened as Particular/NServiceBus.Azure#199

Values provided by users in ConfigSection should take precedence over values provided with Fluent API, while Fluent API overrides default values set by ConfigSeaction

Affects:

• Timeouts
• Sagas
• Subscriptions

Should be (core way): ConfigSection defaults --> Fluent API --> CofnigSection user values
Currently: ConfigSection defaults --> CofnigSection user values --> Fluent API

Task force: @DavidBoike @dbelcham @SeanFeldman @yvesgoeleven

This is the overall epic which incorporates the work for making the Azure Storage Persistence work with core v6.

It includes following subtasks:

• Acceptence Tests #9
• Async/Await support: #14
• Upgrade Azure Storage Dependency to latest major range #21
• Code only configuration API #20
• Update Nuspec #35

Currently, all the Azure Storage operations use the calls without providing an OperationContext instance. Passing a custom instance with OperationContext.ClientRequestID set to a value that has a common prefix for all the Azure operations in a given call could be helpful in:

• determining the cause of different problems
• reasoning about order of operations (just use OperationContext.GlobalRequestCompleted and print the id to a log or a test output)
• when needed, ensuring an order between operations run in parallel with Task.WhenAll (again using the global hook OperationContext.GlobalRequestCompleted + wait handles).

The PR showing a proposed solution is #89

Since the persister has no way of enforcing uniqueness two messages starting the same saga will result in both of them succeeding and creating 2 "records" in storage.

Following Gets will return the first instance stored stored causing inconsistent business behavior.

Originally raised by @SimonCropp here: Particular/NServiceBus.Azure#21

so here https://github.com/Particular/NServiceBus.AzureStoragePersistence/blob/develop/src/NServiceBus.AzureStoragePersistence/Timeout/TimeoutLogic/TimeoutPersister.cs#L348

we do

    bool TryGetTimeoutData(CloudTable timeoutDataTable, string partitionKey, string rowKey, out TimeoutDataEntity result)
    {
        result = (from c in timeoutDataTable.CreateQuery<TimeoutDataEntity>()
                  where c.PartitionKey == partitionKey && c.RowKey == rowKey // issue #191 cannot occur when both partitionkey and rowkey are specified
                  select c).ToList().SafeFirstOrDefault();

        return result != null;

    }

where SafeFirstOrDefault handles the possible storage exception https://github.com/Particular/NServiceBus.AzureStoragePersistence/blob/develop/src/NServiceBus.AzureStoragePersistence/SafeLinqExtensions.cs#L17

but TryGetTimeoutData does a ToList before calling SafeFirstOrDefault so the StorageException will be thrown outside the scope of SafeFirstOrDefault

When_removing_timeouts_from_the_storage seems to be flaky.

e.g. fails on this build: https://builds.particular.net/viewLog.html?buildId=168165&tab=buildResultsDiv&buildTypeId=NServiceBus_Persistence_NServiceBus_AzureStoragePersistence_Build
but is succeeds if run again: https://builds.particular.net/viewLog.html?buildId=168174&tab=buildResultsDiv&buildTypeId=NServiceBus_Persistence_NServiceBus_AzureStoragePersistence_Build

Version 6.2.3
When the saga data contains a DateTime property that is set to DateTime.Min the following exception occurs:

Microsoft.WindowsAzure.Storage.StorageException: Unexpected response code for operation : 0
   at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteSync[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext)
   at Microsoft.WindowsAzure.Storage.Table.CloudTable.ExecuteBatch(TableBatchOperation batch, TableRequestOptions requestOptions, OperationContext operationContext)
   at NServiceBus.SagaPersisters.Azure.AzureSagaPersister.Persist(IContainSagaData saga) in c:\BuildAgent\work\32a824616368e685\src\NServiceBus.Azure\SagaPersisters\Azure\AzureSagaPersister.cs:line 214
   at NServiceBus.SagaPersistenceBehavior.Invoke(IncomingContext context, Action next) in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Sagas\SagaPersistenceBehavior.cs:line 118
   at NServiceBus.BehaviorChain`1.InvokeNext(T context) in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Pipeline\BehaviorChain.cs:line 107
   at NServiceBus.BehaviorChain`1.<>c__DisplayClass4_0.<InvokeNext>b__0() in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Pipeline\BehaviorChain.cs:line 95
   at NServiceBus.SetCurrentMessageBeingHandledBehavior.Invoke(IncomingContext context, Action next) in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Unicast\Behaviors\SetCurrentMessageBeingHandledBehavior.cs:line 17
   at NServiceBus.BehaviorChain`1.InvokeNext(T context) in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Pipeline\BehaviorChain.cs:line 107
   at NServiceBus.BehaviorChain`1.<>c__DisplayClass4_0.<InvokeNext>b__0() in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Pipeline\BehaviorChain.cs:line 95
   at NServiceBus.LoadHandlersBehavior.Invoke(IncomingContext context, Action next) in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Unicast\Behaviors\LoadHandlersBehavior.cs:line 45
   at NServiceBus.BehaviorChain`1.InvokeNext(T context) in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Pipeline\BehaviorChain.cs:line 107
   at NServiceBus.BehaviorChain`1.<>c__DisplayClass4_0.<InvokeNext>b__0() in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Pipeline\BehaviorChain.cs:line 95
   at NServiceBus.InvokeSagaNotFoundBehavior.Invoke(IncomingContext context, Action next) in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Unicast\Behaviors\InvokeSagaNotFoundBehavior.cs:line 17
   at NServiceBus.BehaviorChain`1.InvokeNext(T context) in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Pipeline\BehaviorChain.cs:line 107
   at NServiceBus.BehaviorChain`1.<>c__DisplayClass4_0.<InvokeNext>b__0() in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Pipeline\BehaviorChain.cs:line 95
   at NServiceBus.BehaviorChain`1.InvokeNext(T context) in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Pipeline\BehaviorChain.cs:line 107
   at NServiceBus.BehaviorChain`1.<>c__DisplayClass4_0.<InvokeNext>b__0() in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Pipeline\BehaviorChain.cs:line 95
   at NServiceBus.ExecuteLogicalMessagesBehavior.Invoke(IncomingContext context, Action next) in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Unicast\Messages\ExecuteLogicalMessagesBehavior.cs:line 24
   at NServiceBus.BehaviorChain`1.InvokeNext(T context) in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Pipeline\BehaviorChain.cs:line 107
   at NServiceBus.BehaviorChain`1.<>c__DisplayClass4_0.<InvokeNext>b__0() in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Pipeline\BehaviorChain.cs:line 95
   at NServiceBus.BehaviorChain`1.InvokeNext(T context) in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Pipeline\BehaviorChain.cs:line 107
   at NServiceBus.BehaviorChain`1.<>c__DisplayClass4_0.<InvokeNext>b__0() in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Pipeline\BehaviorChain.cs:line 95
   at NServiceBus.BehaviorChain`1.InvokeNext(T context) in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Pipeline\BehaviorChain.cs:line 107
   at NServiceBus.BehaviorChain`1.<>c__DisplayClass4_0.<InvokeNext>b__0() in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Pipeline\BehaviorChain.cs:line 95
   at NServiceBus.BehaviorChain`1.InvokeNext(T context) in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Pipeline\BehaviorChain.cs:line 107
   at NServiceBus.BehaviorChain`1.<>c__DisplayClass4_0.<InvokeNext>b__0() in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Pipeline\BehaviorChain.cs:line 95
   at NServiceBus.BehaviorChain`1.InvokeNext(T context) in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Pipeline\BehaviorChain.cs:line 107
   at NServiceBus.BehaviorChain`1.<>c__DisplayClass4_0.<InvokeNext>b__0() in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Pipeline\BehaviorChain.cs:line 95
   at NServiceBus.UnitOfWorkBehavior.Invoke(IncomingContext context, Action next) in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\UnitOfWork\UnitOfWorkBehavior.cs:line 42
   at NServiceBus.BehaviorChain`1.InvokeNext(T context) in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Pipeline\BehaviorChain.cs:line 107
   at NServiceBus.BehaviorChain`1.<>c__DisplayClass4_0.<InvokeNext>b__0() in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Pipeline\BehaviorChain.cs:line 95
   at NServiceBus.ForwardBehavior.Invoke(IncomingContext context, Action next) in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Unicast\Behaviors\ForwardBehavior.cs:line 19
   at NServiceBus.BehaviorChain`1.InvokeNext(T context) in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Pipeline\BehaviorChain.cs:line 107
   at NServiceBus.BehaviorChain`1.<>c__DisplayClass4_0.<InvokeNext>b__0() in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Pipeline\BehaviorChain.cs:line 95
   at NServiceBus.ChildContainerBehavior.Invoke(IncomingContext context, Action next) in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Unicast\Behaviors\ChildContainerBehavior.cs:line 17
   at NServiceBus.BehaviorChain`1.InvokeNext(T context) in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Pipeline\BehaviorChain.cs:line 107
   at NServiceBus.BehaviorChain`1.<>c__DisplayClass4_0.<InvokeNext>b__0() in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Pipeline\BehaviorChain.cs:line 95
   at NServiceBus.ProcessingStatisticsBehavior.Invoke(IncomingContext context, Action next) in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Monitoring\ProcessingStatisticsBehavior.cs:line 23
   at NServiceBus.BehaviorChain`1.InvokeNext(T context) in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Pipeline\BehaviorChain.cs:line 107
   at NServiceBus.BehaviorChain`1.Invoke() in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Pipeline\BehaviorChain.cs:line 52
   at NServiceBus.Pipeline.PipelineExecutor.Execute[T](BehaviorChain`1 pipelineAction, T context) in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Pipeline\PipelineExecutor.cs:line 129
   at NServiceBus.Unicast.Transport.TransportReceiver.OnTransportMessageReceived(TransportMessage msg) in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Unicast\Transport\TransportReceiver.cs:line 413
   at NServiceBus.Unicast.Transport.TransportReceiver.ProcessMessage(TransportMessage message) in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Unicast\Transport\TransportReceiver.cs:line 344
   at NServiceBus.Unicast.Transport.TransportReceiver.TryProcess(TransportMessage message) in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Unicast\Transport\TransportReceiver.cs:line 230
   at NServiceBus.Azure.Transports.WindowsAzureServiceBus.AzureServiceBusDequeueStrategy.TryProcessMessage(Object obj) in C:\BuildAgent\work\2f57832e2eee436e\src\Transport\Receiving\AzureServiceBusDequeueStrategy.cs:line 137
Request Information
RequestID:f91dcb9a-0002-0016-2cb5-11a71b000000
RequestDate:Wed, 28 Oct 2015 19:16:21 GMT
StatusMessage:Accepted
ErrorCode:OutOfRangeInput

TL;DR:
Table scans in timeouts may impact performance.

Details:
The current approach for querying for next timeouts is following:

if (lastSuccessfulRead.HasValue)
{
    query = from c in timeoutDataTable.CreateQuery<TimeoutDataEntity>()
        where c.PartitionKey.CompareTo(lastSuccessfulRead.Value.ToString(partitionKeyScope)) >= 0
            && c.PartitionKey.CompareTo(now.ToString(partitionKeyScope)) <= 0
            && c.OwningTimeoutManager == endpointName
        select c;
}
else
{
    query = from c in timeoutDataTable.CreateQuery<TimeoutDataEntity>()
            where c.OwningTimeoutManager == endpointName
            select c;
}

link to full TimeoutPersister.cs

The predicates are based on either scanning through partition keys, or scanning over a property. Both queries will issue full table scans which in environments with massive number of timeouts is inefficient.

Proposal
Provide a new design for timeouts based on creating secondary indexes based on the endpointName and the TimeoutData.Time to enable effective look ups by the query.
This is similar to the Cassandra secondary indexes creating additional rows with the key based on an indexed value and a row containing identifiers of index rows.

The algorithm for inserting a timeout would be:

• write the secondary index entry pointing to the TimeoutData.Id
• write the timeout under identifier

The query for a specific period of time would access all the buckets based on the time period + endpointName gathering all the identifiers. In the second step, it'd query for timeouts by their primary key.

Advantages:

• no table scans (you can access do only 20.000 IOPS/account, this includes messages!)

Disadvantages:

• select N+1. It'll require picking timeouts one by one, but this can be done concurrently

Looking forward to your comments: @WilliamBZA @Particular/azure-maintainers

• PartitionRowKeyTuple
  • SecondaryIndexKeyBuilder
  • SecondaryIndexPersister
  • SecondaryIndexTableEntity
  • AzureSagaPersister
  • AzureStorageSagaDefaults
  • AzureSubscriptionStorage
  • Subscription
  • AzureSubscriptionStorageDefaults
  • TimeoutDataEntity
  • TimeoutPersister
  • AzureTimeoutStorageDefaults
  • CloudTableExtensions
  • SafeLinqExtensions

Today you get a One of the request inputs is out of range. (see full ex below) if you have a datetime property where the date is earlier than

64-bit value expressed as Coordinated Universal Time (UTC). The supported DateTime range begins from 12:00 midnight, January 1, 1601 A.D. (C.E.), UTC. The range ends at December 31, 9999.

https://msdn.microsoft.com/en-us/library/azure/dd179338.aspx

Should we consider a better error message?

Connects to Particular/NServiceBus.Azure#309

Issue by yvesgoeleven
10/23/2013 7:57:34 PM +00:00

Originally opened as Particular/NServiceBus.Azure#55

If anything goes wrong it seems to fall back to raven all the time, instead of throwing the exception it is supposed to?

See https://github.com/Particular/PlatformDevelopment/issues/792

This is small enough to fall under Maintainer Prio

Issue by yvesgoeleven
11/27/2013 7:19:48 AM +00:00

Originally opened as Particular/NServiceBus.Azure#76

Experience users trip up on this every once in a while, where they forget to change a connection string and unwillingly use local dev.

On the other hand, removing these would break the F5 experience for first time users.Thoughts?

Connects to: Particular/NServiceBus.AzureStorageQueues#107

It seems that binaries folder is created under /src/ folder, and it should be 1 level up.

We always check the expiry of a databus property, even if that property did not have a TTL associated, maybe that's not needed (or it should not be allowed not to specify a TTL)

In some scenarios this may also cause high memory usage over time. From a customer report

"We didn't have an expiry, so on our environment, we had 33 endpoints all enumerating through the 20,000 storage entries, every 5minutes, to see if they expired and need deleting. Over time this was holding on to memory and resources."

Currently ConditionalWeakTable is used for storing sagas' entities' ETags. With v6 ContextBag could be used as it's passed to every method of ISagaPersister. This would remove the concurrent structure, potentially improving performance and lower the pressure of GC related threads (no need to sweep values from CWT).

See:
https://github.com/Particular/NServiceBus.Persistence.AzureStorage/blob/develop/src/NServiceBus.Persistence.AzureStorage/SagaPersisters/AzureSagaPersister.cs#L21

Currently, each persister requires to be configured independently:

config.UsePersistence<AzureStoragePersistence, StorageType.Subscriptions>().ConnectionString(connectionString);
config.UsePersistence<AzureStoragePersistence, StorageType.Sagas>().ConnectionString(connectionString);
config.UsePersistence<AzureStoragePersistence, StorageType.Timeouts>().ConnectionString(connectionString);

If connection string is the same, should be enough to do the following to configure all persisters

config.UsePersistence<AzureStoragePersistence>().ConnectionString(connectionString);

TODOS:

• Implement
• Document

Need to execute AT for persisters.

Similar to https://github.com/Particular/NServiceBus.AzureServiceBus/blob/develop/packaging/nuget/nservicebus.azureservicebus.nuspec#L22

Failed here: https://builds.particular.net/viewLog.html?buildId=166519&tab=buildResultsDiv&buildTypeId=NServiceBus_Persistence_NServiceBus_AzureStoragePersistence_Build

Passed here: https://builds.particular.net/viewLog.html?buildId=166522&

error:

 Expected: True
  But was:  False

at NServiceBus.Persistence.AzureStorage.ComponentTests.Timeouts.When_removing_timeouts_from_the_storage.<TryRemove_should_work_with_concurrent_operations>d__7.MoveNext() in C:\Build\src\NServiceBus.Persistence.AzureStorage.ComponentTests\Timeouts\When_removing_timeouts_from_the_storage.cs:line 140
--- End of stack trace from previous location where exception was thrown ---
at NUnit.Framework.Internal.ExceptionHelper.Rethrow(Exception exception)
at NUnit.Framework.Internal.AsyncInvocationRegion.AsyncTaskInvocationRegion.WaitForPendingOperationsToComplete(Object invocationResult)
at NUnit.Framework.Internal.Commands.TestMethodCommand.RunAsyncTestMethod(TestExecutionContext context)
« Hide stacktrace

The test that was supposed to be fixed in this PR fails on the build server. It doesn't fail because of the assertion (as it is all green). It fails because of some messages left marked as unprocessed by FailTestOnErrorMessageFeature.

The exception that is thrown on stopping the endpoint:

System.Exception : Some failed messages were not handled by the recoverability feature.

/cc @timbussmann

Issue by SeanFeldman
3/11/2015 7:43:46 PM +00:00

Originally opened as Particular/NServiceBus.Azure#255

Current implementation of Timeouts and Subscription storages rely on TableServiceContext that is obsoleted and needs to be replaced with CloudTable

Related issue: Particular/NServiceBus.AzureStorageQueues#6

/cc @yvesgoeleven

In v6 assignments like this are no longer needed:

Data.OrderId = message.CorrelationId;

in SecondaryIndexPersister we have

var exec = await table.ExecuteAsync(TableOperation.Retrieve<SecondaryIndexTableEntity>(key.PartitionKey, key.RowKey))
                .ConfigureAwait(false);
var secondaryIndexEntry = exec.Result as SecondaryIndexTableEntity;

I assume exec.Result can only ever be a SecondaryIndexTableEntity so it should be a direct cast not an as

Upgrade to latest stable of the WindowsAzure.Storage dependency, range 6.0.0 to 7.0.0.

Note that some functionality that we use may have been obsoleted, and will cause a rewrite of certain functionality.

• Timeouts and Subscription storages rely on TableServiceContext that is obsoleted #5
• Docs to include:
  • Add configuration for TimeoutStateBlobName

From customer question:

If we had a TimeToLive of 7days for databus properties, does it need to check every 5 minutes if that 7 days has expired?

From a suggestion on the google groups:

To add logic to the saga persister that unsupported datatypes would be json serialized and stored, instead of throwing a runtime error.

Also asked in a more recent GG post

Copied from the original issue:
Particular/docs.particular.net#1666

Currently “NServiceBus/Persistence” is not used by Azure Storage persisters (AzureSubscriptionStorageConfig, AzureSagaPersisterConfig, AzureTimeoutPersisterConfig) as default connection string causing users to be confused (Desk case for example)

Also, this forces for scenarios where all persisters use the same storage account to define all three sections and replicate connection string for each.

Imagine having the the following sagas:

public class MyFirstCoolSaga
{
    public class Data : ContainSagaData
    {
        public long Id { get; set; }
    }
}

public class MySecondEvenCoolerSata
{
    public class Data : ContainSagaData
    {
        public long Id { get; set; }
    }
}

Both saga data classes will now store their data in Data. This can result in issues.

Comparing this to NHibernate, NHibernate uses the name from the parent/outer class. The sagas seen in the following image use the same construction as the above mentioned snippet.

Proposal

1. Use the same behavior as NHibernate, use the outer class name
2. Use the outer class name and the nested class name
3. User the full type name (namespace + outer + nested names)

References

• This issue was found while implementing Particular/EndToEnd#198