im5tu / openmessage Goto Github PK

When using the awaitable in memory dispatcher and an exception is thrown, then that exception is not raised to the awaiting caller

            catch(Exception e)
            {
                if (message is ISupportAcknowledgement acknowledgement)
                    await acknowledgement.AcknowledgeAsync(false, e);

                throw;
            }

Need some sample projects for the following:

• Multiple Serializers
• Azure Service Bus
• Azure Event Hubs
• Protobuf Serializer
• In Memory Channel
• Service Fabric Queue

For greater visibility, this log message should include the queue name

Dependabot couldn't parse the global.json found at /global.json.

The error Dependabot encountered was:

Dependabot::DependencyFileNotParseable

For testing purposes, it is handy to insert middleware at the beginning of a pipeline. At the moment we only support adding to the end

We should consider an extension in the testing package to allow insert middleware at the beginning (or anywhere in the existing pipeline)

Message pump retry behaviour should be configurable. At the moment it performs an immediate retry in a while loop which causes the pump to hammer the service if it is not available:

The behaviour may need to be lifted up into MessagePump<T> and should ideally make use of polly policies

https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_DeleteMessageBatch.html

In a unit test, when running .ConfigureAllHandlers(), then IHandler<> types are not found because the entry assembly is the test host:

testhost, Version=15.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a

We should consider using a different assembly (Assembly.GetExecutingAssembly()) or take in a specific class .ConfigureAllHandlers<Program>()

Logging the name of the queue that cannot be found would be super helpful in troubleshooting the error. Currently we get the below error

fail: OpenMessage.AWS.SQS.SqsMessagePump[0]
      AWS.SimpleQueueService.NonExistentQueue; see the SQS docs.
Amazon.SQS.AmazonSQSException: AWS.SimpleQueueService.NonExistentQueue; see the SQS docs.

Given we are dispatching a type Person

IDispatcher<Person> personDispatcher = GetDispatcher();

But my service code publishes an internal derived type PersonWithProperties

private class PersonWithProperties : ISupportProperties { }

await personDispatcher .DispatchAsync(new PersonWithProperties())

Then PersonHandler is unable to deserialize the type to PersonWithProperties because it is private to the dispatcher code:

In this case, where we cant find the type of the message, should we deserialize it to the handler type (Person) instead?

Replaces #5

See if we can leverage service fabric's reliable queues as a provider: https://azure.microsoft.com/en-gb/documentation/articles/service-fabric-reliable-services-reliable-collections/

Create an in memory channel that passes straight through without serialization.

When you dispatch a message to SQS, then it will automatically wrap the message like this:

{
    "MessageId": "23bfcdfd-ebfb-4402-8fa3-6d4289c3ec29",
    "ReceiptHandle": "23bfcdfd-ebfb-4402-8fa3-6d4289c3ec29#d71f87d5-a09f-4dbe-8a5d-c7c0fcbe65e3",
    "MD5OfBody": "33ef45e587fd5b08bdbc3d802b494f27",
    "Body": "{\"Property\": null}"
}

When you dispatch a message to SNS, then it will automatically wrap the message like this:

{
    "MessageId": "23bfcdfd-ebfb-4402-8fa3-6d4289c3ec29",
    "Type": "Notification",
    "Timestamp": "2019-12-12T17:56:40.209374Z",
    "Message": "{\"Property\": null}"
}

However, when you have an SQS that is subscribed to an SNS then it will wrap the message twice:

{
    "MessageId": "23bfcdfd-ebfb-4402-8fa3-6d4289c3ec29",
    "ReceiptHandle": "23bfcdfd-ebfb-4402-8fa3-6d4289c3ec29#d71f87d5-a09f-4dbe-8a5d-c7c0fcbe65e3",
    "MD5OfBody": "33ef45e587fd5b08bdbc3d802b494f27",
    "Body": "{\"MessageId\": \"c690b2be-2513-4cce-bb9d-ad7a4bf60db6\", \"Type\": \"Notification\", \"Timestamp\": \"2019-12-12T17:34:31.218142Z\", \"Message\":\"\\\"Property\\\":null\" }"
}

Should we automatically un-wrap these messages?

There will be a slight performance hit where we will need to parse the message first to work out if it is wrapped. There is also a chance that we attempt to unwrap a message that 'looks' like the wrapper

There is an issue with MessageAttributes being 'lost'

The SNS message attributes are not automatically pulled over to SQS. This means that any kind of correlation will not flow through the system nicely

If an exception is thrown in the pipeline then AwaitableMessage<T> will simply return

A potential fix is to modify ISupportAcknowledgement.AcknowledgeAsync(bool success) to also accept an exception/object/dictionary

Looking at the following code:

Assuming that the extension is always used, the if check can never be true as we add the interface IBroker to the service collection as the service type. The secondary part to the check should be ImplementationType.

I think LoggerScopeMiddleware<T> is pointless:

https://github.com/Im5tu/OpenMessage/blob/1e0931040e23ee96f892f0a8bbfcff3da4f98884/src/OpenMessage/Pipelines/Middleware/LoggerScopeMiddleware.cs

The scope only applies when using the same logger. If you request a new logger from the container, it wont container any scopes

It's been a long while since this project got any love, so it's time to embark on a big upgrade of the components to modern thinking.

Design

• Each provider will be constructed as a unique hosted service;
• Each provider publishes a new Message type to a central channel, simplifying the current design;
• Consumer of the new Message type will have standardised extension points and
• Message type:
  • Abstract by design so that consumers can register capabilities, eg: INegativelyAck, IDeferable etc.

Providers

Since the original use case for the library existing has moved on. It's worth taking a look at newer providers and getting them into the ecosystem.

• Apache Kafka #21 (NEW!)
• AWS SQS #23 (NEW!)
• AWS SNS (NEW!)
• AWS Kinesis (NEW!)
• AWS EventBridge (NEW!)
• Azure Event Hubs #27
• Azure Service Bus #28
• InMemory
• NATS #25 (NEW!)
• RabbitMq #22 (NEW!)

Serializers

The serializers are massively out of date and could do with an upgrade. Provide integrations for the following:

• Hyperion (NEW!)
• Jil
• JsonDotNet
• MessagePack (NEW!)
• MsgPack (NEW!)
• Protobuf (NEW!)
• ServiceStackJson (NEW!)
• Utf8Json (NEW!)
• Wire (NEW!)

Sending/Receiving over TCP. TLS must be supported.

To go along with the Service Bus provider, we should also have one for Event Hubs

Consider raising events related to callbacks instead of configuring on the container and expose as part of IBroker(T)

As part of the samples, I realised that we will end up doing a lot of the following:

services.GetRequiredService<IEnumerable<IBroker>>();

and having to self manage the disposal. Need to come up with a better pattern for this eg:

interface IBrokerHost : IDisposable {}

Sending/Receiving over UDP