ParseInboundEmailWebhookAsync doesn't work when encoding is iso-8859-1 about stronggrid HOT 11 CLOSED

I'm curious: what led you to conclude the encoding was the source of the problem?

Also, are you able to capture the HTTP payload sent to you by SendGrid and post it here? That would allow me to try to reproduce the issue.

from stronggrid.

outlook_payload.txt
Please see an attached json from InboundEmail object. which were generated from payload from sendgrid. I modified email addresses from it because of security.

During debugging when I changed encoding name to UTF-8, the method didn't freeze on me. So I realized that encoding is causing an issue in async version of this method.

Thanks

from stronggrid.

For future reference, please refrain from converting the payload you receive from SendGrid to JSON or to any other format. Makes it nearly impossible for me to use it to attempt to reproduce the issue you are reporting. What I need is the actual data you receive. Nothing else is really useful. If you need help saving the actual data you receive from SendGrid, I have provided some sample C# code that will allow you to do that here.

Having said that, I did my best to convert the JSON you provided back into the original payload, and I wrote a unit test to validate that we can parse this payload:

[Fact]
public async Task InboundEmailAsync_bsohi()
{
	// Arrange
	var parser = new WebhookParser();
	using (var stream = File.OpenRead(@"C:\temp\original_payload.txt"))
	{
		// Act
		var inboundEmail = await parser.ParseInboundEmailWebhookAsync(stream).ConfigureAwait(false);

		// Assert
		inboundEmail.Charsets.ShouldNotBeNull();
		inboundEmail.Charsets.Length.ShouldBe(5);
		inboundEmail.Charsets.ShouldContain(new KeyValuePair<string, string>("html", "iso-8859-1"));
		inboundEmail.Charsets.ShouldContain(new KeyValuePair<string, string>("text", "iso-8859-1"));
	}
}

The unit test completes successfully.

So, at this point, I can't reproduce the problem you are reporting. I suggest you use the unit test I provided above to see if you can reproduce.

from stronggrid.

Hi Jericho,

Sorry about that. Please see an attached payload which I captured when sending an email from Outlook email client.

It failed with async method

thanks for quickly looking into this.
outlook_payload_1.txt

Thank you

from stronggrid.

Excellent, this is exactly the payload I needed.

However, I am still unable to reproduce. My unit test completes successfully with this payload you just provided:

from stronggrid.

This is weird.. When I try to run test case through console app it works but when it is used in our API project it doens't work.

I compared the dll reference and it is same in both projects.

Not sure where to look further. Any suggestion?
Thanks

from stronggrid.

I have a theory: what if the problem you are experiencing is not so much due to the specific encoding (iso-8859-1 in your situation) but rather to the fact that the payload you receive from SendGrid contains more than one encoding.

If my theory is right, you could change the "iso" encoding in your payload to "UTF-8", thus reducing the number of encodings in your payload to only one, and the problem would go away. Conversely, you could change the "iso" encoding to the name of any other supported encoding (such as us-ascii or IBM870 or iso-8859-5 for example) and you would continue to experience the problem. In your original comment, you mentioned that you already tried changing the encoding to UTF-8 and the problem went away which seems to indicate that my theory might be correct. Can you please try to change the encoding to other values and confirm to me that you are still experiencing the problem?

The reason why I came up with this theory is that you provided the line of code in the StrongGrid library where the execution seems to be stuck. This line of code is where StrongGrid asynchronously parses the payload using each of the unique encodings in your payload and, importantly, it does so by reading the payload stream for each encoding. Maybe the asynchronous tasks are deadlocked because they try to read from the stream at the same time. The same logic is executed in a synchronous way (meaning one at a time) in ParseInboundEmailWebhook which would explain why you are not experiencing the same problem when using the non-async method.

Keep in mind that this is just a theory at this point. I don't have a way to reproduce so I can't really prove or disprove my theory. Also, one thing that is bothering me is that I can't explain why this is a problem in your API project but you and I are unable to reproduce in any other environment.

If you help me confirm that my theory is correct, I'll figure out a way to avoid the deadlock problem and provide you a beta nuget package to test.

from stronggrid.

I have another theory: your API controller probably looks similar to this right?

[HttpPost]
public async Task<IActionResult> Post()
{
    var parser = new StrongGrid.WebhookParser();
    var inboundEmail = await parser.ParseInboundEmailWebhookAsync(Request.Body).ConfigureAwait(false);

    // do something with the parsed email
            
    return Ok();
}

Maybe the issue is that we are trying to repeatedly read the content of the request body? (remember, we read it once for each encoding)

Maybe it would be preferable for you to read the content of the body once and make a copy of it. You would then pass the copy to the StrongGrid parser which would subsequently read from this copy rather than the Request.Body.

Should be something similar to this:

[HttpPost]
public async Task<IActionResult> Post()
{
    // Make a copy of the payload which the StrongGrid parser will be able to read multiple times
    var duplicateStream = new MemoryStream();
    Request.Body.CopyTo(duplicateStream);
    
    var parser = new StrongGrid.WebhookParser();
    var inboundEmail = await parser.ParseInboundEmailWebhookAsync(duplicateStream).ConfigureAwait(false);

    // do something with the parsed email
    
    return Ok();
}

from stronggrid.

Thanks for providing me different approach.

Copying stream to memory stream before calling ParseInboundEmailWebhookAsync did the trick.

Thanks for all the help.

one more thing, Do we need to call ConfigureAwait on the method?

from stronggrid.

I'm glad one of my suggestions helped.

Regarding your question about ConfigureAwait: let me say two things:

1. It is considered best practice to always call ConfigureAwait when invoking an asynchronous method. I personally make a point of always calling it whenever I await an async call.
2. Are you asking because you are not calling it and you are unsure whether you should? If so, maybe that's the root of the problem you experienced and maybe copying the response content to a memory stream would be unnecessary if you made sure to call ConfigureAwait. Let me know.

from stronggrid.

Hi,

That was my first step to use ConfigureAwait but it was not working either.

you can close the issue.
Thank you
Balwinder

from stronggrid.