Gemini 1.5 Pro via Vertex AI API gives StatusCode="Unavailable" System.IO.IOException about dotnet-docs-samples HOT 13 CLOSED

@jskeet I reproduced the issue @fatihyildizhan reported.

If you change the model name in GeminiQuickstart to this:

        string model = "gemini-1.5-pro-preview-0409" //"gemini-1.0-pro-vision"

And run the sample, it will fail most of the time with this error:

[xUnit.net 00:00:08.11]     GeminiQuickstartTest.TestGenerateContentAsync [FAIL]
  Failed GeminiQuickstartTest.TestGenerateContentAsync [7 s]
  Error Message:
   Grpc.Core.RpcException : Status(StatusCode="Unavailable", Detail="Error reading next message. IOException: The request was aborted. IOException: The response ended prematurely while waiting for the next frame from the server.", DebugException="System.IO.IOException: The request was aborted.")
---- System.IO.IOException : The request was aborted.
-------- System.IO.IOException : The response ended prematurely while waiting for the next frame from the server.
  Stack Trace:
     at Grpc.Net.Client.Internal.HttpContentClientStreamReader`2.MoveNextCore(CancellationToken cancellationToken)
   at Google.Api.Gax.Grpc.AsyncResponseStream`1.MoveNextAsync(CancellationToken cancellationToken)
   at GeminiQuickstart.GenerateContent(String projectId, String location, String publisher, String model) in /Users/atamel/dev/github/meteatamel/dotnet-docs-samples/aiplatform/api/AIPlatform.Samples/GeminiQuickstart.cs:line 82
   at GeminiQuickstart.GenerateContent(String projectId, String location, String publisher, String model) in /Users/atamel/dev/github/meteatamel/dotnet-docs-samples/aiplatform/api/AIPlatform.Samples/GeminiQuickstart.cs:line 82
   at GeminiQuickstartTest.TestGenerateContentAsync() in /Users/atamel/dev/github/meteatamel/dotnet-docs-samples/aiplatform/api/AIPlatform.Samples.Tests/GeminiQuickstartTest.cs:line 35
--- End of stack trace from previous location ---
----- Inner Stack Trace -----
   at System.Net.Http.Http2Connection.ThrowRequestAborted(Exception innerException)
   at System.Net.Http.Http2Connection.Http2Stream.CheckResponseBodyState()
   at System.Net.Http.Http2Connection.Http2Stream.TryReadFromBuffer(Span`1 buffer, Boolean partOfSyncRead)
   at System.Net.Http.Http2Connection.Http2Stream.ReadDataAsync(Memory`1 buffer, HttpResponseMessage responseMessage, CancellationToken cancellationToken)
   at Grpc.Net.Client.Internal.StreamExtensions.ReadMessageAsync[TResponse](Stream responseStream, GrpcCall call, Func`2 deserializer, String grpcEncoding, Boolean singleMessage, CancellationToken cancellationToken)
   at Grpc.Net.Client.Internal.GrpcCall`2.ReadMessageAsync(Stream responseStream, String grpcEncoding, Boolean singleMessage, CancellationToken cancellationToken)
   at Grpc.Net.Client.Internal.HttpContentClientStreamReader`2.MoveNextCore(CancellationToken cancellationToken)
----- Inner Stack Trace -----
   at System.Net.Http.Http2Connection.<ReadFrameAsync>g__ThrowMissingFrame|57_1()
   at System.Net.Http.Http2Connection.ReadFrameAsync(Boolean initialFrame)
   at System.Net.Http.Http2Connection.ProcessIncomingFramesAsync()

Once in a while, the sample works fine (maybe 1 out of 5 or so). There seems to be a problem between the GAPIC library and the Gemini 1.5 endpoint when streaming is being used.

I don't know how to further debug this, so assigning back to you for recommendations.

from dotnet-docs-samples.

I believe this is related to grpc/grpc-dotnet#2358

Two workarounds - these are alternatives to each other:

• You can specify an app switch to disable dynamic window sizing:
  • AppContext.SetSwitch("System.Net.SocketsHttpHandler.Http2FlowControl.DisableDynamicWindowSizing", true);
• You can use Grpc.Core instead of Grpc.Net.Client:
  • Add a dependency to Grpc.Core version 2.46.6
  • Add a using directive for Google.Api.Gax.Grpc
  • In the PredictionServiceClientBuilder object initializer, add GrpcAdapter = GrpcCoreAdapter.Instance

from dotnet-docs-samples.

@fatihyildizhan Is it always IOException or does it work once in a while?

from dotnet-docs-samples.

@fatihyildizhan Is it always IOException or does it work once in a while?

Hi, thanks for the quick response.

• When I try 10 times it completes maybe 1 time.
• It comes up with a few sentences stream then throws this error.

Non-stream option always work.

 var response = await predictionServiceClient.GenerateContentAsync(generateContentRequest);
 string? fullText = response.Candidates.FirstOrDefault()?.Content.Parts.FirstOrDefault()?.Text;

I've made some requests.

from dotnet-docs-samples.

Thanks @jskeet. I tested AppContext.SetSwitch and it fixed the issue for me. @fatihyildizhan, let us know if the workaround works for you as well.

from dotnet-docs-samples.

Hi @meteatamel and @jskeet I've tried both workarounds but unfortunately didn't solve the issue. If the response is very short then there is no problem. But, for longer responses it still crashes.

I think AsyncResponseStream is already using Grpc.Core.

AsyncResponseStream<GenerateContentResponse> responseStream = response.GetResponseStream();

from dotnet-docs-samples.

What platform are you using? Note that your red arrow is pointing to the .NET Framework version. If you're using .NET 6+, you won't be using that version.

AsyncResponseStream uses types from the Grpc.Core namespace, but that's not the same as it using the Grpc.Core package.

Do you have a way of reproducing this, other than what's in the sample? (Both workarounds worked perfectly for me, running them multiple times.)

from dotnet-docs-samples.

We are using .NET 7

Mostly first 4 or 5 requests work fine then I see this exception.

@jskeet Is it possible to share full code like the examples here: https://github.com/GoogleCloudPlatform/dotnet-docs-samples/blob/main/aiplatform/api/AIPlatform.Samples/GeminiQuickstart.cs

Maybe I'm missing some point. I don't want to take up any more of your time either. Thank you very much for following the topic.

from dotnet-docs-samples.

We are using .NET 7

That's good to know - I'll try that. (I've tried both .NET 6 and .NET 8, and they both work with both workarounds.)

@jskeet Is it possible to share full code like the examples here

I'm not sure what you mean. Is that the precise sample code you're using, or have you modified it? The tests I ran we basically calling that exact code, just passing in my own project ID. If you're using a slightly different request, it would be good to know that difference.

from dotnet-docs-samples.

As an extra data point, I just ran the GeminiQuickstart.cs 10 times in a row on .NET 6 and it worked fine for me every single time. So, @fatihyildizhan, it'd be great to get your sample that fails once in a while and see how it's different from GeminiQuickstart.cs.

from dotnet-docs-samples.

I've just tried the sample code with .NET 7, without the GrpcAdapter change, but with the AppContext.SetSwitch change, and it's worked perfectly.

from dotnet-docs-samples.

I use exactly the same example as in GeminiQuickstart.cs.

I think the problem is specific to us for some reason. No one else seems to have this problem. For this reason, you can close this issue.

Thank you very much for your time.

from dotnet-docs-samples.

Okay - I'm still confused, but happy to close the issue. You might want to try updating to .NET 8 and see if that helps, too. (After all, .NET 7 reaches end of life in just over 2 weeks.)

from dotnet-docs-samples.