streamText with image data for Anthropic on Edge Runtime fires Illegal invocation error about ai HOT 9 CLOSED

globalThis.atob should be available on the edge runtime: https://vercel.com/docs/functions/runtimes/edge-runtime

@twxxk how are you invoking the code example? Inside a route for the app router?

from ai.

Hi @twxxk, I am facing the exact same issue.

When I try to send an image using streamText (ai version 3.1.8) in my app router, it works in the local and non-edge environments, but not with edge runtime.

I convert my image to Uint8Array (which seems to be what the ai source code does in the background) before appending it to the messages array. However, in my deployment logs, I get:

[POST] /api/chat reason=EDGE_FUNCTION_INVOCATION_FAILED, status=500, user_error=true
Status Code 405 Method Not Allowed

Would you mind sharing any tips on solving this issue?
I would appreciate it!

from ai.

@ttanida can you share you client & server code?

from ai.

Hi @lgrammel,

thanks for taking a look at this!
I used:

[email protected]
ai-sdk/[email protected]

On the client-side, I take a screenshot of my screen using:

canvasElement.width = videoElement.videoWidth;
canvasElement.height = videoElement.videoHeight;
canvasContext.drawImage(
      videoElement,
      0,
      0,
      canvasElement.width,
      canvasElement.height
    );

const imageUrl = canvasElement.toDataURL("image/jpeg", 1); // Quality range is 0 to 1

The screen shot imageUrl is then sent to my chat API inside the messages array.
On server-side, my app/api/chat/route.js looks like this:

import { openai } from "@ai-sdk/openai";
import { StreamingTextResponse, streamText } from "ai";

export const runtime = "edge";
const model = "gpt-4o";

function dataURLToUint8Array(dataURL) {
  const base64String = dataURL.split(",")[1]; // Remove the Data URL prefix
  const buffer = Buffer.from(base64String, "base64");
  return new Uint8Array(buffer);
}

export async function POST(req) {
  const { messages } = await req.json();

  let lastElement = messages[messages.length - 1];
  if (Array.isArray(lastElement.content)) {
    const imageUrl = lastElement.content[1].image;
    const firstPart = imageUrl.slice(0, 40);
    const lastPart = imageUrl.slice(-40);
    console.log(`${firstPart}...${lastPart}`);

    // Convert the Data URL to Uint8Array
    lastElement.content[1].image = dataURLToUint8Array(
      lastElement.content[1].image
    );
  }

  const result = await streamText({
    model: openai(model),
    system: "You are a helpful assistant.",
    messages,
  });

  return new StreamingTextResponse(result.toAIStream());
}

The logged imageUrl looks like this:

data:image/jpeg;base64,/9j/4AAQSkZJRgABA...Jh6tUF9JricpNomQZsLLz+Lw1GXezv4bj8z6r//Z

The messages array looks like this:

[
  {
  role: 'assistant',
  content: 'Hello Tim! How can I help you today?'
},
  {
  role: 'user',
  content: [
  { type: 'text', text: 'Tell me what you see in the image.’ },
  { type: 'image', image: theImage }
]
}
]

where theImage is the output of dataURLToUint8Array.

The chat works perfectly (both with and without images present in the messages array) when executed locally or non-edge run time (i.e. commenting out line "export const runtime = "edge";").

The chat works perfectly in edge run time only when no images are present in messages.
When an image is present (like the above example), then I get the warning:

[POST] /api/chat reason=EDGE_FUNCTION_INVOCATION_FAILED, status=500, user_error=true

Status Code
405
Method Not Allowed

and the error:

Error:

TypeError: Right-hand side of 'instanceof' is not an object
    at (node_modules/.pnpm/[email protected][email protected][email protected][email protected][email protected][email protected][email protected]/node_modules/ai/dist/index.mjs:34:20)

However, the error seems to me more like the symptome of the underlying issue and not the cause?

I would appreciate if you could help me solve this issue!

from ai.

Having same issue - sending image data to OpenAI gpt-4-turbo works locally, but not on Vercel using Edge.

Sending messages without image data or an image URL streams fine on Vercel without any other changes.

"ai": "3.1.12",
"@ai-sdk/anthropic": "0.0.14",
"@ai-sdk/google": "0.0.14",
"@ai-sdk/openai": "0.0.13",

Seeing in Vercel logs:

[POST] /api/chat reason=EDGE_FUNCTION_INVOCATION_FAILED, status=500, user_error=true

TypeError: Right-hand side of 'instanceof' is not an object
    at (../../node_modules/@ai-sdk/provider-utils/dist/index.mjs:33:0)
    at (../../node_modules/ai/dist/index.mjs:34:20)
    at (../../node_modules/ai/dist/index.mjs:21:6)
    at (../../node_modules/ai/dist/index.mjs:1456:42)
    at (src/app/api/chat/route.ts:121:34)
    at (../../node_modules/next/dist/esm/server/future/route-modules/app-route/module.js:207:0)
    at (../../node_modules/next/dist/esm/server/future/route-modules/app-route/module.js:122:0)
    at (../../node_modules/next/dist/esm/server/future/route-modules/app-route/module.js:269:0)
    at (../../node_modules/next/dist/esm/server/web/edge-route-module-wrapper.js:81:0)
    at (../../node_modules/next/dist/esm/server/web/adapter.js?4b18:158:0)

from ai.

Yes, I also get 'instanceof' error with 3.1.12.
Workaround: If you disable runtime = 'edge', the issue does not happen as ttanida said.

I would also mention the official document says the function accepts the image parameter as ArrayBuffer | Uint8Array | Buffer | URL but passing the URL to Anthropic throws "URL image parts' functionality not supported."

from ai.

Hello @lgrammel,

are there any updates on this (maybe the issue should be reopened)?

To make processing image data on the edge possible, I am currently using a previous version of ai:

[email protected]
ai-sdk/[email protected]

However, if possible, I would like to update to the latest version.

from ai.

There are several issues going on here.

@twxxk Anthropic does not support URL image parts. #1817

@lucasishuman I've traced this to

// src/is-abort-error.ts
function isAbortError(error) {
  return error instanceof DOMException && (error.name === "AbortError" || error.name === "TimeoutError"); // line 33
}

my hunch is that DOMException is not available as an object on Edge. The Edge docs say it's available tho.

from ai.

Thanks for the additional info - I disabled 'edge' and that unblocked me for now.

from ai.