3.0.20 create a breaking change for `StreamingTextResponse` about ai HOT 32 OPEN

Update on solutions:

• if you use useChat or useCompletion, it will continue to work. If you see issues, it's most likely caused by a mismatch of the client and server version of the AI SDK, or you need to rebuild/refresh
• If you use your own client-side code, you can
  • pin the version of the AI SDK to 3.0.19
  • or use readDataStream (see #1316 (comment) )
  • or switch to AI Core on the server side and use streamText with toTextStreamResponse (v3.0.21 and newer)

    export async function POST(req: Request) {
      const { prompt } = await req.json();
    
      const result = await experimental_streamText({
        model: openai.completion('gpt-3.5-turbo-instruct'),
        maxTokens: 2000,
        prompt,
      });
    
      return result.toTextStreamResponse();
    }

    More on streamText: https://sdk.vercel.ai/docs/ai-core/stream-text

from ai.

Vercel AI SDK demo is suffering from this

from ai.

Update on solutions:

• if you use useChat or useCompletion, it will continue to work. If you see issues, it's most likely caused by a mismatch of the client and server version of the AI SDK, or you need to rebuild/refresh
• If you use your own server-side code, you can
  • pin the version of the AI SDK to 3.0.19
  • use streamMode: "text" in useChat / useCompletion (requires v3.0.23)
• If you use your own client-side code, you can
  • pin the version of the AI SDK to 3.0.19
  • or use readDataStream (see #1316 (comment) )
  • or switch to AI Core on the server side and use streamText with toTextStreamResponse (v3.0.21 and newer)

    export async function POST(req: Request) {
      const { prompt } = await req.json();
    
      const result = await experimental_streamText({
        model: openai.completion('gpt-3.5-turbo-instruct'),
        maxTokens: 2000,
        prompt,
      });
    
      return result.toTextStreamResponse();
    }

    More on streamText: https://sdk.vercel.ai/docs/ai-sdk-core/generating-text#streaming-text

from ai.

I agree. This was not intended to be a breaking change. StreamingTextResponse is meant to be used by useChat/useCompletion, and your use case unfortunately is outside the envisioned use. Because of some other changes that we are currently making (splitting out packages), it is sadly not possible to revert at the moment.

I suggest either pinning v3.0.19 for the time being or moving to the stream data protocol.

We'll include the simple text stream use case in the new streamText API.

from ai.

I agree with the new protocol to support more types of messages and actually LobeChat needs it too.

But I don't think the StreamingTextResponse should be changed in this PATCH version. It's a really breaking change for others depend on the ai sdk.

Even use major version, I still think there is demand on pure text streaming response. Other library even has document how to intergate with StreamingTextResponse like pro-chat. Most users are just need a text output. If I need to handle with the protocol, why to use this StreamingTextResponse?

I really suggestion another method to add with the new protocol, like StreamingChatResponse and make the StreamingTextResponse still pure text output.

from ai.

Difficult to understand, even if it should be used in useChat, I also think this is a bug because from the literal meaning, we would think that StreamingTextResponse is to convert Stream into text... rather than adding 0: 1:.

from ai.

I see. We have switched to a new protocol that supports different types of messages (tool calls, etc). This provides the foundation to robustly support richer LLM functionality such as tool calling, annotations, etc.

If you use StreamData without useChat / useCompletion, you need to update your code to the new protocol.

In the stream data protocol, each message is a line with a type (indicated by a number) separated from the value.

E.g. 0:some chunk is a text chunk some chunk. (message types: https://github.com/vercel/ai/blob/main/packages/core/shared/stream-parts.ts )

With readDataStream you can parse the chunks: https://github.com/vercel/ai/blob/main/packages/core/shared/read-data-stream.ts

Example usage: https://github.com/vercel/ai/blob/main/packages/core/shared/parse-complex-response.ts#L52

For the basic use case that resembles the original text streaming, you just need to process text parts.

 for await (const { type, value } of readDataStream(reader, {
    isAborted: () => abortControllerRef?.current === null,
  })) {
    if (type === 'text') {
       doSomething(value);
    }
}

from ai.

Ok, I will pin to v3.0.19 currently. By the way, is there any document about the new stream data protocol? I will need it when I working on the claude function call feature.

from ai.

@martgra If I understand correctly, you are returning custom text chunks. In the new protocol, text chunks are lines that look like this 0:my-chunk\n. You can use formatStreamPart to achieve this, e.g. formatStreamPart("text", delta) (see https://github.com/vercel/ai/blob/main/packages/core/shared/stream-parts.ts#L363 ). Please be aware that this is an internal API though.

Suspected something like this :) Now it streams!

from ai.

this is how I use the Response of StreamingTextResponse.

/**
 * Fetch data using stream method
 */
export const fetchSSE = async (fetchFn: () => Promise<Response>, options: FetchSSEOptions = {}) => {
  const response = await fetchFn();

  if (!response.ok) {
    const chatMessageError = await getMessageError(response);

    options.onErrorHandle?.(chatMessageError);
    return;
  }

  const returnRes = response.clone();

  const data = response.body;

  if (!data) return;
  let output = '';
  const reader = data.getReader();
  const decoder = new TextDecoder();

  let done = false;
  let finishedType: SSEFinishType = 'done';

  while (!done) {
    try {
      const { value, done: doneReading } = await reader.read();
      done = doneReading;
      const chunkValue = decoder.decode(value, { stream: true });

      output += chunkValue;
      options.onMessageHandle?.(chunkValue);
    } catch (error) {
      done = true;

      if ((error as TypeError).name === 'AbortError') {
        finishedType = 'abort';
        options?.onAbort?.(output);
      } else {
        finishedType = 'error';
        console.error(error);
      }
    }
  }

  await options?.onFinish?.(output, { observationId, traceId, type: finishedType });

  return returnRes;
};

I don't use useChat and useCompletion because of I need lots of customization and hooks are too weak for lobe-chat's usage. I just use StreamingTextResponse to simplify the handle of openai 's response.

from ai.

There is no documentation yet other than the code. It was intended to be internal - so pinning the version will be important if you use it. Given your use case, it makes sense to standardize and document it, so we'll look into it.

PR for future raw text stream response: #1318

from ai.

That's a nice direction to standardize the data stream protocol. I had just used regex to match the function call output and it can't be extandible. I'm glad if there is something more powerful then pure text.

You can check our document If you are interested in the LobeChat Plugin. And the function call I implemented is here: https://github.com/lobehub/lobe-chat/blob/main/src/store/chat/slices/plugin/action.ts#L181-L264

from ai.

Hi @lgrammel Would be great to have some updated documentation on this. I struggle with the AIStream - as it stopped working when moving to 3.0.21. I use useChat.

from ai.

Update on solutions:

• if you use useChat or useCompletion, it will continue to work. If you see issues, it's most likely caused by a mismatch of the client and server version of the AI SDK, or you need to rebuild/refresh

• If you use your own client-side code, you can

  • pin the version of the AI SDK to 3.0.19

  • or use readDataStream (see 3.0.20 create a breaking change for StreamingTextResponse #1316 (comment) )

  • or switch to AI Core on the server side and use streamText with toTextStreamResponse (v3.0.21 and newer)

    export async function POST(req: Request) {
      const { prompt } = await req.json();
    
      const result = await experimental_streamText({
        model: openai.completion('gpt-3.5-turbo-instruct'),
        maxTokens: 2000,
        prompt,
      });
    
      return result.toTextStreamResponse();
    }

    More on streamText: https://sdk.vercel.ai/docs/ai-core/stream-text

Matching the versions to '3.0.21' on both client and server SDK did fix the formatting issue. Thanks!

from ai.

IIUC it's not possible to use readDataStream because it's not exported, is this correct?

Thanks - I'll work on getting it exported.

Update: #1334 (published in v3.0.22)

from ai.

@lgrammel may I ask if useChat sometime will use the SSE protocol directly? Seems like this text stream is less flexible than actually including more metadata in the stream like providers do?

Possibly. For the time being we have no such plans though.

Allright - thanks for answer. I do actually like this useChat better than the GenAI aiState and UIState framework. Its so lightweight and easy to use.

from ai.

@flexchar I understand that this is frustrating, and I agree it's unfortunate. It was not an intentional breakage, and unfortunately it happened during a split into multiple packages, which made the immediate revert impossible. By now, reverting would break the code of people who have already upgraded.

Here is something close to an upgrade guide: #1316 (comment)

The docs should be updated, e.g. https://sdk.vercel.ai/docs/api-reference/stream-data - where did you still see experimental_streamData?

from ai.

Ahh, I may have mixed up with another experimental_stream* functions.

Thanks for fast reply.

Shit happens. Good thing, we don't have to revert since it works on the latest versions if both are the same. But in the future, a greater communication - perhaps a pinned issue or README block would be super nice. Ideally on the release part on Github. :)

from ai.

Solutions

• if you use useChat or useCompletion, it will continue to work. If you see issues, it's most likely caused by a mismatch of the client and server version of the AI SDK, or you need to rebuild/refresh
• If you use your own server-side code, you can
  • pin the version of the AI SDK to 3.0.19
  • use streamMode: "text" in useChat / useCompletion (requires v3.0.23)
• If you use your own client-side code, you can
  • pin the version of the AI SDK to 3.0.19
  • or use readDataStream (see #1316 (comment) )
  • or switch to AI Core on the server side and use streamText with toTextStreamResponse (v3.0.21 and newer)

    export async function POST(req: Request) {
      const { prompt } = await req.json();
    
      const result = await experimental_streamText({
        model: openai.completion('gpt-3.5-turbo-instruct'),
        maxTokens: 2000,
        prompt,
      });
    
      return result.toTextStreamResponse();
    }

    More on streamText: https://sdk.vercel.ai/docs/ai-sdk-core/generating-text#streaming-text

(old)

useChat and useCompletion support the stream data protocol and should continue to work. Can you elaborate on how you are using the result of StreamingTextResponse?

from ai.

Interesting - love the store!

Not sure how well this suites your use case, and it is not fully integrated with the stream data protocol yet, but in the future you can also define tools with the new streamText API: https://sdk.vercel.ai/docs/ai-core/stream-text#terminal-chatbot-with-tools

from ai.

For what it's worth, I actually had the same issue here as well. We parse the output of the stream ourselves and the update to 3.0.20 broke things for us as well which led me here 😄.

If it's any help, this is the code we have:

...
export default async function handler(req: Request, res: NextApiResponse) {
	if (req.method === "POST") {
...
		try {
			const response = await openai.chat.completions.create({
				...
				stream: true,
			});

			const stream = OpenAIStream(response);
			return new StreamingTextResponse(stream);
		} catch (error) {
			console.error("Error calling OpenAI API:", error);
			return NextResponse.json(
				{ error: "Error calling OpenAI API" },
				{ status: 500 },
			);
		}
	} else {
		// Handle any non-POST requests
		return NextResponse.json(
			{ error: "Error calling OpenAI API" },
			{ status: 405 },
		);
	}
}

It sounds like the right move is to switch to readDataStream so I will try that now. Thanks a lot!

from ai.

I'm having a same problem.

Frontend: `'use client';
import { useChat, type Message } from 'ai/react';
import React, { useRef, useEffect } from 'react';
import { Remark, useRemark } from 'react-remark';
import ScrollToBottom from './ScrollToBottom';
const BACKEND = process.env.NEXT_PUBLIC_BACKEND;
export default function ChatComponent() {
const options = {
api: BACKEND + '/api/chat'
};
const { input, handleInputChange, handleSubmit, isLoading, messages, setMessages, setInput } = useChat(options);

const inputRef = useRef<HTMLInputElement>(null);

return (
    <>
        {/* {JSON.stringify(messages)} */}
        {messages.length > 0 && (
            <div className="h-[600px] overflow-y-scroll p-2">
                {messages?.map((message: Message, OutIndex) => {
                    return (
                        <ScrollToBottom key={OutIndex}>
                            {message.role === 'assistant' ? <h3 className="text-lg font-semibold mt-2">GPT-4</h3> : <h3 className="text-lg font-semibold mt-2">User</h3>}
                            {message?.content?.split('\n').map((currentTextBlock: string, index: number) => {
                                if (currentTextBlock === '') {
                                    return;
                                } else {
                                    return <Remark key={index}>{currentTextBlock}</Remark>;
                                }
                            })}
                        </ScrollToBottom>
                    );
                })}
            </div>
        )}

        <form className="mt-4 px-4" onSubmit={handleSubmit}>
            <p>User Message</p>
            <input
                className="w-full bg-slate-100 p-2 outline-none"
                type="url"
                ref={inputRef}
                name="url"
                aria-autocomplete="list"
                required
                disabled={isLoading}
                placeholder="Enter URL to scrape*"
            />
            <textarea
                className="w-full bg-slate-100 p-2  outline-none"
                placeholder={'Enter your additional prompt...'}
                value={input}
                onChange={handleInputChange}
                disabled={isLoading}
            />
            <button disabled={isLoading} className="rounded-md bg-blue-600 text-white p-2 mt-2">
                {isLoading ? 'Loading...' : 'Send'}
            </button>
        </form>
    </>
);

}
`

Backend:
`import OpenAI from 'openai';
import { OpenAIStream, StreamingTextResponse } from 'ai';

// Create an OpenAI API client (that's edge friendly!)
const openai = new OpenAI({
apiKey: process.env.OPENAI_API_KEY
});

// IMPORTANT! Set the runtime to edge
export const runtime = 'edge';

export async function POST(req) {
const { messages } = await req.json();

messages.push({ role: 'system', content: 'tell me joke.' });

console.log(messages);

// Ask OpenAI for a streaming chat completion given the prompt
const response = await openai.chat.completions.create({
    model: 'gpt-4',
    stream: true,
    messages
});

// Convert the response into a friendly text-stream
const stream = OpenAIStream(response);
// Respond with the stream
return new StreamingTextResponse(stream);

}
`

Response:

from ai.

Hi @lgrammel Would be great to have some updated documentation on this. I struggle with the AIStream - as it stopped working when moving to 3.0.21. I use useChat.

The documentation is up-to-date. Can you elaborate what stopped working? Are you using the same version of the AI SDK for client & server and have rebuilt/refreshed?

from ai.

@lgrammel

For me the stream is no longer displayed in the UI. I use useChat, the exact same implementation as an earlier version of AI Chatbot template by vercel.

1. I think the same version of SDK for client and server. Im on a nextjs stack and the package is referenced only once
2. Yes I have rebuilt and refreshed.

I can elaborate:

I have a custom parser that looks like:

function parseMessageStream(): AIStreamParser {
  return (data: string): string => {
    const json = JSON.parse(data)
    let delta: string
    if (json.data && json.data.message && json.data.message.content) {
      delta = json.data.message.content
    } else {
      delta = ''
    }
    return delta
  }
}

When I look at tokens with the callbacks it parses my stream. However useChat is unable to display the tokens or completion client side.

I suspect there is something in the new data protocol I am missing. So instead of head scratching for an extended period of time I was hoping you had some magic for me 🪄

from ai.

@martgra If I understand correctly, you are returning custom text chunks. In the new protocol, text chunks are lines that look like this 0:my-chunk\n. You can use formatStreamPart to achieve this, e.g. formatStreamPart("text", delta) (see https://github.com/vercel/ai/blob/main/packages/core/shared/stream-parts.ts#L363 ). Please be aware that this is an internal API though.

from ai.

IIUC it's not possible to use readDataStream because it's not exported, is this correct?

from ai.

@lgrammel may I ask if useChat sometime will use the SSE protocol directly? Seems like this text stream is less flexible than actually including more metadata in the stream like providers do?

from ai.

@lgrammel may I ask if useChat sometime will use the SSE protocol directly? Seems like this text stream is less flexible than actually including more metadata in the stream like providers do?

Possibly. For the time being we have no such plans though.

from ai.

Hi Lars, I came across this. It still works but not streaming - the response is not shown until it's done. Versions matched on 3.0.22.

First of all, I am frustrated that a breaking change did not cause a major version change. I really believe that this is not a good practice. Also, I cannot seem to find any upgrade guide. This leaves me wondering whether it's my fault or a fault at the library.

Nevertheless, thank you for working on this.

@lgrammel

UPDATE:

I end up discovering that Wrangler broke the streaming so it is not this library, even though the timing did not workout to the benefit.

cloudflare/workers-sdk#5614

I would still be interested in an upgrade guide. It seems to me that either I am confused (could be) or the official docs don't reflect the changes - I still see experimental_streamData in many places.

from ai.

@lgrammel, my API endpoint (with AI SDK) is consumed by my Next.js web app via useChat, as well as by my Chrome extension via SSE. How can I use the same API endpoint for both applications without downgrading the AI SDK?

from ai.

Could you upgrade both parts to the latest version? @GorvGoyl

from ai.

@GorvGoyl if your endpoint send a text stream (not an ai stream), then you could use streamMode: "text" in useChat

from ai.