vercel / ai Goto Github PK

View Code? Open in Web Editor NEW

7.7K 51.0 1.0K 6.34 MB

Build AI-powered applications with React, Svelte, Vue, and Solid

Home Page: https://sdk.vercel.ai/docs

License: Other

TypeScript 98.68% JavaScript 1.13% Vue 0.19%

artificial-intelligence huggingface javascript language-model llm openai react svelte typescript vercel

ai's Introduction

Vercel AI SDK

The Vercel AI SDK is a library for building AI-powered streaming text and chat UIs.

Features

React, Svelte, Vue and Solid helpers for streaming text responses and building chat and completion UIs
React Server Components API for streaming Generative UI
First-class support for OpenAI, Anthropic, Mistral, Perplexity, AWS Bedrock, Azure, Google Gemini, Hugging Face, Fireworks, Cohere, LangChain, Replicate, Ollama, and more.
Node.js, Serverless, and Edge Runtime support
Lifecycle callbacks for saving completed streaming responses to a database (in the same request)

Installation

pnpm install ai

View the full documentation and examples on sdk.vercel.ai/docs.

Authors

This library is created by Vercel and Next.js team members, with contributions from:

Jared Palmer (@jaredpalmer) - Vercel
Shu Ding (@shuding_) - Vercel
Max Leiter (@max_leiter) - Vercel
Malte Ubl (@cramforce) - Vercel
Justin Ridgewell (@jridgewell)

Contributors

Related: Deploy your own Next.js AI Chatbot

If you're looking to for a full AI Chatbot application to jumpstart your AI journey, you should checkout our sister OSS AI Chatbot project or click the button below to deploy your own to Vercel.

ai's People

Stargazers

Watchers

Forkers

lsalling a7med3bdulbaset derekrmiller martinsione jordienr nickheiner theluk ali-hussein-dev pynchmeister danielroe esponges aej11a ahmadrosid stylessh rich-harris juukee jonnyquan mindkhichdi itsnotaka muhtalipdede alfasignde liby aiscot youq-enterprises malezjaa tangye1234 hebilicious rogeriolimadev richardsonjf unbanksytv juandavidgf abaso007 alejandrosuarez telecloud-ca claudiovb tomtobac tonywhite11 derailexander 0x5844 jdorfman ratulhasan christian-armstrong25 erriccc e-roy swiecki spellcraftai cloud2303 bronifty automationkit amorist about-hector pixelp23 moelasec dishengai carlosziegler kumar045 lili21 alexanderinum okpainmo baturalpk christopherom antoinewg juanmartinrivas pacifiquem leonard123456 pradeepdotco guangds wjohnsonup magic0x11 exampleex eltociear 0x1jiangyang danyray420 wfc-2023 sayalchitrakar rrafraf falconilham lautapercuspain zeropointlabs jaredder stevendab zakinator123 geekcheng karenmorgane257 kjwlegend faktanauti john-oula btc168168 ruseelol sonhoangkofax hao1919 arsan-official criptotimsah ysfydz 464657226z lechathei jianfenkezhan ashelioy 464657226x devanvalenzuel1

ai's Issues

Adding chains from Langchain to work with streaming/calling

First of all, great job on this npm package! It's really helping me a lot with my first steps into AI dev.

I've been trying to add a chain from LangChain working, but unfortunately I'm not able to figure out how to do this.
My desired flow is the following: 1) obtain relevant documents from Pinecone 2) Add these with MapReduceDocumentsChain 3) query OpenAI with those documents as embeds.

This is the code I currently have. It uses the Nuxt+Langchain example as boilerplate. The current outcome is two separate calls with two responses. I've also tried to .apply() the chain rather than .call() but the result is the same. I want to be able to chain the LangChainStream to the streamToResponse, essentially.

Any help or pointers would be much appreciated!

import { LangChainStream, streamToResponse, Message } from 'ai'
import { CallbackManager } from 'langchain/callbacks'
import { ChatOpenAI } from 'langchain/chat_models/openai'
import { AIChatMessage, HumanChatMessage } from 'langchain/schema'
import { PineconeClient } from '@pinecone-database/pinecone'
import { PineconeStore } from 'langchain/vectorstores/pinecone'
import { OpenAIEmbeddings } from 'langchain/embeddings/openai'
import { loadQAMapReduceChain } from 'langchain/chains'
export const runtime = 'edge'
export default defineEventHandler(async (event: any) => {
  // Extract the `prompt` from the body of the request
  const { messages } = await readBody(event)

  const { stream, handlers } = LangChainStream()

  const llm = new ChatOpenAI({
    modelName: 'gpt-3.5-turbo-0613',
    temperature: 0.5,
    openAIApiKey: process.env.OPENAI_API_KEY,
    streaming: true,
    callbackManager: CallbackManager.fromHandlers(handlers)
  })

  const lastMessage = messages[messages.length - 1]
  let pineconeResult
  if (lastMessage.role === 'user') {
    const client = new PineconeClient()
    await client.init({
      apiKey: process.env.PINECONE_API_KEY ?? '',
      environment: process.env.PINECONE_ENVIRONMENT ?? ''
    })
    const pineconeIndex = client.Index(process.env.PINECONE_INDEX ?? '')
    const vectorStore = await PineconeStore.fromExistingIndex(new OpenAIEmbeddings(), { pineconeIndex })

    // Search the vector DB
    pineconeResult = await vectorStore.similaritySearch(lastMessage.content, 1)

    // the way to use loadQAMapReduceChain. The result is a call.
    const chain = loadQAMapReduceChain(llm)
    await chain.call({
      input_documents: pineconeResult,
      question: lastMessage.content
    })
  }

  // call as shown in the 'ai' package documentation/examples
  llm
    .call(
      (messages as Message[]).map(message =>
        message.role === 'user'
          ? new HumanChatMessage(message.content)
          : new AIChatMessage(message.content)
      )
    )
    // eslint-disable-next-line no-console
    .catch(console.error)

  return streamToResponse(stream, event.node.res)
})

Support for OpenAI Function Calling and streaming?

Is there any examples of using this with the new Function Calling feature in OpenAI's Chat Completion API?

Or is there more work needed to support this?

Thanks!

Broken links to /api-reference

Problem:
There are some broken links in the docs, here are two I found.

Page: https://sdk.vercel.ai/docs/guides/openai
Link 1: OpenAIStream - https://sdk.vercel.ai/api-reference#openaistream
Link 2: StreamingTextResponse - https://sdk.vercel.ai/api-reference#streamingtextresponse

Hope this is helpful.

Stream never closes with LangChainStream using postman

Have a few issues with LangChainStream:

Appears stream never closes when API is called
Tokens are not getting set or sent down stream, but are coming through when adding callback functions to LangChainStream

Example code:

import { ChatAnthropic } from "langchain/chat_models/anthropic";
import { ChatOpenAI } from 'langchain/chat_models/openai';

import { BaseChatMessage } from "langchain/dist/schema";
import {
  ChatPromptTemplate,
  HumanMessagePromptTemplate,
  PromptTemplate,
  SystemMessagePromptTemplate,
} from "langchain/prompts";
import { StreamingTextResponse, LangChainStream } from "ai";
import { CallbackManager } from "langchain/callbacks";

export const runtime = "edge";

if (!process.env.ANTHROPIC_API_KEY) {
  throw new Error("Missing env var from Anthropic");
}

const createSummaryPrompt = async (
  persona: string,
  preprocessed_data: string
): Promise<BaseChatMessage[]> => {
  const chatPrompt = PromptTemplate.fromTemplate(
      `This is your title: {title}. blah blah blah. \User Data: {userData}.`
);

  const formattedPrompt = await chatPrompt.formatPromptValue({
    persona,
    transcriptData: JSON.stringify(preprocessed_data),
  });

  return formattedPrompt.toChatMessages();
};

export const POST = async (req: Request) => {
  const { title, user_data } =
    (await req.json()) as { title?: string; user_data: string };

  if (!preprocessed_data) {
    return new Response("No transcript in the request", { status: 400 });
  }

  const { stream, handlers } = LangChainStream({
    onStart: async () => console.log("start"),
    onToken: async (token) => console.log(token),
    onCompletion: async () => console.log("end"),
  });

  const llm = new ChatAnthropic({
    apiKey: process.env.ANTHROPIC_API_KEY,
    modelName: "claude-v1.3-100k",
    temperature: 0.1,
    maxTokensToSample: 100000,
    streaming: true,
    verbose: true,
    callbacks: [handlers],
  });

  const prompt = await createSummaryPrompt(persona, preprocessed_data);

  llm
    .call(prompt)
    .then((res) => console.log(stream))
    .catch(console.error).finally(() => console.log("done"));
    
  console.log(stream)
  return new StreamingTextResponse(stream);
};

Example cURL:

curl --location 'http://localhost:3000/api/generate-summary' \
--header 'Content-Type: text/plain' \
--header 'Cookie: <secret>' \
--data '{"user_data":"blah blah blah","title":"sales chief"}'

Postman screenshot - Connection stays open and never ends. No data returns or streams.

Console output:

start
<ref *1> ReadableStream {
  _state: 'readable',
  _reader: undefined,
  _storedError: undefined,
  _disturbed: false,
  _readableStreamController: ReadableStreamDefaultController {
  _controlledReadableStream: [Circular *1],
  _queue: S {
  _cursor: 0,
  _size: 0,
  _front: { _elements: [], _next: undefined },
  _back: { _elements: [], _next: undefined }
},
  _queueTotalSize: 0,
  _started: true,
  _closeRequested: false,
  _pullAgain: false,
  _pulling: false,
  _strategySizeAlgorithm: [Function],
  _strategyHWM: 0,
  _pullAlgorithm: [Function],
  _cancelAlgorithm: [Function]
}
}
[llm/start] [1:llm:ChatAnthropic] Entering LLM run with input: {
  "messages": [
    [
      {
        "type": "human",
        "data": {
          "content": "redacted",
          "additional_kwargs": {}
        }
      }
    ]
  ]
}
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
token!
[llm/end] [1:llm:ChatAnthropic] [7.56s] Exiting LLM run with output: {
  "generations": [
    [
      {
        "text": "redacted",
        "message": {
          "type": "ai",
          "data": {
            "content": " redacted",
            "additional_kwargs": {}
          }
        }
      }
    ]
  ]
}
<ref *1> ReadableStream {
  _state: 'readable',
  _reader: ReadableStreamDefaultReader {
  _ownerReadableStream: [Circular *1],
  _closedPromise_resolve: [Function],
  _closedPromise_reject: [Function],
  _closedPromise: Promise {
  [Symbol(async_id_symbol)]: 1974,
  [Symbol(trigger_async_id_symbol)]: 0,
  [Symbol(kResourceStore)]: undefined,
  [Symbol(kResourceStore)]: undefined,
  [Symbol(kResourceStore)]: undefined
},
  _readRequests: S {
  _cursor: 77,
  _size: 1,
  _front: {
  _elements: [
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  {
  _chunkSteps: [Function: _chunkSteps],
  _closeSteps: [Function: _closeSteps],
  _errorSteps: [Function: _errorSteps]
}
],
  _next: undefined
},
  _back: {
  _elements: [
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  {
  _chunkSteps: [Function: _chunkSteps],
  _closeSteps: [Function: _closeSteps],
  _errorSteps: [Function: _errorSteps]
}
],
  _next: undefined
}
}
},
  _storedError: undefined,
  _disturbed: true,
  _readableStreamController: ReadableStreamDefaultController {
  _controlledReadableStream: [Circular *1],
  _queue: S {
  _cursor: 0,
  _size: 0,
  _front: { _elements: [], _next: undefined },
  _back: { _elements: [], _next: undefined }
},
  _queueTotalSize: 0,
  _started: true,
  _closeRequested: false,
  _pullAgain: false,
  _pulling: true,
  _strategySizeAlgorithm: [Function],
  _strategyHWM: 0,
  _pullAlgorithm: [Function],
  _cancelAlgorithm: [Function]
}
}
done
token!
token!
token!
token!
token!
token!
done!!!
token!
token!
token!
token!
token!
token!
token!
[llm/end] [1:llm:ChatAnthropic] [8.54s] Exiting LLM run with output: {
  "generations": [
    [
      {
        "text": " redacted",
        "message": {
          "type": "ai",
          "data": {
            "content": " redactedl",
            "additional_kwargs": {}
          }
        }
      }
    ]
  ]
}
<ref *1> ReadableStream {
  _state: 'readable',
  _reader: ReadableStreamDefaultReader {
  _ownerReadableStream: [Circular *1],
  _closedPromise_resolve: [Function],
  _closedPromise_reject: [Function],
  _closedPromise: Promise {
  [Symbol(async_id_symbol)]: 2452,
  [Symbol(trigger_async_id_symbol)]: 2239,
  [Symbol(kResourceStore)]: undefined,
  [Symbol(kResourceStore)]: undefined,
  [Symbol(kResourceStore)]: undefined,
  [Symbol(kResourceStore)]: undefined,
  [Symbol(kResourceStore)]: undefined,
  [Symbol(kResourceStore)]: undefined
},
  _readRequests: S {
  _cursor: 100,
  _size: 1,
  _front: {
  _elements: [
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  {
  _chunkSteps: [Function: _chunkSteps],
  _closeSteps: [Function: _closeSteps],
  _errorSteps: [Function: _errorSteps]
}
],
  _next: undefined
},
  _back: {
  _elements: [
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  undefined,
  {
  _chunkSteps: [Function: _chunkSteps],
  _closeSteps: [Function: _closeSteps],
  _errorSteps: [Function: _errorSteps]
}
],
  _next: undefined
}
}
},
  _storedError: undefined,
  _disturbed: true,
  _readableStreamController: ReadableStreamDefaultController {
  _controlledReadableStream: [Circular *1],
  _queue: S {
  _cursor: 0,
  _size: 0,
  _front: { _elements: [], _next: undefined },
  _back: { _elements: [], _next: undefined }
},
  _queueTotalSize: 0,
  _started: true,
  _closeRequested: false,
  _pullAgain: false,
  _pulling: true,
  _strategySizeAlgorithm: [Function],
  _strategyHWM: 0,
  _pullAlgorithm: [Function],
  _cancelAlgorithm: [Function]
}
}
done

Something isn't connecting here I guess if we're getting an array of undefined. Any ideas where I'm going wrong here? Any help much appreciated!

( additional side note: Would be so cool to have a useStream() for features that are not chat and are not completions! e.g. one off API calls formed from app data like mine. I am passing things to useChat({body: {title:, user_data}}) )

Next JS starter Template gives me an 500 status code internal Error

After doing all the steps that are mentioned here and generating my API key gives me an error

I didn't change anything inside the code

Next Hugging Face not creating project

Running on macos m1

npx create-next-app --example https://github.com/vercel-labs/ai/tree/main/examples/next-huggingface next-huggingface-app

Returns error:

npx: installed 1 in 1.664s

Aborting installation.
Unexpected error. Please report it as a bug:
 { Error: EEXIST: file already exists, mkdir '/Users/me/Library/Preferences/create-next-app-nodejs'
    at Object.fs.mkdirSync (fs.js:840:3)
    at Conf._ensureDirectory (/Users/me/.npm/_npx/33439/lib/node_modules/create-next-app/dist/index.js:38:34811)
    at Conf.get store [as store] (/Users/me/.npm/_npx/33439/lib/node_modules/create-next-app/dist/index.js:38:33449)
    at Conf._get (/Users/me/.npm/_npx/33439/lib/node_modules/create-next-app/dist/index.js:38:36766)
    at Conf.get (/Users/me/.npm/_npx/33439/lib/node_modules/create-next-app/dist/index.js:38:31582)
    at run (/Users/me/.npm/_npx/33439/lib/node_modules/create-next-app/dist/index.js:325:26974)
    at Module.68994 (/Users/me/.npm/_npx/33439/lib/node_modules/create-next-app/dist/index.js:325:30991)
    at __nccwpck_require__ (/Users/me/.npm/_npx/33439/lib/node_modules/create-next-app/dist/index.js:325:38793)
    at /Users/me/.npm/_npx/33439/lib/node_modules/create-next-app/dist/index.js:325:39657
    at Object.<anonymous> (/Users/me/.npm/_npx/33439/lib/node_modules/create-next-app/dist/index.js:325:39702)
  errno: -17,
  syscall: 'mkdir',
  code: 'EEXIST',
  path: '/Users/me/Library/Preferences/create-next-app-nodejs' }

Copying from the docs copies borken text

For example, if I tried to copy buildOpenAssistantPrompt function I get this

function
 
buildOpenAssistantPrompt
(
  messages
:
 { content
:
 
string
; role
:
 
'system'
 
|
 
'user'
 
|
 
'assistant'
 }[]
) {
  
return
 (
    messages
      
.map
(({ content
,
 role }) 
=>
 {
        
if
 (role 
===
 
'user'
) {
          
return
 
`<|prompter|>
${
content
}
<|endoftext|>`
;
        } 
else
 {
          
return
 
`<|assistant|>
${
content
}
<|endoftext|>`
;
        }
      })
      
.join
(
''
) 
+
 
'<|assistant|>'
  );
}

Add support for Anthropic

AzureOpenAI support

Thanks so much for the great package.

Using Azure endpoints instead of openAI API requires being able to specify the api endpoint url and give your api key slightly differently, specifically a header of api-key: YOURKEY vs openai's Authorization: Bearer X header.

My question is how would you prefer Azure be implemented? I see that now already it is possible to pass in headers. Is that the preferred way moving forward? If not, please advise how you would want the integration to look and I can submit a pull request. Thank you!

Getting function invocation timeout in serverless function streaming

I'm trying to stream responses from serverless functions with the below code. It's working fine in the local env (nodejs server) but I'm getting Function Invocation timeouts when I tested in vercel deployment. Is there something wrong with the code?

Backend code:

import type { VercelRequest, VercelResponse } from '@vercel/node';
import { IFilters } from '@/types/requestTypes';
import { StreamingTextResponse, LangChainStream, Message, streamToResponse } from 'ai';
import { CallbackManager } from 'langchain/callbacks';
import { ChatOpenAI } from 'langchain/chat_models/openai';
import {
  AIChatMessage,
  BaseChatMessage,
  HumanChatMessage,
} from 'langchain/schema';


export default async function handler(
  req: VercelRequest,
  res: VercelResponse,
) {
  console.log('API Called time ', Date.now());
  const { messages,filters } = JSON.parse(req.body);
  console.log('messages', messages);
  console.log('Rqeust Json ', req.body);
  console.log('Filters in request ', filters);

  
  console.log('messages', messages);
  console.log('Filters in request ', filters);
  const { stream, handlers } = LangChainStream({
    onStart: async () => {
      console.log('Stream Start time ', Date.now());
    },
    onToken: async (token: string) => {
      console.log(token);
    },
    onCompletion: async (completion: string) => {
      console.log('Stream End time ', Date.now());
    },
  });

  const llm = new ChatOpenAI({
    streaming: true,
    callbackManager: CallbackManager.fromHandlers(handlers),
  });
  llm.call([new HumanChatMessage(messages[0].content)]).catch(console.error);

  streamToResponse(stream, res);
}

Frontend code:

const {
    messages,
    input: aiSearchInput,
    handleInputChange,
    handleSubmit,
  } = useChat({
    api: '/api/streamTest',
    body: {
      // question: aiSearchInput,
      filters: {
        tickers: selectedStocksList,
        sources: [EARNING_TRANSCRIPT_REPORT_TYPE],
        timeframe: selectedQuarters,
      },
    },
    onResponse: () => {
      // steaming started
      steamStarted = Date.now();
      console.log('API latency:', steamStarted - apiHitting, 'milliseconds');
    },
    onFinish: () => {
      // steaming enden
      steamEnded = Date.now();
      console.log('Stream latency:', steamEnded - steamStarted, 'milliseconds');
      console.log('Total latency:', steamEnded - apiHitting, 'milliseconds');
      console.log('Stream End:', steamEnded);
      console.log('Stream Started');
    },
  });

Nuxt: Clicking the "Deploy" vercel button on examples doc does not correctly stream the data

Testing out the Nuxt version and clicking the "Deploy" button here creates and deploys the Nuxt version of the AI chat bot to Vercel server, but does not stream the data response. Only the full response is shown at once. I wanted to quickly test the chatbot demo but I guess I need to manually set up and configure everything, instead? Thanks for any clarification.

This may be due to the fact this deployment template does not include vercel edge preset. If I set nuxt config to vercel edge and deploy, it causes 502 gateway error see this issue

README file for npm

Visiting the npm package page there is no README file.

Would be nice to see here :)

next-openai example out of the box fails to work

Steps to reproduce

npx create-next-app --example https://github.com/vercel-labs/ai/tree/main/examples/next-openai next-openai-app
npm i
create .env file with OPENAI_API_KEY with a valid open api key
type in input box - error returned from api request

Allow submitting form without adding a message in useChat hook

Currently we do nothing if no text is input in the form - https://github.com/vercel-labs/ai/blob/fd69e9a5628d1cd21a7c4a36584b4c577e0cc0a6/packages/core/react/use-chat.ts#L253

But, submitting a new API request without adding a new user message is a good way to get the LLM to continue its previous response, for example if the user hit stop, or the previous response reached the token limit. This is how the OpenAI playground works.

Unable to fetch server using useChat hook

I tried to set up the useChat hook in my Next.js application to fetch data from the server, but I'm encountering difficulties in fetching the server. I have verified the correctness of the endpoint URL and the presence of the route.ts file. I suspect there might be an issue with the implementation of either the hook or the API route. Any assistance in troubleshooting this problem would be greatly appreciated.

import { useChat } from 'ai/react';

client side

export default function  () {...
 const { messages, input, handleInputChange, handleSubmit } = useChat({api:'pages/api/chat/route.ts'});

 <div className="max-w-sm mx-auto" id="input-container">
                {messages.map(m => (<div key={m.id}> {m.role === 'user' ? 'User: ' : 'AI: '}
                  {m.content}</div>))}

                <form onSubmit={handleSubmit}>
                  
                  <input type="text" value={input} onChange={handleInputChange}
                    className="w-full px-4 py-2 border border-gray-300 rounded-md focus:outline-none focus:ring-2 focus:ring-indigo-500" placeholder="Ask a question..." />

                  <button type="submit"
                    className="mt-4 w-full px-4 py-2 bg-indigo-500 text-white font-bold rounded-md hover:bg-indigo-600 focus:outline-none focus:bg-indigo-600">Ask</button>
                </form>
              </div>

Server side...
"
import { Configuration, OpenAIApi } from 'openai-edge';
import { OpenAIStream, StreamingTextResponse } from 'ai';

// Create an OpenAI API client (that's edge friendly!)
const config = new Configuration({
apiKey: process.env.OPENAI_API_KEY
})
const openai = new OpenAIApi(config)

// IMPORTANT! Set the runtime to edge
export const runtime = 'edge'

export async function POST(req: Request) {
  // Extract the `messages` from the body of the request
  const { messages } = await req.json()
 
  // Ask OpenAI for a streaming chat completion given the prompt
  const response = await openai.createChatCompletion({
    model: 'gpt-3.5-turbo',
    stream: true,
    messages
  })
  // Convert the response into a friendly text-stream
  const stream = OpenAIStream(response)
  // Respond with the stream
  return new StreamingTextResponse(stream)
}

Allow changing `body` with `useCompletion`

I've got an endpoint that takes custom additional data - it would make using useCompletion a lot easier in many situations if it exposed setBody or something similar.

What about Serverless Functions?

The docs talk about using these tools in the context of an edge function in a NextJS app. What if I want a standalone Serverless Function?

Strange question marks �� in cyrillic symbols

Hi! Thanks for great lib. Screenshot from https://chat.vercel.ai:

It seems like an encoding issue, and it seems to appear only on Vercel deployment.

Could not resolve "crypto" deployed in Vercel with SvelteKit Edge Runtime

Hi, first of all I want to express my gratitude for this SDK library. I migrated all my code from implementing streaming from scratch to this. Now, when I tried to deploy to Vercel with SvelteKit Edge Runtime I'm getting this issue in the deployment:

Run npm run preview to preview your production build locally.
--
11:37:54.638 |  
11:37:54.638 | > Using @sveltejs/adapter-vercel
11:37:54.708 | ✘ [ERROR] Could not resolve "crypto"
11:37:54.709 |  
11:37:54.709 | .svelte-kit/output/server/chunks/index5.js:3:19:
11:37:54.709 | 3 │ import crypto from "crypto";
11:37:54.709 | ╵                    ~~~~~~~~
11:37:54.709 |  
11:37:54.709 | The package "crypto" wasn't found on the file system but is built into node. Are you trying to bundle for node? You can use "platform: 'node'" to do that, which will remove this error.

It successfully builds locally. I search my code and there's no import crypto from "crypto" there and looking at the local build files it seems to be pointing to this AI SDK i.e. import { u as useChat } from "../../../chunks/index5.js"; but still I can't find import crypto from "crypto" in the built file.

Thanks in advance

No matching version found for [email protected].

All the projects located in https://github.com/vercel-labs/ai/tree/main/examples have the version 2.0.1 in package.json.

This version can't by found by the registry.

However 2.0.0 works perfectly fine.

next-langchain git:(main) ✗ npm i
npm ERR! code ETARGET
npm ERR! notarget No matching version found for [email protected].
npm ERR! notarget In most cases you or one of your dependencies are requesting
npm ERR! notarget a package version that doesn't exist.

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/fertostado/.npm/_logs/2023-06-15T21_57_28_736Z-debug-0.log
➜  next-langchain git:(main) ✗ yarn install
yarn install v1.22.19
info No lockfile found.
[1/4] 🔍  Resolving packages...
Couldn't find any versions for "ai" that matches "2.0.1"
? Please choose a version of "ai" from this list: 2.0.0
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
warning "ai > [email protected]" has unmet peer dependency "svelte@^3.29.0".
[4/4] 🔨  Building fresh packages...

success Saved lockfile.
✨  Done in 15.89s.

Fix proposed in #81

error Failed to convert the response to stream. Received status code: 400.

error node_modules/ai/dist/index.mjs (110:10) @ AIStream
error Failed to convert the response to stream. Received status code: 400.

I'm following the tutorial, but I got this error.

// ./app/api/chat/route.js
import { Configuration, OpenAIApi } from 'openai-edge'
import { OpenAIStream, StreamingTextResponse } from 'ai'

const config = new Configuration({
  apiKey: process.env.OPENAI_API_KEY
})
const openai = new OpenAIApi(config)

export const runtime = 'edge'

export async function POST (req) {
  const { messages } = await req.json()
  console.log(messages)
  const response = await openai.createChatCompletion({
    model: 'gpt-3.5-turbo',
    stream: true,
    messages
  })
  const stream = OpenAIStream(response)
  return new StreamingTextResponse(stream)
}

Link to "Hugging Face" is misspelled and leads to wrong domain

The homepage at https://sdk.vercel.ai/docs has a link to https:// + hugginface.co/ which is missing a g in there. The domain it does link to seems to be grabbed by a party that's doing all sorts of non-AI-stuff, I'm unsure how legit it even is what they have on their site.

I presume the correct link should be https://huggingface.co/, right?

See these search results in this repo for the files to be changed, at the moment:

https://github.com/vercel-labs/ai/blob/4315d174a897adfae4461bf5315ff68bd54e2f93/docs/pages/docs/guides.mdx?plain=1#L7

and

https://github.com/vercel-labs/ai/blob/4315d174a897adfae4461bf5315ff68bd54e2f93/docs/pages/docs/index.mdx?plain=1#L120

OpenAI Completions API - 404 Error

I was following the guide in the docs, but can't make it work. The response to Completion API always has status 404.

/routes/api/completion/+server.ts

import { OPENAI_API_KEY } from '$env/static/private';

import { Configuration, OpenAIApi } from 'openai-edge';
import { OpenAIStream, StreamingTextResponse } from 'ai';

import type { RequestHandler } from './$types';

const config = new Configuration({
	apiKey: OPENAI_API_KEY
});
const openai = new OpenAIApi(config);

export const POST = (async ({ request }) => {
	const { prompt } = await request.json();

	const response = await openai.createCompletion({
		model: 'gpt-3.5-turbo',
		stream: true,
		prompt: prompt
	});

	const stream = OpenAIStream(response);

	return new StreamingTextResponse(stream);
}) satisfies RequestHandler;

Error:

Response {
  [Symbol(realm)]: null,
  [Symbol(state)]: {
    aborted: false,
    rangeRequested: false,
    timingAllowPassed: true,
    requestIncludesCredentials: true,
    type: 'default',
    status: 404,
    timingInfo: {
      startTime: 808392.5959000029,
      redirectStartTime: 0,
      redirectEndTime: 0,
      postRedirectStartTime: 808392.5959000029,
      finalServiceWorkerStartTime: 0,
      finalNetworkResponseStartTime: 0,
      finalNetworkRequestStartTime: 0,
      endTime: 0,
      encodedBodySize: 153,
      decodedBodySize: 0,
      finalConnectionTimingInfo: null
    },
    cacheState: '',
    statusText: 'Not Found',
    headersList: HeadersList {
      cookies: null,
      [Symbol(headers map)]: [Map],
      [Symbol(headers map sorted)]: null
    },
    urlList: [ [URL] ],
    body: { stream: undefined }
  },
  [Symbol(headers)]: HeadersList {
    cookies: null,
    [Symbol(headers map)]: Map(21) {
      'date' => [Object],
      'content-type' => [Object],
      'transfer-encoding' => [Object],
      'connection' => [Object],
      'access-control-allow-origin' => [Object],
      'openai-organization' => [Object],
      'openai-processing-ms' => [Object],
      'openai-version' => [Object],
      'strict-transport-security' => [Object],
      'x-ratelimit-limit-requests' => [Object],
      'x-ratelimit-limit-tokens' => [Object],
      'x-ratelimit-remaining-requests' => [Object],
      'x-ratelimit-remaining-tokens' => [Object],
      'x-ratelimit-reset-requests' => [Object],
      'x-ratelimit-reset-tokens' => [Object],
      'x-request-id' => [Object],
      'cf-cache-status' => [Object],
      'server' => [Object],
      'cf-ray' => [Object],
      'content-encoding' => [Object],
      'alt-svc' => [Object]
    },
    [Symbol(headers map sorted)]: null
  }
}
Error: Failed to convert the response to stream. Received status code: 404.
    at Module.AIStream (D:/Github/bard/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/ai/dist/chunk-265FSSO4.mjs:74:11)
    at Module.OpenAIStream (D:/Github/bard/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/ai/dist/chunk-TJMME6CL.mjs:19:32)
    at POST (D:/Github/bard/src/routes/api/completion/+server.ts:21:40)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Module.render_endpoint (D:/Github/bard/node_modules/.pnpm/@[email protected][email protected][email protected]/node_modules/@sveltejs/kit/src/runtime/server/endpoint.js:47:18)
    at async resolve (D:/Github/bard/node_modules/.pnpm/@[email protected][email protected][email protected]/node_modules/@sveltejs/kit/src/runtime/server/respond.js:403:17)
    at async Module.respond (D:/Github/bard/node_modules/.pnpm/@[email protected][email protected][email protected]/node_modules/@sveltejs/kit/src/runtime/server/respond.js:274:20)
    at async file:///D:/Github/bard/node_modules/.pnpm/@[email protected][email protected][email protected]/node_modules/@sveltejs/kit/src/exports/vite/dev/index.js:505:22

Vue integration

I'd be very happy to pitch in to add a Vue integration (and perhaps a Nuxt one too) if you are interested 😊

We could use a utility like https://github.com/Kong/swrv to ensure implementation details are largely similar across libraries.

Request Body

without using ai-sdk, a response call to an api path is like this for my project:

    const response = await fetch("/api/gpt3-output", {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
      },
      body: JSON.stringify({
        promptText: prompt,
        maxTokens: Number(gpt3Length),
        temperature: Number(gpt3Temperature),
        topP: Number(gpt3TopP),
        presencePenalty: Number(gpt3PresencePenalty),
        frequencyPenalty: Number(gpt3FrequencyPenalty),
      }),
    })

and then the api file is like this:

  const { promptText, maxTokens, temperature, topP, presencePenalty, frequencyPenalty} = (await req.json())
  const payload: OpenAIStreamPayload = {
    model: "gpt-4",
    messages: [
      { role: "system", content: "You are a helpful assistant." },
      { role: "user", content: promptText },
    ],
    max_tokens: maxTokens as number,
    temperature: temperature as number,
    top_p: topP as number,
    presence_penalty: presencePenalty as number,
    frequency_penalty: frequencyPenalty as number,
    stream: true,
  }
  const stream = await OpenAIStream(payload)
  return new Response(stream)

how can i pass the request body parameters with the ai-sdk?

Local GGML model support

I would like to request local GGML model support for this package. It would be great if users can deploy models locally using the GGML-v3 model. Thanks so much for the great package.

Feedback for “useCompletion”

Should it be import useCompletion in the svelte code snippet rather that useChat?

No matching version for "ai" that matches "2.1.1"

Looking forward to using this starter and the integrations. Appreciate the work here. Seeing an issue after trying to deploy the next.js openai starter:

Vercel CLI 30.2.1
--
09:27:41.741 | Installing dependencies...
09:27:41.992 | yarn install v1.22.17
09:27:42.011 | info No lockfile found.
09:27:42.015 | [1/4] Resolving packages...
09:27:44.822 | error Couldn't find any versions for "ai" that matches "2.1.1"
09:27:44.822 | info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
09:27:44.842 | Error: Command "yarn install" exited with 1
09:27:45.232 | BUILD_UTILS_SPAWN_1: Command "yarn install" exited with 1

Looks like the latest release is 2.1.0.

https://github.com/vercel-labs/ai/releases/tag/ai%402.1.0

Also this seems like it's a similar problem as #82

Question: when will OpenAI's gpt-4.0 be supported?

Will there be OpenAI support for gpt-4.0 and if so what is timeline?

How to track token usage

The non-streaming chat/completions API from OpenAI has a usage object in the response:

{
  "id": "chatcmpl-123",
  "object": "chat.completion",
  "created": 1677652288,
  "choices": [{
    "index": 0,
    "message": {
      "role": "assistant",
      "content": "\n\nHello there, how may I assist you today?",
    },
    "finish_reason": "stop"
  }],
  "usage": {
    "prompt_tokens": 9,
    "completion_tokens": 12,
    "total_tokens": 21
  }
}

Which is great when you have different users and want to put limits depending on usage.

Any recommended way to do the same with the Vercel AI SDK?

`docs` segment is missing when navigate using a link

This Link:

We can use the [`useCompletion`](/api-reference/use-completion) hook to make it easy to wire up the UI.

leads to:
https://sdk.vercel.ai/api-reference/use-completion
which is not found

instead of:
https://sdk.vercel.ai/docs/api-reference/use-completion

Have you verified whether the abort signal is actually functioning in the edge runtime?

Based on my experiments so far, it appears that the AbortController doesn't function properly on Vercel's hosted edge runtime. This observation aligns with the issue discussed in detail on the following GitHub issue: vercel/next.js#50364.

Add wider support for `fetch` parameters including `credentials`

Currently only additional headers are allowed to be passed to fetch alongside the URL. Unfortunately it's not possible to utilize cookie based authentication with credentials option so I would love to see ability to pass other options for fetch.

Prompt engineering example Step 3 not working as intended

In the prompt engineering docs in the third step of the example readers get encouraged to "re-submit the identical prompt a few times with temperature set to 1." with the intention to produce more varied outcomes due to higher temperatures.
There seems to be an issue (with caching?) here. I'm getting the same response, no matter how often i submit the prompt.

sveltekit getting-started docs not working

I am using the directions from here: https://sdk.vercel.ai/docs/getting-started#prerequisites

Following everything as explained, setting my OPENAI_API_KEY in .env, creating a /routes/api/completion.js file and pasting the associated code, and keep getting the following error when clicking the "generate slogan" button in my /src/routes/index.svelte (or /src/routes/+page.svelte)

Error: Not found: /api/completion

Am I missing something here?

Error: Failed to convert the response to stream. Received status code: 429. Sveltekit

//src/routes/api/completion/+server.ts
import { OPENAI_API_KEY } from "$env/static/private";

import { Configuration, OpenAIApi } from "openai-edge";
import { OpenAIStream, StreamingTextResponse } from "ai";

import type { RequestHandler } from "./$types";

const config = new Configuration({
  apiKey: OPENAI_API_KEY,
});
const openai = new OpenAIApi(config);

export const POST = (async ({ request }) => {
  const { prompt } = await request.json();

  const response = await openai.createCompletion({
    model: "text-davinci-003",
    max_tokens: 16,
    stream: true,
    prompt: prompt,
  });

  const stream = OpenAIStream(response);

  return new StreamingTextResponse(stream);
}) satisfies RequestHandler;

//src/routes/+page.svelte
<script>
  import { useCompletion } from "ai/svelte";

  const { input, handleSubmit, completion } = useCompletion({
    api: "/api/completion",
  });
</script>

<main>
  <form on:submit={handleSubmit}>
    <input
      type="text"
      bind:value={$input}
      placeholder="Describe your business..."
    />
    <button type="submit">Generate Slogan</button>
  </form>
  <p>{$completion}</p>
</main>

Unnecessary div tag in documentation

This div tag needs to be removed:

Example using Streaming Response for FastAPI.

Lots of people write their Langchain apis in Python, not using RSC.

A common tech stack is using FastAPI on the backend with NextJS/React for the frontend. It would be great to show an example of this using FastAPI Streaming Response.

This would really help us building Quivr..

Dark mode support

Add dark mode pls your docs are burning my eyes.

Streaming is not working on M1 mac browsers & Postman

We tested the SDK in vercel edge on various devices using the exact code provided in the documentation. Here are the findings

Apple M1 Mac browsers - Response is not streaming.
Android chrome - Working
Windows browsers - Working
Mac with Intel chip - working
iOS browsers - working
Postman - Not working

Is there a bug in SDK or is there any special config we should use to enable streaming on the above devices?

Make streaming work with langchain agents

Is it possible to make streaming work with agents ? I am looking for a solution to stream intermediate steps and answer so users can see the tool selection process from UI along with streaming answer at end.

This is the code i am using on backend

//...imports
export const runtime = "edge";

export const POST = async (req: Request) => {
  const { prompt, debugOptions } = generateInputSchema.parse(await req.json());
  const patchedOptions = { ...config.defaultDebugOptions, ...debugOptions };
  const { stream, handlers } = LangChainStream();

  const model = new ChatOpenAI({
    temperature: 0,
    openAIApiKey: env.OPENAI_API_KEY,
    modelName: "gpt-4-0613",
    callbackManager: CallbackManager.fromHandlers(handlers),
    streaming: true,
  });

  const tools = getTools(patchedOptions);

  const executor = await initializeAgentExecutorWithOptions(tools, model, {
    agentType: "openai-functions",
    maxIterations: 10,
    returnIntermediateSteps: true,
  });

  await executor.call({ input: prompt }).finally(() => {
    handlers.handleChainEnd();
  });
  
  return new StreamingTextResponse(stream);
};

The stream for this is only giving full final answer at the end instead of letter by letter for me with useCompletion hook.

LangChainStream shouldn't close on LLMEnd

The LangChainStream uses handleLLMEnd instead of handleChainEnd to close the stream, for sequence streams this would cause an issue.

Below is a sample that gives the following error:
Error in handler Handler, handleLLMNewToken: TypeError: The stream is closing or closed and cannot be written to

const { stream, handlers } = LangChainStream()

  const llm = new ChatOpenAI({
    modelName: "gpt-3.5-turbo", temperature: 0.0,  n: 1,
    streaming: true,
    callbackManager: CallbackManager.fromHandlers(handlers)
  })
  const inquiryChain = new LLMChain({
    llm, prompt: new PromptTemplate({
      template: templates.inquiryTemplate,
      inputVariables: ["prompt", "conversationHistory"],
    }),
    outputKey: "inquiry",

  });
  
  const reviewTemplate = `You are a poem generator, your job is to write a poem about 
  {inquiry}:`;
const reviewPromptTemplate = new PromptTemplate({
  template: reviewTemplate,
  inputVariables: ["inquiry"],
});
const reviewChain = new LLMChain({
  llm: llm,
  prompt: reviewPromptTemplate,
});
  const overallChain = new SequentialChain({
    chains: [inquiryChain, reviewChain],
    inputVariables:["prompt", "conversationHistory"],
    verbose: true,
  });

CallbackManager in langchain is deprecated

In the langchain example it's using callBackManager to get the handlers, this is now deprecated and should be replaced with handlers

i.e. the following line:

callbackManager: CallbackManager.fromHandlers(handlers)

should be replaced with:

 callbacks: [handlers],

Nuxt Error on Deployment using 'vercel-edge' preset: "An internal error occurred with Vercel. BAD_GATEWAY". Works locally.

When setting the nitro preset to vercel-edge and deploying the example code from here to Vercel, I get following server response:

An internal error occurred with Vercel.

BAD_GATEWAY

It WORKS locally, but not when deployed to Vercel

Also, when not using the vercel-edge preset, the code works when deployed EXCEPT it will not stream the data (only show the server data response when it is completed, instead of incremently).

Anyone got any ideas how to get stream work using vercel-edge?

My nuxt.config.ts file:

export default defineNuxtConfig({
  nitro: {
    preset: 'vercel-edge'
  },
  devtools: { enabled: true },
  modules: ['@nuxtjs/tailwindcss'],
  runtimeConfig: {
    openaiApiKey: ''
  }
})

Feedback for “Getting Started”

To enhance user experience, it would be beneficial to allow for user preference-based Tabs for the "Next/Sveltekit" framework. For instance, if a user chooses Sveltekit, the default tab selected in subsequent code snippets should be Sveltekit.

Helpers (or best practices) for non-streaming API response?

Thanks for creating this great tool! I started to create something similar last night, and was glad to see this today.

I'm using the new Function Calling API to generate improvement recommendations that are then highlighted in a block of text. (Similar to Grammarly.) I'm not doing a full agent-style loop or anything, just using the function to ensure the results are in the right schema.

With JSON's required closing tags, I can't parse or render a partial (streamed) response. (I'm not even sure if FC can be streamed.) Is there a best practice for how to return these non-streaming results back to the client in a way that is compatible with useCompletion and useChat? I'm currently using the following and it seems to work well enough.

const response = await openai.createChatCompletion({
    model: 'gpt-3.5-turbo-0613',
    messages: [...],
    functions: [
      { name: 'makeRecommendation', parameters: recommendationSchema },
    ],
    function_call: { name: 'makeRecommendation' },
    max_tokens: 1000,
    temperature: 0.7,
  });

  const parsedResponse = await response.json();

  const calledFunction = JSON.parse(
    parsedResponse.choices[0].message.function_call.arguments,
  );

  return new Response(JSON.stringify(calledFunction.recommendations), {
    headers: new Headers({ 'Cache-Control': 'no-cache' }),
  });

Is there an equivalent of OpenAIStream and StreamingTextResponse to similarly abstract non-streaming responses? Is that somewhere you'd be open to a contribution?


const h = require('@huggingface/inference')

const Hf = new h.HfInference(process.env.HUGGINGFACE_API_KEY)


// Request the HuggingFace API for the response
async function start() {
  const response = await Hf.textGenerationStream({
    model: 'oasst-sft-4-pythia-12b-epoch-3.5',
    inputs: "Hello",
    parameters: {
      max_new_tokens: 200,
      // @ts-expect-error -- The HfInference API doesn't have types for `typical_p` even though it's necessary
      typical_p: 0.2,
      repetition_penalty: 1,
      truncate: 1000,
    },
  })

  const completion = response.generated_text
  console.log(completion)
}

start()

The output is undefined. What is wrong here?