@gfortaine we actually use @microsoft/fetch-event-source for the playground to do streaming with POST 👍

Thank you all for sharing your solutions here! I agree that @smervs solution currently looks like the best option available for the openai-node package. Here's a more complete example with proper error handling and no extra dependencies:

try {
    const res = await openai.createCompletion({
        model: "text-davinci-002",
        prompt: "It was the best of times",
        max_tokens: 100,
        temperature: 0,
        stream: true,
    }, { responseType: 'stream' });
    
    res.data.on('data', data => {
        const lines = data.toString().split('\n').filter(line => line.trim() !== '');
        for (const line of lines) {
            const message = line.replace(/^data: /, '');
            if (message === '[DONE]') {
                return; // Stream finished
            }
            try {
                const parsed = JSON.parse(message);
                console.log(parsed.choices[0].text);
            } catch(error) {
                console.error('Could not JSON parse stream message', message, error);
            }
        }
    });
} catch (error) {
    if (error.response?.status) {
        console.error(error.response.status, error.message);
        error.response.data.on('data', data => {
            const message = data.toString();
            try {
                const parsed = JSON.parse(message);
                console.error('An error occurred during OpenAI request: ', parsed);
            } catch(error) {
                console.error('An error occurred during OpenAI request: ', message);
            }
        });
    } else {
        console.error('An error occurred during OpenAI request', error);
    }
}

This could probably be refactored into a streamCompletion helper function (that uses either callbacks or es6 generators to emit new messages).

Apologies there's not an easier way to do this within the SDK itself – the team will continue evaluating how to get this added natively, despite the lack of support in the current sdk generator tool we're using.

from openai-node.

You can use axios stream response type. But you still need to parse the returned data.

const res = await openai.createCompletion({
  model: "text-davinci-002",
  prompt: "Say this is a test",
  max_tokens: 6,
  temperature: 0,
  stream: true,
}, { responseType: 'stream' });

res.on('data', console.log)

from openai-node.

📦 Client and server side streaming solution via npm

Hey everyone! After some tinkering, I've created a working client and server side solution for this. The GitHub project is here, and you can try the client/browser demo here. (Demo is in React but solution is framework agnostic)

You can now drop in support for streaming chat completions in both the server (Node.js) and client (browser) via the npm package openai-ext. This solution was inspired by everyone's work above, especially @YoseptF and @edelauna. Thanks everyone for working on this together.

This solution supports stopping completions, too.

Full usage examples below.

To install via npm:

npm i openai-ext@latest

Browser / Client

👁️ View live demo

Use the following solution in a browser environment:

import { OpenAIExt } from "openai-ext";

// Configure the stream (use type ClientStreamChatCompletionConfig for TypeScript users)
const streamConfig = {
  apiKey: `123abcXYZasdf`, // Your API key
  handler: {
    // Content contains the string draft, which may be partial. When isFinal is true, the completion is done.
    onContent(content, isFinal, xhr) {
      console.log(content, "isFinal?", isFinal);
    },
    onDone(xhr) {
      console.log("Done!");
    },
    onError(error, status, xhr) {
      console.error(error);
    },
  },
};

// Make the call and store a reference to the XMLHttpRequest
const xhr = OpenAIExt.streamClientChatCompletion(
  {
    model: "gpt-3.5-turbo",
    messages: [
      { role: "system", content: "You are a helpful assistant." },
      { role: "user", content: "Tell me a funny joke." },
    ],
  },
  streamConfig
);

// If you'd like to stop the completion, call xhr.abort(). The onDone() handler will be called.
xhr.abort();

Node.js / Server

Use the following solution in a Node.js or server environment:

import { Configuration, OpenAIApi } from 'openai';
import { OpenAIExt } from "openai-ext";

const apiKey = `123abcXYZasdf`; // Your API key
const configuration = new Configuration({ apiKey });
const openai = new OpenAIApi(configuration);

// Configure the stream (use type ServerStreamChatCompletionConfig for TypeScript users)
const streamConfig = {
  openai: openai,
  handler: {
    // Content contains the string draft, which may be partial. When isFinal is true, the completion is done.
    onContent(content, isFinal, stream) {
      console.log(content, "isFinal?", isFinal);
    },
    onDone(stream) {
      console.log('Done!');
    },
    onError(error, stream) {
      console.error(error);
    },
  },
};

const axiosConfig = {
  // ...
};

// Make the call to stream the completion
OpenAIExt.streamServerChatCompletion(
  {
    model: 'gpt-3.5-turbo',
    messages: [
      { role: 'system', content: 'You are a helpful assistant.' },
      { role: 'user', content: 'Tell me a funny joke.' },
    ],
  },
  streamConfig,
  axiosConfig
);

If you'd like to stop the completion, call stream.destroy(). The onDone() handler will be called.

const response = await OpenAIExt.streamServerChatCompletion(...);
const stream = response.data;
stream.destroy();

You can also stop completion using an Axios cancellation in the Axios config.

Let me know if you have any suggestions. PRs always welcome! I'll be adding a live demo to the project's Storybook site soon.

Update: I've added support for Node.js/server chat completion streaming. This library now supports both server and client. Woohoo 😎

Update 2: Live React demo now available -- view here

from openai-node.

I solved it using the inbuilt node http / https module:

const prompt = "Sample prompt. What's 2+2?"

const req = https.request({
	hostname:"api.openai.com",
	port:443,
	path:"/v1/completions",
	method:"POST",
	headers:{
		"Content-Type":"application/json",
		"Authorization":"Bearer "+ KEY_API  
	}
}, function(res){
	res.on('data', (chunk) => {
		console.log("BODY: "+chunk);
	});
	res.on('end', () => {
		console.log('No more data in response.');
	});
})

const body = JSON.stringify({
	model:"text-davinci-003",
	prompt:prompt,
	temperature:0.6,
	max_tokens:512,
	top_p:1.0,
	frequency_penalty:0.5,
	presence_penalty:0.7,
	stream:true
})

req.on('error', (e) => {
	console.error("problem with request:"+e.message);
		});

req.write(body)

req.end()

from openai-node.

For server-side typescript can try:

const response = await openai.createChatCompletion({
    model: "gpt-3.5-turbo",
    messages: messages,
    stream: true,
}, { responseType: 'stream' });

const stream = response.data as unknown as IncomingMessage

stream.on('data', (chunk: Buffer) => {
   // Messages in the event stream are separated by a pair of newline characters.
   const payloads = chunk.toString().split("\n\n")
   for (const payload of payloads) {
       if (payload.includes('[DONE]')) return;
       if (payload.startsWith("data:")) {
           const data = payload.replaceAll(/(\n)?^data:\s*/g, ''); // in case there's multiline data event
           try {
               const delta = JSON.parse(data.trim())
               console.log(delta.choices[0].delta?.content)
           } catch (error) {
               console.log(`Error with JSON.parse and ${payload}.\n${error}`)
           }
       }
   }
})

stream.on('end', () => console.log('Stream done'))
stream.on('error', (e: Error) => console.error(e))

from openai-node.

@smervs your code is working for me, but it logs as

<Buffer 64 61 74 61 3a 20 7b 22 69 64 22 3a 20 22 63 6d 70 6c 2d 36 4a 6e 56 35 4d 70 4d 41 44 4f 41 61 56 74 50 64 30 56 50 72 45 42 4f 62 34 48 54 6c 22 2c ... 155 more bytes>

Do you know how to parse this response?

here

res.data.on('data', data => console.log(data.toString()))

from openai-node.

This format still waits and gives you the entire response at the end though no? Is there not a way to get the results as they stream back as per the OpenAI frontend?

from openai-node.

@gfortaine we actually use @microsoft/fetch-event-source for the playground to do streaming with POST 👍

Thank you all for sharing your solutions here! I agree that @smervs solution currently looks like the best option available for the openai-node package. Here's a more complete example with proper error handling and no extra dependencies:

try {
    const res = await openai.createCompletion({
        model: "text-davinci-002",
        prompt: "It was the best of times",
        max_tokens: 100,
        temperature: 0,
        stream: true,
    }, { responseType: 'stream' });
    
    res.data.on('data', data => {
        const lines = data.toString().split('\n').filter(line => line.trim() !== '');
        for (const line of lines) {
            const message = line.replace(/^data: /, '');
            if (message === '[DONE]') {
                return; // Stream finished
            }
            try {
                const parsed = JSON.parse(message);
                console.log(parsed.choices[0].text);
            } catch(error) {
                console.error('Could not JSON parse stream message', message, error);
            }
        }
    });
} catch (error) {
    if (error.response?.status) {
        console.error(error.response.status, error.message);
        error.response.data.on('data', data => {
            const message = data.toString();
            try {
                const parsed = JSON.parse(message);
                console.error('An error occurred during OpenAI request: ', parsed);
            } catch(error) {
                console.error('An error occurred during OpenAI request: ', message);
            }
        });
    } else {
        console.error('An error occurred during OpenAI request', error);
    }
}

This could probably be refactored into a streamCompletion helper function (that uses either callbacks or es6 generators to emit new messages).

Apologies there's not an easier way to do this within the SDK itself – the team will continue evaluating how to get this added natively, despite the lack of support in the current sdk generator tool we're using.

@schnerd Here it is (streamCompletion helper function code inspired by this snippet, courtesy of @rauschma) 👍 :

// https://2ality.com/2018/04/async-iter-nodejs.html#generator-%231%3A-from-chunks-to-lines
async function* chunksToLines(chunksAsync) {
  let previous = "";
  for await (const chunk of chunksAsync) {
    const bufferChunk = Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk);
    previous += bufferChunk;
    let eolIndex;
    while ((eolIndex = previous.indexOf("\n")) >= 0) {
      // line includes the EOL
      const line = previous.slice(0, eolIndex + 1).trimEnd();
      if (line === "data: [DONE]") break;
      if (line.startsWith("data: ")) yield line;
      previous = previous.slice(eolIndex + 1);
    }
  }
}

async function* linesToMessages(linesAsync) {
  for await (const line of linesAsync) {
    const message = line.substring("data :".length);

    yield message;
  }
}

async function* streamCompletion(data) {
  yield* linesToMessages(chunksToLines(data));
}

try {
  const completion = await openai.createCompletion(
    {
      model: "text-davinci-003",
      max_tokens: 100,
      prompt: "It was the best of times",
      stream: true,
    },
    { responseType: "stream" }
  );

  for await (const message of streamCompletion(completion.data)) {
    try {
      const parsed = JSON.parse(message);
      const { text } = parsed.choices[0];

      process.stdout.write(text);
    } catch (error) {
      console.error("Could not JSON parse stream message", message, error);
    }
  }

  process.stdout.write("\n");
} catch (error) {
  if (error.response?.status) {
    console.error(error.response.status, error.message);

    for await (const data of error.response.data) {
      const message = data.toString();

      try {
        const parsed = JSON.parse(message);

        console.error("An error occurred during OpenAI request: ", parsed);
      } catch (error) {
        console.error("An error occurred during OpenAI request: ", message);
      }
    }
  } else {
    console.error("An error occurred during OpenAI request", error);
  }
}

from openai-node.

There are examples above that show how to consume the token stream with events and callbacks, but a potentially simpler alternative is to use an async iterator that yields tokens, even if you ultimately want to just push each token into a stream or callback anyways.

Usage:

const messages = [
    { role: 'user', content: 'what is the meaning of life?' }
]

let answer = ''
for await (const token of streamChatCompletion(messages)) {
    answer += token
    // do something async here if you want
}
console.log('answer finished:', answer)

The implementation is easy because an Axios' response set to streaming already lets you consume its chunks with an async iterator:

const { OpenAIApi } = require('openai')
const openai = new OpenAIApi(...)

async function* streamChatCompletion(messages) {
    const response = await openai.createChatCompletion(
        {
            model: 'gpt-3.5-turbo',
            messages,
            stream: true,
        },
        {
            responseType: 'stream',
        },
    )

    for await (const chunk of response.data) {
        const lines = chunk
            .toString('utf8')
            .split('\n')
            .filter((line) => line.trim().startsWith('data: '))

        for (const line of lines) {
            const message = line.replace(/^data: /, '')
            if (message === '[DONE]') {
                return
            }

            const json = JSON.parse(message)
            const token = json.choices[0].delta.content
            if (token) {
                yield token
            }
        }
    }
}

You can see this impl in my Telegram bot: https://github.com/danneu/telegram-chatgpt-bot/blob/24b76f880094b87a5c0a9a42c3571bbecfb12caa/openai.ts#L25

Or, instead of returning an async iterator, you just replace yield token with stream.push(token) or onToken(token) or whatever makes most sense for your app (remembering to change function* back to function).

from openai-node.

Thanks! @smervs currently getting: Property 'on' does not exist on type 'AxiosResponse<CreateCompletionResponse, any>' when trying though - have you had any luck?

can you try this?

res.data.on('data', console.log)

from openai-node.

Here is a fetch based client fully generated from SDK auto-generation tool 🎉 cc @schnerd @santimirandarp @blakeross @gtokman @dan-kwiat : #45 (comment)

(Bonus : it is wrapped by @vercel/fetch to provide retry (429 Network Error, ...) & DNS caching)

import { createConfiguration, OpenAIApi } from "@fortaine/openai";
import { streamCompletion } from "@fortaine/openai/stream";

import dotenv from "dotenv-flow";
dotenv.config({
  node_env: process.env.APP_ENV || process.env.NODE_ENV || "development",
  silent: true,
});

const configurationOpts = {
  authMethods: {
    apiKeyAuth: {
      accessToken: process.env.OPENAI_API_KEY,
    },
  },
};

const configuration = createConfiguration(configurationOpts);

const openai = new OpenAIApi(configuration);

try {
  const completion = await openai.createCompletion({
    model: "text-davinci-003",
    prompt: "1,2,3,",
    max_tokens: 193,
    temperature: 0,
    stream: true,
  });

  for await (const message of streamCompletion(completion)) {
    try {
      const parsed = JSON.parse(message);
      const { text } = parsed.choices[0];

      process.stdout.write(text);
    } catch (error) {
      console.error("Could not JSON parse stream message", message, error);
    }
  }
  process.stdout.write("\n");
} catch (error) {
  if (error.code) {
    try {
      const parsed = JSON.parse(error.body);
      console.error("An error occurred during OpenAI request: ", parsed);
    } catch (error) {
      console.error("An error occurred during OpenAI request: ", error);
    }
  } else {
    console.error("An error occurred during OpenAI request", error);
  }
}

from openai-node.

@schnerd Please find a PR : #45, as well as an updated example. Comments are welcome 👍 :

http://www.github.com/gfortaine/fortbot

import { Configuration, OpenAIApi } from "@fortaine/openai";
import { streamCompletion } from "@fortaine/openai/stream";

const configuration = new Configuration({
  apiKey: process.env.OPENAI_API_KEY,
});
const openai = new OpenAIApi(configuration);

try {
  const completion = await openai.createCompletion(
    {
      model: "text-davinci-003",
      max_tokens: 100,
      prompt: "It was the best of times",
      stream: true,
    },
    { responseType: "stream" }
  );

  for await (const message of streamCompletion(completion.data)) {
    try {
      const parsed = JSON.parse(message);
      const { text } = parsed.choices[0];

      process.stdout.write(text);
    } catch (error) {
      console.error("Could not JSON parse stream message", message, error);
    }
  }

  process.stdout.write("\n");
} catch (error) {
  if (error.response?.status) {
    console.error(error.response.status, error.message);

    for await (const data of error.response.data) {
      const message = data.toString();

      try {
        const parsed = JSON.parse(message);

        console.error("An error occurred during OpenAI request: ", parsed);
      } catch (error) {
        console.error("An error occurred during OpenAI request: ", message);
      }
    }
  } else {
    console.error("An error occurred during OpenAI request", error);
  }
}

from openai-node.

@gfortaine This solution works great with next.js API endpoints running on localhost. But once you deploy to Vercel, streaming responses via serverless functions are prohibited by AWS Lambda. You can get around this limitation by switching to next.js' experimental new Edge runtime, but then as far as I can tell that doesn't work with axios... which your solution relies on. So I still haven't found a way to actually stream openAI responses via next.js in production. Any ideas?

from openai-node.

java okHttpClient

BufferedSource source = response.body().source(); Buffer buffer = new Buffer(); StringBuilder result = new StringBuilder(); while (!source.exhausted()) { long count = response.body().source().read(buffer, 8192); // handle data in buffer. String r = buffer.readUtf8(); log.info("result:" + r); result.append(r); buffer.clear(); }

result eg ：
(非常多的这样的数据)
data: {"id": "cmpl-xxxx", "object": "text_completion", "created": 1672230176, "choices": [{"text": "\u672f", "index": 0, "logprobs": null, "finish_reason": null}], "model": "text-davinci-003"}
data: {"id": "cmpl-xxxx", "object": "text_completion", "created": 1672230176, "choices": [{"text": "\uff1a", "index": 0, "logprobs": null, "finish_reason": null}], "model": "text-davinci-003"}

from openai-node.

Yes I also found this strange, sometimes the OpenAI API returns multiple segments of
data: {}
that are not comma seperated and hence hard to parse as JSON
What I did:
string replace all "data: {" with ", {" instead of the first occurence (there just use "{")

then it can be parsed via JSON.parse, and one can extract all the text parts via .choices[0].text

from openai-node.

In my use case streams is more useful for the request data though, so that you can concatenate the results from different requests.

There is no dependency here apart from dotenv.

This is for the response anyways. Uses fetch which is now built into node v19 (and prev. versions using experimental flags)

import * as dotenv from 'dotenv';

// I just used a story as a string with backticks
import { text } from './string.mjs';
dotenv.config();

const apiUrl = 'https://api.openai.com/v1/completions';
const apiKey = process.env.OPENAI_API_KEY;

const fetchOptions = {
  method: 'POST',
  headers: {
    Accept: 'application/json',
    'Content-Type': 'application/json',
    Authorization: `Bearer ${apiKey}`,
  },
  body: JSON.stringify({
    model: 'text-davinci-003',
    //queues the model to return a summary, works fine.
    prompt: `Full Text: ${text}
         Summary:`,
    temperature: 0,
    max_tokens: 1000,
    presence_penalty: 0.0,
    stream: true,
    //    stop: ['\n'],
  }),
};

fetch(apiUrl, fetchOptions).then(async (response) => {
  const r = response.body;
  if (!r) throw new Error('No response body');
   
  const d = new TextDecoder('utf8');
  const reader = await r.getReader();
  let fullText = ''
  while (true) {
    const { value, done } = await reader.read();
    if (done) {
      console.log('done');
      break;
    } else {
      const decodedString = d.decode(value);
      console.log(decodedString);
      try {
        //fixes string not json-parseable otherwise
        fullText += JSON.parse(decodedString.slice(6)).choices[0].text;
      } catch (e) {
        // the last line is data: [DONE] which is not parseable either, so we catch that.
        console.log(
          e, '\n\n\n\n'
          'But parsed string is below\n\n\n\n',
        );
        console.log(fullText);
      }
    }
  }
});

Also simplest code without any library:

/* eslint-disable camelcase */
import * as dotenv from 'dotenv';

import { text } from './string.mjs';

//populates `process.env` with .env variables
dotenv.config();

const apiUrl = 'https://api.openai.com/v1/completions';
const apiKey = process.env.OPENAI_API_KEY;

const fetchOptions = {
  method: 'POST',
  headers: {
    Accept: 'application/json',
    'Content-Type': 'application/json',
    Authorization: `Bearer ${apiKey}`,
  },
  body: JSON.stringify({
    model: 'text-davinci-003',
    prompt: `Full Text: ${text}
         Summary:`,
    temperature: 0,
    max_tokens: 1000,
    presence_penalty: 0.0,
    //    stream: true,
    //    stop: ['\n'],
  }),
};

fetch(apiUrl, fetchOptions).then(async (response) => {
  const r = await response.json();
  console.log(r);
});

from openai-node.

My take on it, with punctuation detection to prevent response over-spamming

const openAiCompletion = async (messages, onText) => {
    try {
        const response = await fetch('https://api.openai.com/v1/chat/completions', {
            method: 'POST',
            headers: {
                'Authorization': `Bearer ${OPENAI_TOKEN}`,
                'Content-Type': 'application/json',
            },
            body: JSON.stringify({
                messages,
                model: "gpt-3.5-turbo",
                max_tokens: 2048,
                stream: true
            })
        });

        const decoder = new TextDecoder('utf8');
        const reader = response.body.getReader();

        let fullText = ''
        let lastFire = 0

        async function read() {
            const { value, done } = await reader.read();

            if (done) return onText(fullText)

            const delta = decoder.decode(value).match(/"delta":\s*({.*?"content":\s*".*?"})/)?.[1]

            if (delta) {
                const content = JSON.parse(delta).content

                fullText += content

                //Detects punctuation, if yes, fires onText once per .5 sec
                if (/[\p{P}\p{S}]/u.test(content)) {
                    const now = Date.now();

                    if (now - lastFire > 500) {
                        lastFire = now
                        onText(fullText)
                    }
                }
            }

            await read()

        }

        await read()

        return fullText
    } catch (error) {
        return error;
    }
}

use it like this

const aiResponse = await openAiCompletion(prompt, (text) => { 
//update UI or whatever
})
        
//do something with full response
console.log(aiResponse)

from openai-node.

I've no idea why none of the anwers posted before worked for me, what ended up working is something similar to what #18 (comment) said.

So in case there's someone else still looking for another options, here's what ended up working for me:

 const chat= async (
    prompt: string,
    previousChats: IpreviousChats[],
  ) => {
    const apiKey = window.localStorage.getItem(LOCAL_STORAGE_KEY);
    const url = "https://api.openai.com/v1/chat/completions";

    const xhr = new XMLHttpRequest();
    xhr.open("POST", url);
    xhr.setRequestHeader("Content-Type", "application/json");
    xhr.setRequestHeader("Authorization", "Bearer " + apiKey);

    xhr.onprogress = function(event) {
      console.log("Received " + event.loaded + " bytes of data.");
      console.log("Data: " + xhr.responseText);
      const newUpdates = xhr.responseText
      .replace("data: [DONE]", "")
      .trim()
      .split('data: ')
      .filter(Boolean)

      
      const newUpdatesParsed = newUpdates.map((update) => {
        const parsed = JSON.parse(update);
        return parsed.choices[0].delta?.content || '';
      }
      );

      const newUpdatesJoined = newUpdatesParsed.join('')
      console.log('current message so far',newUpdatesJoined);
    };

    xhr.onreadystatechange = function() {
      if (xhr.readyState === 4) {
        if (xhr.status === 200) {
          console.log("Response complete.");
          console.log("Final data: " + xhr.responseText);
        } else {
          console.error("Request failed with status " + xhr.status);
        }
      }
    };

    const data = JSON.stringify({
      model: currentChatModelRef.current,
      messages: [
        ...previousChats,
        {
          role: "user",
          content: prompt,
        }],
      temperature: 0.5,
      frequency_penalty: 0,
      presence_penalty: 0,
      stream: true,
    });

    xhr.send(data);
  }

good luck everyone :DDD

from openai-node.

I'm using code that's roughly like this:

let response = await fetch("https://api.openai.com/v1/engines/davinci/completions",
  {
    headers: {
      "Content-Type": "application/json",
      Authorization: "Bearer " + OPENAI_KEY,
    },
    method: "POST",
    body: JSON.stringify({
      prompt: selected,
      temperature: 0.75,
      top_p: 0.95,
      max_tokens: 10,
      stream: true,
      stop: ["\n\n"],
    }),
  }
);

const reader = es.body?.pipeThrough(new TextDecoderStream()).getReader();

while (true) {
  const res = await reader?.read();
  if (res?.done) break;
  console.log(res?.value);
}

Look at the console.log output to see the format - you have to trim the data: part off, and then JSON.parse() it.

from openai-node.

Here's a quick and dirty workaround.

Edit: If you are using NextJS, a better solution can be found here https://vercel.com/blog/gpt-3-app-next-js-vercel-edge-functions.

Server-Side:

// Import the Readable stream module
import { Readable } from "stream";

// Set the response headers
res.setHeader("Content-Type", "text/event-stream");
res.setHeader("Cache-Control", "no-cache");
res.setHeader("Connection", "keep-alive");

// Generate the response using the OpenAI API
const response = await openai.createCompletion({
    prompt: "It was the best of times",
    stream: true,
    ...
}, { responseType: 'stream' });

// Convert the response to a Readable stream (this is a temporary workaround)
const stream = response.data as any as Readable;

// Process the data stream
let streamHead = true; // Flag to indicate whether a message begins the stream or is a continuation
stream.on("data", (chunk) => {
    try {
        // Parse the chunk as a JSON object
        const data = JSON.parse(chunk.toString().trim().replace("data: ", ""));
        console.log(data);
        // Write the text from the response to the output stream
        res.write(JSON.stringify({text: data.choices[0].text, streamHead: streamHead}));
        streamHead = false;
        // Send immediately to allow chunks to be sent as they arrive
        res.flush();
    } catch (error) {
        // End the stream but do not send the error, as this is likely the DONE message from createCompletion
        console.error(error);
        res.end();
    }
});

// Send the end of the stream on stream end
stream.on("end", () => {
    res.end();
});

// If an error is received from the completion stream, send an error message and end the response stream
stream.on("error", (error) => {
    console.error(error);
    res.end(JSON.stringify({ error: true, message: "Error generating response." }));
});

Client-Side:

// Query your endpoint
const res = await fetch('/yourapi/', {...})
// Create a reader for the response body
const reader = res.body.getReader();
// Create a decoder for UTF-8 encoded text
const decoder = new TextDecoder("utf-8");
let result = "";
// Function to read chunks of the response body
const readChunk = async () => {
    return reader.read().then(({ value, done }) => {
        if (!done) {
            const dataString = decoder.decode(value);
            const data = JSON.parse(dataString);
            console.log(data);

            if (data.error) {
                console.error("Error while generating content: " + data.message);
            } else {
                result = data.streamHead ? data.text : result + data.text;
                return readChunk();
            }
        } else {
            console.log("done");
        }
    });
};

await readChunk();

The result variable is updated as the content arrives.

from openai-node.

i finally managed to get the stream working on chrome, but not firefox. The route i picked yields tokens with the async iterator and write them as the body of the response, and that is picked up as a readable stream.

index.js

setResult('');
  const response = await fetch("/api/generate", {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ 
      querySystem: islandSystemInput, 
      queryUser: islandUserInput 
    }),
  });

  const reader = response.body?.pipeThrough(new TextDecoderStream()).getReader();
  
  while (true) {
    const res = await reader?.read();
    if (res?.value?.toString() !== undefined){
      setResult(result => result + res?.value);
    }
    if (res?.done) break; 
  }
}

generate.js

async function* streamChatCompletion(messages) {
  const completion = await openai.createCompletion(
      {
          model: 'gpt-4-0314',
          messages: messages,
          max_tokens: 10,
          stream: true,
          stop: ["\n\n"],
      },
      {
          responseType: 'stream',
      },
  )

  for await (const chunk of completion.data) {
      const lines = chunk
          .toString('utf8')
          .split('\n')
          .filter((line) => line.trim().startsWith('data: '))

      for (const line of lines) {
          const message = line.replace(/^data: /, '')
          if (message === '[DONE]') {
              return
          }
          const json = JSON.parse(message)
          const token = json.choices[0].delta.content
          if (token) {
            yield token;
          }
      }
  }
}

my generate.js is mostly vanilla otherwise, just calling the above function with a for await loop

I hope this is helpful for someone.

from openai-node.

I created an MIT library that helps you compose and run AI pipelines (including GPT streaming). It supports Vercel Edge Functions out of the box.
Check it out at https://client.aigur.dev

from openai-node.

Here's another SDK that supports streaming but in a more first class way https://github.com/fern-openai/openai-node

It also has callbacks for onError and onFinish

If you have feedback, please file an issue on that repo! You can make a PR as a proof of concept, but the SDK is autogenerated by Fern so any code changes will have to go in the generator.

from openai-node.

@gtokman @blakeross may be useful: https://github.com/dan-kwiat/openai-edge

from openai-node.

Here you have the custom hook I'm using with React, in case you find it useful.
setInputMessages receives the list of messages from which we expect a completion and triggers the fetch. partialText returns the partial text of the response as it is received in real time, and finally fullText returns the response when it is complete. You have the types I use at the beginning of the file. It still lacks error handling, as it is still under development.
I'm sorry if there is any bad practice or incorrectness, as I'm fairly new to React.

import { useState, useEffect } from 'react'

enum Role {
  ASSISTANT = 'assistant',
  USER = 'user',
}

type Message = {
  role: Role
  content: string
}

const API_URL = 'https://api.openai.com/v1/chat/completions'
const OPENAI_API_KEY = import.meta.env.VITE_OPENAI_API_KEY
const OPENAI_CHAT_MODEL = 'gpt-3.5-turbo'

const utf8Decoder = new TextDecoder('utf-8')

const decodeResponse = (response?: Uint8Array) => {
  if (!response) {
    return ''
  }

  const pattern = /"delta":\s*({.*?"content":\s*".*?"})/g
  const decodedText = utf8Decoder.decode(response)
  const matches: string[] = []

  let match
  while ((match = pattern.exec(decodedText)) !== null) {
    matches.push(JSON.parse(match[1]).content)
  }
  return matches.join('')
}

export const useStreamCompletion = () => {
  const [partialText, setPartialText] = useState('')
  const [fullText, setFullText] = useState('')
  const [inputMessages, setInputMessages] = useState<Message[]>([])
  const abortController = new AbortController()

  useEffect(() => {
    if (!inputMessages.length) return

    const onText = (text: string) => {
      setPartialText(text)
    }

    const fetchData = async () => {
      try {
        const response = await fetch(API_URL, {
          method: 'POST',
          headers: {
            Authorization: `Bearer ${OPENAI_API_KEY}`,
            'Content-Type': 'application/json',
          },
          body: JSON.stringify({
            messages: inputMessages,
            model: OPENAI_CHAT_MODEL,
            stream: true,
          }),
          signal: abortController.signal, // assign the abort controller signal to the fetch request
        })

        if (!response.ok) {
          const error = await response.json()
          throw new Error(error.error)
        }

        if (!response.body) throw new Error('No response body')

        const reader = response.body.getReader()

        let fullText = ''

        async function read() {
          const { value, done } = await reader.read()

          if (done) return onText(fullText)

          const delta = decodeResponse(value)

          if (delta) {
            fullText += delta
            onText(fullText.trim())
          }

          await read()
        }

        await read()

        setFullText(fullText)
      } catch (error) {
        console.error(error)
      }
    }

    fetchData()

    return () => {
      abortController.abort()
    }
  }, [inputMessages])

  return { partialText, fullText, setInputMessages }
}

from openai-node.

This post really blew up 😆 here's the code I actually wound up using for this, works on both frontend and backend, not yet updated for chat. It's okay code. Rewriting with generators could be nice.

Usage:

await streamOne(modelName, prompt, onToken, onDone, onError, options);

Source

import { Configuration, OpenAIApi } from 'openai';
const OPENAI_API_KEY = 'sk-...';
const configuration = new Configuration({ apiKey: OPENAI_API_KEY });
const api = new OpenAIApi(configuration);

type OtherOptions = {
  maxTokens?: number;
  temp?: number;
  n?: number;
  stop?: string | string[];
}

type CompleteResponse = {
  responses: string[];
  cost: number;
  tokenUsage: number;
  promptTokens: number;
  completionTokens: number;
};

type CompleteOneResponse = {
  response: string;
  cost: number;
  tokenUsage: number;
  promptTokens: number;
  completionTokens: number;
};

export type OpenAIModel =
  'ada' |
  'babbage' |
  'curie' |
  'davinci' |
  'text-ada-001' |
  'text-babbage-001' |
  'text-curie-001' |
  'text-davinci-001' |
  'text-davinci-002' |
  'text-davinci-003' |
  'code-cushman-001' |
  'code-davinci-002';

export const modelToPrice: Record<OpenAIModel, number> = {
  'ada': 0.0004,
  'babbage': 0.0005,
  'curie': 0.002,
  'davinci': 0.02,
  'text-ada-001': 0.0004,
  'text-babbage-001': 0.0005,
  'text-curie-001': 0.002,
  'text-davinci-001': 0.02,
  'text-davinci-002': 0.02,
  'text-davinci-003': 0.02,
  'code-cushman-001': 0.0,
  'code-davinci-002': 0.0,
};

export async function complete(model: OpenAIModel, prompt: string, otherOptions?: OtherOptions): Promise<CompleteResponse> {
  const price = modelToPrice[model];
  if (price === undefined) throw new Error('Unknown model: ' + model);

  let max_tokens = otherOptions?.maxTokens || 80;
  let temperature = otherOptions?.temp || 0.3;
  let n = otherOptions?.n || 1;
  let best_of = n;
  let stop = otherOptions?.stop || '\n';
  let frequency_penalty = 0.2;
  let presence_penalty = 0.1;

  const res = await api.createCompletion({ model, prompt, max_tokens, temperature, n, best_of, stop, frequency_penalty, presence_penalty, stream: false });

  const tokenUsage = res.data.usage.total_tokens;
  const cost = tokenUsage / 1000 * price;
  const responses = res.data.choices.map(choice => choice.text);
  const promptTokens = res.data.usage.prompt_tokens;
  const completionTokens = res.data.usage.completion_tokens;

  return { responses, cost, tokenUsage, promptTokens, completionTokens };
}

export async function completeOne(model: OpenAIModel, prompt: string, otherOptions?: OtherOptions): Promise<CompleteOneResponse> {
  const res = await complete(model, prompt, otherOptions);
  return { response: res.responses[0], cost: res.cost, tokenUsage: res.tokenUsage, promptTokens: res.promptTokens, completionTokens: res.completionTokens };
}

type OnToken = (token: string) => any;
type OnDone = () => any;
type OnError = (msg: string) => any;

// TODO: estimate prompt tokens / cost including prompt tokens
export async function streamOne(model: OpenAIModel, prompt: string, onToken: OnToken, onDone: OnDone, onError: OnError, otherOptions?: OtherOptions): Promise<void> {
  // verify model and get price
  const price = modelToPrice[model];
  if (price === undefined) throw new Error('Unknown model: ' + model);

  // set options
  let max_tokens = otherOptions?.maxTokens || 80;
  let temperature = otherOptions?.temp || 0.3;
  let n = otherOptions?.n || 1;
  let best_of = n;
  let stop = otherOptions?.stop || '\n';
  let frequency_penalty = 0.2;
  let presence_penalty = 0.1;

  // create stream
  const fetchPromise = fetch(`https://api.openai.com/v1/completions`, {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'Authorization': `Bearer ${OPENAI_API_KEY}`,
    },
    body: JSON.stringify({ model, prompt, max_tokens, temperature, n, best_of, stop, frequency_penalty, presence_penalty, stream: true }),
  });
  const response = await fetchPromise;
  const reader = response.body.getReader();
  const decoder = new TextDecoder();

  // keep track of tokens
  let concat = '';
  let completionTokenCount = 0;

  // read stream
  let gotReaderDone = false;
  let gotDoneMessage = false;
  while (true) {
    // get next chunk
    const { done, value } = await reader.read();
    if (done) {
      gotReaderDone = true;
      break;
    }
    const text = decoder.decode(value);

    // split chunk into lines
    // todo: there's probs a better way to do this
    const lines = text.split('\n').filter(line => line.trim() !== '');
    for (const line of lines) {
      // remove the data: prefix
      const lineMessage = line.replace(/^data: /, '');

      // if we got the done message, stop
      if (lineMessage === '[DONE]') {
        gotDoneMessage = true;
        break; // return;
      }

      // try to parse the line as JSON, and if it works, get the token and call the callback
      try {
        const parsed = JSON.parse(lineMessage);
        const token = parsed.choices[0].text;
        concat += token;
        completionTokenCount++;
        onToken(token);
      } catch (error) {
        // todo: handle error better -- retry? inform caller?
        console.error(`Could not JSON parse stream message`, { text, lines, line, lineMessage, error });

        try {
          let errorInfo = JSON.parse(text);
          console.error(`Error info`, errorInfo);
          if (errorInfo.message) return onError(errorInfo.message);
          if (errorInfo.error.message) return onError(errorInfo.error.message);
        } catch (error) {
          // ignore
        }
      }
    }
    if (gotDoneMessage) break;
  }

  const cost = completionTokenCount / 1000 * price;
  console.log(`Streamed ${completionTokenCount} tokens. $${cost}`);
  console.log('Final text:', concat);
  onDone();
}

from openai-node.

My take on it, with punctuation detection to prevent response over-spamming

const openAiCompletion = async (messages, onText) => {
    try {
        const response = await fetch('https://api.openai.com/v1/chat/completions', {
            method: 'POST',
            headers: {
                'Authorization': `Bearer ${OPENAI_TOKEN}`,
                'Content-Type': 'application/json',
            },
            body: JSON.stringify({
                messages,
                model: "gpt-3.5-turbo",
                max_tokens: 2048,
                stream: true
            })
        });

        const decoder = new TextDecoder('utf8');
        const reader = response.body.getReader();

        let fullText = ''
        let lastFire = 0

        async function read() {
            const { value, done } = await reader.read();

            if (done) return onText(fullText)

            const delta = decoder.decode(value).match(/"delta":\s*({.*?"content":\s*".*?"})/)?.[1]

            if (delta) {
                const content = JSON.parse(delta).content

                fullText += content

                //Detects punctuation, if yes, fires onText once per .5 sec
                if (/[\p{P}\p{S}]/u.test(content)) {
                    const now = Date.now();

                    if (now - lastFire > 500) {
                        lastFire = now
                        onText(fullText)
                    }
                }
            }

            await read()

        }

        await read()

        return fullText
    } catch (error) {
        return error;
    }
}

use it like this

const aiResponse = await openAiCompletion(prompt, (text) => { 
//update UI or whatever
})
        
//do something with full response
console.log(aiResponse)

I have found some errors in the execution of the code due to the regex because sometimes I have found multiple deltas.

I leave here my solution in case it could be useful for someone.

  __cleanResponse = (res) => {
          const deltas = [];
          const splitted = res.split('\n');
          let finished = false;
          for (let i = 0; i < splitted.length; i += 1) {
              try {
                  let test = splitted[i];
                  if (test === 'data: [DONE]' || test.indexOf('data: [DONE]') >= 0) {
                      finished = true;
                      continue;
                  }
                  if (test.length === 0) continue;
                  if (test.startsWith('data:')) test = test.slice(5);
                  test = JSON.parse(test);
                  if (test.choices[0].delta.content) deltas.push(test.choices[0].delta.content);
              } catch (err) {
                  console.error(Error during message clean: " + err);
                  continue;
              }
          }
          return { deltas, finished };
      }

    __readResponse = async (reader, previousText = '', cbStream = () => { }) => {
        let fullText = previousText;
        const decoder = new TextDecoder('utf8');

        const { value, done } = await reader.read();

        if (done) return cbStream(fullText);
        const { deltas, finished } = this.__cleanResponse(decoder.decode(value));
        let inserted = false;
        if (deltas.length > 0) {
            for (let deltaIndex = 0; deltaIndex < deltas.length; deltaIndex += 1) {
                if (!inserted) inserted = true;
                fullText += deltas[deltaIndex];
            }
            if (inserted) cbStream(fullText);
        }
        if (!finished) fullText += await this.__readResponse(reader, fullText, cbStream);
        return fullText;
    }

  function ask() {
    let fullText = '';
    fetch('https://api.openai.com/v1/chat/completions', {
                method: 'POST',
                headers: {
                    Authorization: "Bearer" + this.apiKey,
                    'Content-Type': 'application/json',
                },
                body: JSON.stringify({
                    messages,
                    model: 'gpt-3.5-turbo',
                    stream: true,
                }),
            }).then(async (res) => {
                const reader = res.body.getReader();
                fullText += await this.__readResponse(reader, fullText, cbStream);
            }).catch((err) => {
                error = true;
                console.error(err);
            });
  }

from openai-node.

I'm not sure if this is useful for you guys, but here's my modified version of eventsource with added support for setting method and payload/body which should be sufficient to have a fully featured SSE connection to openai https://gist.github.com/FurriousFox/f43eaf9645302e51ab01cf0b1853aa4e

something like this should then work

const EventSource = require("./eventsource.js");

let es = new EventSource("https://api.openai.com/v1/chat/completions", {
    method: "POST",
    headers: {
        "Content-Type": "application/json",
        "Authorization": `Bearer ${api_key}`
    },
    payload: JSON.stringify({
        "model": "gpt-3.5-turbo",
        "messages": [{
            "role": "system",
            "content": instruction
        }, {
            "role": "user",
            "content": prompt
        }],
        stream: true,
    }),
});
es.onmessage = (e) => {
    if (e.data == "[DONE]") {
        es.close();
    } else {
        let delta = JSON.parse(e.data).choices[0].delta.content;
        if (delta) {
            console.log(delta);
        }
    }
};

from openai-node.

I've developed a custom hook in React with TypeScript to stream data from the OpenAI API. This solution is based on several approaches listed here in the comments.
https://github.com/MatchuPitchu/open-ai

The custom hook allows streaming API responses, as well as canceling the response stream and resetting the messages context. Additionally, the project supports code syntax highlighting and the ability to copy code snippets to the clipboard, as well as displaying additional meta data for each response.

import { useCallback, useState } from 'react';
import type { DeepRequired } from '@/utils/type-helpers';

export type GPT35 = 'gpt-3.5-turbo' | 'gpt-3.5-turbo-0301';
export type GPT4 = 'gpt-4' | 'gpt-4-0314' | 'gpt-4-32k' | 'gpt-4-32k-0314';
export type Model = GPT35 | GPT4;

export type ChatRole = 'user' | 'assistant' | 'system' | '';

export type ChatCompletionResponseMessage = {
  content: string; // content of the completion
  role: ChatRole; // role of the person/AI in the message
};

export type ChatMessageToken = ChatCompletionResponseMessage & {
  timestamp: number;
};

export type ChatMessageParams = ChatCompletionResponseMessage & {
  timestamp?: number; // timestamp of completed request
  meta?: {
    loading?: boolean; // completion state
    responseTime?: string; // total elapsed time between completion start and end
    chunks?: ChatMessageToken[]; // returned chunks of completion stream
  };
};

export type ChatMessage = DeepRequired<ChatMessageParams>;

export type ChatCompletionChunk = {
  id: string;
  object: string;
  created: number;
  model: Model;
  choices: {
    delta: Partial<ChatCompletionResponseMessage>;
    index: number;
    finish_reason: string | null;
  }[];
};

type RequestOptions = {
  headers: Record<string, string>;
  method: 'POST';
  body: string;
  signal: AbortSignal;
};

export type OpenAIStreamingProps = {
  apiKey: string;
  model: Model;
};

const OPENAI_COMPLETIONS_URL = 'https://api.openai.com/v1/chat/completions';
const MILLISECONDS_PER_SECOND = 1000;

const updateLastItem = <T>(currentItems: T[], updatedLastItem: T) => {
  const newItems = currentItems.slice(0, -1);
  newItems.push(updatedLastItem);
  return newItems;
};

// transform chat message structure with metadata to a limited shape that OpenAI API expects
const getOpenAIRequestMessage = ({ content, role }: ChatMessage): ChatCompletionResponseMessage => ({
  content,
  role
});

const getOpenAIRequestOptions = (
  apiKey: string,
  model: Model,
  messages: ChatMessage[],
  signal: AbortSignal
): RequestOptions => ({
  headers: {
    'Content-Type': 'application/json',
    Authorization: `Bearer ${apiKey}`
  },
  method: 'POST',
  body: JSON.stringify({
    model,
    messages: messages.map(getOpenAIRequestMessage),
    // TODO: define value: max_tokens: 100,
    stream: true
  }),
  signal
});

// transform chat message into a chat message with metadata
const createChatMessage = ({ content, role, meta }: ChatMessageParams): ChatMessage => ({
  content,
  role,
  timestamp: Date.now(),
  meta: {
    loading: false,
    responseTime: '',
    chunks: [],
    ...meta
  }
});

export const useOpenAIChatStream = ({ model, apiKey }: OpenAIStreamingProps) => {
  const [messages, setMessages] = useState<ChatMessage[]>([]);
  const [controller, setController] = useState<AbortController | null>(null);
  const [isLoading, setIsLoading] = useState<boolean>(false);

  const resetMessages = () => setMessages([]);

  const abortStream = () => {
    // abort fetch request by calling abort() on the AbortController instance
    if (!controller) return;
    controller.abort();
    setController(null);
  };

  const closeStream = (startTimestamp: number) => {
    // determine the final timestamp, and calculate the number of seconds the full request took.
    const endTimestamp = Date.now();
    const differenceInSeconds = (endTimestamp - startTimestamp) / MILLISECONDS_PER_SECOND;
    const formattedDiff = `${differenceInSeconds.toFixed(2)}s`;

    // update last entry of message list with the final details
    setMessages((prevMessages) => {
      const lastMessage = prevMessages.at(-1);
      if (!lastMessage) return [];

      const updatedLastMessage = {
        ...lastMessage,
        timestamp: endTimestamp,
        meta: {
          ...lastMessage.meta,
          loading: false,
          responseTime: formattedDiff
        }
      };

      return updateLastItem(prevMessages, updatedLastMessage);
    });
  };

  const submitPrompt = useCallback(
    async (newPrompt: ChatMessageParams[]) => {
      // a) no new request if last stream is loading
      // b) no request if empty string as prompt
      if (isLoading || !newPrompt[0].content) return;

      setIsLoading(true);

      const startTimestamp = Date.now();
      const chatMessages: ChatMessage[] = [...messages, ...newPrompt.map(createChatMessage)];

      const newController = new AbortController();
      const signal = newController.signal;
      setController(newController);

      try {
        const response = await fetch(
          OPENAI_COMPLETIONS_URL,
          getOpenAIRequestOptions(apiKey, model, chatMessages, signal)
        );

        if (!response.body) return;
        // read response as data stream
        const reader = response.body.getReader();
        const decoder = new TextDecoder('utf-8');

        // placeholder for next message that will be returned from API
        const placeholderMessage = createChatMessage({ content: '', role: '', meta: { loading: true } });
        let currentMessages = [...chatMessages, placeholderMessage];

        // eslint-disable-next-line no-constant-condition
        while (true) {
          const { done, value } = await reader.read();
          if (done) {
            closeStream(startTimestamp);
            break;
          }
          // parse chunk of data
          const chunk = decoder.decode(value);
          const lines = chunk.split(/(\n){2}/);

          const parsedLines: ChatCompletionChunk[] = lines
            .map((line) => line.replace(/(\n)?^data:\s*/, '').trim()) // remove 'data:' prefix
            .filter((line) => line !== '' && line !== '[DONE]') // remove empty lines and "[DONE]"
            .map((line) => JSON.parse(line)); // parse JSON string

          for (const parsedLine of parsedLines) {
            let chunkContent: string = parsedLine.choices[0].delta.content ?? '';
            chunkContent = chunkContent.replace(/^`\s*/, '`'); // avoid empty line after single backtick
            const chunkRole: ChatRole = parsedLine.choices[0].delta.role ?? '';

            // update last message entry in list with the most recent chunk
            const lastMessage = currentMessages.at(-1);
            if (!lastMessage) return;

            const updatedLastMessage = {
              content: `${lastMessage.content}${chunkContent}`,
              role: `${lastMessage.role}${chunkRole}` as ChatRole,
              timestamp: 0,
              meta: {
                ...lastMessage.meta,
                chunks: [
                  ...lastMessage.meta.chunks,
                  {
                    content: chunkContent,
                    role: chunkRole,
                    timestamp: Date.now()
                  }
                ]
              }
            };

            currentMessages = updateLastItem(currentMessages, updatedLastMessage);
            setMessages(currentMessages);
          }
        }
      } catch (error) {
        if (signal.aborted) {
          console.error(`Request aborted`, error);
        } else {
          console.error(`Error during chat response streaming`, error);
        }
      } finally {
        setController(null); // reset AbortController
        setIsLoading(false);
      }
    },
    [apiKey, isLoading, messages, model]
  );

  return { messages, submitPrompt, resetMessages, isLoading, abortStream };
};

from openai-node.

Unfortunately streaming is not currently supported by this library 😢

I'm not sure if the SDK auto-generation tool we use (openai-generator) is able to support event streams. Will have to do more research.

The python openai package does support it: https://pypi.org/project/openai/

If anyone knows of a good way to consume server-sent events in Node (that also supports POST requests), please share!

from openai-node.

Thanks! @smervs currently getting: Property 'on' does not exist on type 'AxiosResponse<CreateCompletionResponse, any>' when trying though - have you had any luck?

from openai-node.

Many thanks for this very insightful discussion 👍

As a side note, it looks like that one could consume Server-Sent Events in Node and at the same supports POST requests (even if it is not spec compliant given that only GET requests should be allowed) cc @schnerd :

• @microsoft/fetch-event-source

• launchdarkly-eventsource

However, it appears that we would lose all the benefits of SDK auto-generation tool. Moreover, it seems that the only TS generator supporting stream at the time of writing is the axios one (typescript-fetch doesn’t expose a method to consume the body as stream).

Hence, @smervs' answer is perfectly valid and should be the accepted one. However, we could enhance it, especially regarding the parser because a few options exist. By example, if we take the one from a customized @microsoft/fetch-event-source (note : the package has been specially retrofitted for the purpose by exporting ./parse), here is the result :

http://www.github.com/gfortaine/fortbot

import { Configuration, OpenAIApi } from "openai";
import * as parse from "@fortaine/fetch-event-source/parse";

const configuration = new Configuration({
  apiKey: process.env.OPENAI_API_KEY,
});
const openai = new OpenAIApi(configuration);

const prompt = "Hello world";
// https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them
const max_tokens = 4097 - prompt.length;

const completion = await openai.createCompletion(
  {
    model: "text-davinci-003",
    max_tokens,
    prompt,
    stream: true,
  },
  { responseType: "stream" }
);

completion.data.on(
  "data",
  parse.getLines(
    parse.getMessages((event) => {
      const { data } = event;

      // https://beta.openai.com/docs/api-reference/completions/create#completions/create-stream
      if (data === "[DONE]") {
        process.stdout.write("\n");
        return;
      }

      const { text } = JSON.parse(data).choices[0];
      process.stdout.write(text);
    })
  )
);

from openai-node.

@gfortaine we actually use @microsoft/fetch-event-source for the playground to do streaming with POST 👍

Thank you all for sharing your solutions here! I agree that @smervs solution currently looks like the best option available for the openai-node package. Here's a more complete example with proper error handling and no extra dependencies:

try {
    const res = await openai.createCompletion({
        model: "text-davinci-002",
        prompt: "It was the best of times",
        max_tokens: 100,
        temperature: 0,
        stream: true,
    }, { responseType: 'stream' });
    
    res.data.on('data', data => {
        const lines = data.toString().split('\n').filter(line => line.trim() !== '');
        for (const line of lines) {
            const message = line.replace(/^data: /, '');
            if (message === '[DONE]') {
                return; // Stream finished
            }
            try {
                const parsed = JSON.parse(message);
                console.log(parsed.choices[0].text);
            } catch(error) {
                console.error('Could not JSON parse stream message', message, error);
            }
        }
    });
} catch (error) {
    if (error.response?.status) {
        console.error(error.response.status, error.message);
        error.response.data.on('data', data => {
            const message = data.toString();
            try {
                const parsed = JSON.parse(message);
                console.error('An error occurred during OpenAI request: ', parsed);
            } catch(error) {
                console.error('An error occurred during OpenAI request: ', message);
            }
        });
    } else {
        console.error('An error occurred during OpenAI request', error);
    }
}

This could probably be refactored into a streamCompletion helper function (that uses either callbacks or es6 generators to emit new messages).

Apologies there's not an easier way to do this within the SDK itself – the team will continue evaluating how to get this added natively, despite the lack of support in the current sdk generator tool we're using.

Hi. Thanks for the great code. It works great in straight Node.js but in React it throws a 'res.data.on is not a function error. Maybe something to do with Webpack. Any insight would be appreciated. Thanks again.

from openai-node.

Hi everyone.@smervs solution works great with straight Node.js but in React it throws a 'res.data.on() is not a function error. Maybe something to do with Webpack. Any insight would be appreciated. Thanks again.

from openai-node.

Here is a much better solution than the hacky one I posted earlier: https://vercel.com/blog/gpt-3-app-next-js-vercel-edge-functions

I found that you need to modify the return new Response(stream) line to return new Response(stream, { headers: { 'Content-Type': 'text/event-stream' } }) in the code from the Edge example to get it to stream the text as it comes in.

from openai-node.

Tôi đã tạo một thư viện MIT giúp bạn soạn và chạy các quy trình AI (bao gồm cả phát trực tuyến GPT). Nó hỗ trợ các Chức năng Vercel Edge ngay lập tức. Kiểm tra nó tại https://client.aigur.dev

@yairhaimo How to integrate for pure Vuejs application, using javascripts

from openai-node.

I released https://github.com/drorm/gish which has a full example of streaming. If you just want to see it in action, just click on the screencast in the README.
The code is at https://github.com/drorm/gish/blob/c88d39fcc97150d3107cf84eca3051fa3b18cd14/src/LLM.ts#L103 , based on a lot of the info in here. Thank you. My license is so steal at your pleasure, MIT :-).

from openai-node.

Nice work @justinmahar your solution is super easy to use and works great with Next.js and React apps 💯

from openai-node.

i finally managed to get the stream working on chrome, but not firefox. The route i picked yields tokens with the async iterator and write them as the body of the response, and that is picked up as a readable stream.

index.js

setResult('');
  const response = await fetch("/api/generate", {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ 
      querySystem: islandSystemInput, 
      queryUser: islandUserInput 
    }),
  });

  const reader = response.body?.pipeThrough(new TextDecoderStream()).getReader();
  
  while (true) {
    const res = await reader?.read();
    if (res?.value?.toString() !== undefined){
      setResult(result => result + res?.value);
    }
    if (res?.done) break; 
  }
}

generate.js

async function* streamChatCompletion(messages) {
  const completion = await openai.createCompletion(
      {
          model: 'gpt-4-0314',
          messages: messages,
          max_tokens: 10,
          stream: true,
          stop: ["\n\n"],
      },
      {
          responseType: 'stream',
      },
  )

  for await (const chunk of completion.data) {
      const lines = chunk
          .toString('utf8')
          .split('\n')
          .filter((line) => line.trim().startsWith('data: '))

      for (const line of lines) {
          const message = line.replace(/^data: /, '')
          if (message === '[DONE]') {
              return
          }
          const json = JSON.parse(message)
          const token = json.choices[0].delta.content
          if (token) {
            yield token;
          }
      }
  }
}

my generate.js is mostly vanilla otherwise, just calling the above function with a for await loop

I hope this is helpful for someone.

It will work in Firefox if u set Content-Type header in the response of /api/generate endpoint this is know issue still didn't fix.

from openai-node.

Inspired by @YoseptF I was able to accomplish the same using just the openai SDK. This utilized Typescript, React, and Zustand.

type ConversationMessage = CreateChatCompletionRequest["messages"][0] & {
  id?: string;
};

export interface ChatCompletionSlice {
  conversationMessages: ConversationMessage[];
  createChatCompletion: (
    request: Omit<CreateChatCompletionRequest, "model">
  ) => Promise<void>;
  resetConversation: () => void;
}

export const createChatCompletionSlice: StateCreator<
  MainSlice & ChatCompletionSlice,
  [],
  [],
  ChatCompletionSlice
> = (set, get) => ({
  conversationMessages: [],
  createChatCompletion: async (request) => {
    const { messages: requestMessages, stream, ...restRequest } = request;
    // Add users message in
    set({
      conversationMessages: get().conversationMessages.concat(requestMessages),
    });
    const messages = get().conversationMessages.map(
      ({ id, ...restMessage }) => restMessage
    );
    const requestData: CreateChatCompletionRequest = {
      messages,
      stream,
      ...restRequest,
      model: get().selectedModelId,
    };

    if (stream) {
      openAi.createChatCompletion(requestData, {
        onDownloadProgress(event: ProgressEvent) {
          const target = event.target as XMLHttpRequest;
          const newUpdates = target.responseText
            .replace("data: [DONE]", "")
            .trim()
            .split("data: ")
            .filter(Boolean);
          let id = "";
          const newUpdatesParsed: string[] = newUpdates.map((update) => {
            const parsed = JSON.parse(update);
            id = parsed.id;
            return parsed.choices[0].delta?.content || "";
          });
          const newUpdatesJoined = newUpdatesParsed.join("");
          const existingMessages = get().conversationMessages.map((x) => x);
          const existingMessageIndex = existingMessages.findLastIndex(
            (x) => x.role === "assistant" && x.id === id
          );

          if (existingMessageIndex !== -1) {
            existingMessages[existingMessageIndex].content = newUpdatesJoined;
            set({
              conversationMessages: existingMessages,
            });
          } else {
            set({
              conversationMessages: existingMessages.concat([
                {
                  role: "assistant",
                  content: newUpdatesJoined,
                  id,
                },
              ]),
            });
          }
        },
      });
    } else {
      const response = await openAi.createChatCompletion(requestData);

      if (response.data) {
        const newMessages: CreateChatCompletionRequest["messages"] =
          response.data.choices
            .filter((x) => x.message !== undefined)
            .map((x) => ({
              role: x.message!.role,
              content: x.message!.content,
            }));
        set({
          conversationMessages: get().conversationMessages.concat(newMessages),
        });
      }
    }
  },
  resetConversation: () => {
    set({ conversationMessages: [] });
  },
});

from openai-node.

@smervs your code is working for me, but it logs as

<Buffer 64 61 74 61 3a 20 7b 22 69 64 22 3a 20 22 63 6d 70 6c 2d 36 4a 6e 56 35 4d 70 4d 41 44 4f 41 61 56 74 50 64 30 56 50 72 45 42 4f 62 34 48 54 6c 22 2c ... 155 more bytes>

Do you know how to parse this response?

from openai-node.

@gfortaine Have got it working using fetch directly instead of the openAI lib but I believe there's a bug with chunksToLine. It appears to assume that chunks will be >= 1 line, but chunks can actually be part of a line. @rauschma's original implementation addresses this.

from openai-node.

@shawnswed You can use onDownloadProgress which you can pass to the createCompletion options. You can use @schnerd 's snippet on progressEvent.currentTarget.response.

from openai-node.

Please move the conversation about the Aigur Client elsewhere.

Note for those using the new ChatGPT model in combination with streaming. The snippet from @schnerd has to be updated from parsed.choices[0].text to parsed.choices[0].delta.content. The typings of CreateChatCompletionResponseChoicesInner do not match when using streaming.

from openai-node.

Related to @ponytojas solution I have found that most of the time the first response contains several token blocks, so when the regular expression is applied it only takes the first one and discards the others

I've added this function for decoding:

const utf8Decoder = new TextDecoder('utf-8')

const decodeResponse = (response?: Uint8Array) => {
  if (!response) {
    return ''
  }

  const pattern = /"content"\s*:\s*"([^"]*)"/g
  const decodedText = utf8Decoder.decode(response)
  const matches: string[] = []

  let match
  while ((match = pattern.exec(decodedText)) !== null) {
    matches.push(match[1])
  }

  return matches.join('')
}