Comments (5)
About api/openai/[...path]:
If using edge, as long as the first byte arrives within 25 seconds, the streaming can continue indefinitely.
If using nodejs, regardless of when the first byte arrives, the total transmission time will be cut off if it exceeds the max-duration.
So the conclusion is to keep api/openai/[...path] as it is——set it as an edge function.
from chatgpt-next-web.
@maxduke Could you please reopen the issue? I think there should be a way to avoid the problem of Edge Functions timing out if they don't receive an API response within 25 seconds. I found the following two links on GitHub, and both approaches set up a keep alive heartbeat in the router:
As the docs explain, Edge Functions don't have a maximum streaming time once they started streaming. So a possible workaround for this issue could be to start streaming empty strings with a 2 second interval so that we keep the connection alive while the LLM API call loads.
https://github.com/orgs/vercel/discussions/3553#discussioncomment-7131497
Specifically, apart from sending a response within 25 seconds (which I currently do) you actually have to keep sending messages periodically (unclear at what frequency, but 15 seconds resolves the issue) to keep the stream alive.
But I'm a programming newbie and can't really handle the code😂😂😂 I hope some skilled expert can implement the heartbeat to solve the 25s timeout issue. Thanks a lot!
reopened. Thanks a lot for the information. Let’s see.
from chatgpt-next-web.
The timeout is currently set to 60 seconds in nextchat, you might want to check if something else is causing the 25-second prompt to run out, Or do you have any error screenshots? Let's examine the causes of early timeouts
from chatgpt-next-web.
The timeout is currently set to 60 seconds in nextchat, you might want to check if something else is causing the 25-second prompt to run out, Or do you have any error screenshots? Let's examine the causes of early timeouts
So I think maybe Edge runtime is involved.
According to https://vercel.com/docs/functions/configuring-functions/duration.
You can't configure a maximum duration for functions using the Edge runtime. They can run indefinitely provided they send an initial response within 25 seconds.
from chatgpt-next-web.
@maxduke
Could you please reopen the issue? I think there should be a way to avoid the problem of Edge Functions timing out if they don't receive an API response within 25 seconds. I found the following two links on GitHub, and both approaches set up a keep alive heartbeat in the router:
As the docs explain, Edge Functions don't have a maximum streaming time once they started streaming. So a possible workaround for this issue could be to start streaming empty strings with a 2 second interval so that we keep the connection alive while the LLM API call loads.
https://github.com/orgs/vercel/discussions/3553#discussioncomment-7131497
Specifically, apart from sending a response within 25 seconds (which I currently do) you actually have to keep sending messages periodically (unclear at what frequency, but 15 seconds resolves the issue) to keep the stream alive.
But I'm a programming newbie and can't really handle the code😂😂😂
I hope some skilled expert can implement the heartbeat to solve the 25s timeout issue. Thanks a lot!
from chatgpt-next-web.
Related Issues (20)
- Cloudflare 的模型前面带@ 类似@cf/meta/llama-2-7b-chat-fp16 如何部署走openai通道 HOT 2
- [Bug] The issue with the large amount of accumulated content
- [Bug] When using the claude 3 opus, send a message and reply with data: [DONE], the other claude model succeeds.
- [Feature Request] docker镜像里有计划更新这个功能吗? HOT 1
- [Bug] Mask中不会默认选中azure的模型 HOT 3
- [Bug] "The Quota has been exceeded" - Version 2.14.1 (20240806.144936) - Mac M1 HOT 6
- [Bug] HOT 4
- [Bug] upstash 同步失败 HOT 1
- [Bug] empty response from server HOT 4
- [Bug] Claude3 opus 回复大段文字失败 HOT 5
- [Bug] 更新到 2.14.1 之后 Gemini Free Tier 无法使用 HOT 1
- [Bug] invalid_request_error with claude and anthropic api HOT 2
- [Feature Request] Custom Endpoint API Claude
- [Bug] 当使用让AI识别图片时,对话没有被总结,以至于后续提问完全忽略图片内容。 HOT 1
- [Bug] 新版本检测错误 HOT 4
- [Feature Request] 对话中希望增加模型名称的显示 HOT 1
- [Bug] 升級到 2.14.1 之後, google gemini 無法使用 HOT 1
- [Bug] windows 端exe无法自定义接口 HOT 1
- [Bug] `CUSTOM_MODELS`可选项配置`-all,+xxx`后导致自定义模型无法使用 HOT 1
- [Feature Request] Add support for prompt caching with Claude HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from chatgpt-next-web.