Comments (5)
Hi :). I assume your using the default 2k context window on open-webui? Until today, my project used a much larger context window if possible (like in the case of command-r). I just pushed an update which contains a new settings window, which allows you to adjust the context window. Please confirm that this causes the increase in vRAM usage / decrease in offloaded layers.
from llocalsearch.
If that's the case, I assume ollama just run out of memory on your system?
from llocalsearch.
Yes, it's certainly quicker when I lower the context window size. Though it seems to be breaking. It froze when trying to pull info from the internet here for maybe a minute or so:
ollama | [GIN] 2024/04/14 - 00:29:19 | 200 | 3.916807898s | 172.30.0.3 | POST "/api/chat"
searxng-1 | 2024-04-14 00:29:19,854 WARNING:searx.engines.google: ErrorContext('searx/search/processors/online.py', 116, "response = req(params['url'], **request_args)", 'searx.exceptions.SearxEngineTooManyRequestsException', None, ('Too many request',)) False
searxng-1 | 2024-04-14 00:29:19,854 ERROR:searx.engines.google: Too many requests
searxng-1 | Traceback (most recent call last):
searxng-1 | File "/usr/local/searxng/searx/search/processors/online.py", line 163, in search
searxng-1 | search_results = self._search_basic(query, params)
searxng-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
searxng-1 | File "/usr/local/searxng/searx/search/processors/online.py", line 147, in _search_basic
searxng-1 | response = self._send_http_request(params)
searxng-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
searxng-1 | File "/usr/local/searxng/searx/search/processors/online.py", line 116, in _send_http_request
searxng-1 | response = req(params['url'], **request_args)
searxng-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
searxng-1 | File "/usr/local/searxng/searx/network/__init__.py", line 164, in get
searxng-1 | return request('get', url, **kwargs)
searxng-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
searxng-1 | File "/usr/local/searxng/searx/network/__init__.py", line 95, in request
searxng-1 | return future.result(timeout)
searxng-1 | ^^^^^^^^^^^^^^^^^^^^^^
searxng-1 | File "/usr/lib/python3.11/concurrent/futures/_base.py", line 456, in result
searxng-1 | return self.__get_result()
searxng-1 | ^^^^^^^^^^^^^^^^^^^
searxng-1 | File "/usr/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
searxng-1 | raise self._exception
searxng-1 | File "/usr/local/searxng/searx/network/network.py", line 289, in request
searxng-1 | return await self.call_client(False, method, url, **kwargs)
searxng-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
searxng-1 | File "/usr/local/searxng/searx/network/network.py", line 272, in call_client
searxng-1 | return Network.patch_response(response, do_raise_for_httperror)
searxng-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
searxng-1 | File "/usr/local/searxng/searx/network/network.py", line 245, in patch_response
searxng-1 | raise_for_httperror(response)
searxng-1 | File "/usr/local/searxng/searx/network/raise_for_httperror.py", line 76, in raise_for_httperror
searxng-1 | raise SearxEngineTooManyRequestsException()
searxng-1 | searx.exceptions.SearxEngineTooManyRequestsException: Too many request, suspended_time=3600
searxng-1 | 2024-04-14 00:29:22,329 ERROR:searx.engines.duckduckgo: engine timeout
searxng-1 | 2024-04-14 00:29:22,423 WARNING:searx.engines.duckduckgo: ErrorContext('searx/engines/duckduckgo.py', 118, 'res = get(query_url)', 'httpx.ConnectTimeout', None, (None, None, 'duckduckgo.com')) False
searxng-1 | 2024-04-14 00:29:22,423 ERROR:searx.engines.duckduckgo: HTTP requests timeout (search duration : 3.0941880460013635 s, timeout: 3.0 s) : ConnectTimeout
backend-1 | 2024/04/14 00:29:22 WARN Error downloading website error="no content found"
And after that went through it then got stuck in a loop:
Here's the full logs:
temp.log
from llocalsearch.
I'm pretty sure that it run out of context. 2k tokens isnt much. You can see an estimate of the current context in the backend logs. I assume that the format instructions arent in the context anymore at this point. Which results in the LLM ignoring the requested structure.
from llocalsearch.
closing for #91
from llocalsearch.
Related Issues (20)
- error maxIterations must be a number with the docker version HOT 1
- Search cards hides LLM response in mobile browser
- Error on question HOT 1
- How can I change the app's root path?
- Why doesn't my search process have source boost on the left?
- searxng.engines.wikidata: Fail to initialize
- Update ollama to allow parallel model load
- Couldnt acess the localhost port 3000 HOT 1
- `maxIterations must be a number` HOT 2
- Error/warning on Apple Silicon HOT 2
- error in installation on MAC HOT 3
- build for arm / mac
- limit db result size to not "overflow" context
- Error: Post 'host.docker.internal:11434': unsupported protocol scheme 'host.docker.internal' HOT 11
- support for more than one ollama backend HOT 1
- Improve rendering for sources and the relevant text chunks
- add used tools to main context window
- make dev exception
- Vllm support (open-ai compatible api)
- Not seeming to work on M1 Mac?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from llocalsearch.