Server disconnected about continue HOT 8 CLOSED

ollama list
NAME ID SIZE MODIFIED
codellama:latest 36893bf9bc7f 3.8 GB 4 hours ago
wizardcoder:34b-python 1064f08e84a4 19 GB About an hour ago

from continue.

Could you share the contents of the Ollama class in your config.py?

from continue.

from continuedev.src.continuedev.core.models import Models
from continuedev.src.continuedev.core.config import CustomCommand, SlashCommand, ContinueConfig
from continuedev.src.continuedev.plugins.context_providers.github import GitHubIssuesContextProvider
from continuedev.src.continuedev.libs.llm.maybe_proxy_openai import MaybeProxyOpenAI

from continuedev.src.continuedev.plugins.steps.open_config import OpenConfigStep
from continuedev.src.continuedev.plugins.steps.clear_history import ClearHistoryStep
from continuedev.src.continuedev.plugins.steps.feedback import FeedbackStep
from continuedev.src.continuedev.plugins.steps.comment_code import CommentCodeStep
from continuedev.src.continuedev.plugins.steps.share_session import ShareSessionStep
from continuedev.src.continuedev.plugins.steps.main import EditHighlightedCodeStep
from continuedev.src.continuedev.plugins.context_providers.search import SearchContextProvider
from continuedev.src.continuedev.plugins.context_providers.diff import DiffContextProvider
from continuedev.src.continuedev.plugins.context_providers.url import URLContextProvider
from continuedev.src.continuedev.plugins.context_providers.terminal import TerminalContextProvider
from continuedev.src.continuedev.libs.llm.ollama import Ollama

config = ContinueConfig(
allow_anonymous_telemetry=False,
models=Models(
default=Ollama(model="wizardcoder:34b-python")
),
system_message="",
temperature=0.5,
custom_commands=[
CustomCommand(
name="test",
description="Write unit tests for the higlighted code",
prompt="Write a comprehensive set of unit tests for the selected code. It should setup, run tests that check for correctness including important edge cases, and teardown. Ensure that the tests are complete and sophisticated. Give the tests just as chat output, don't edit any file.",
)
],
slash_commands=[
SlashCommand(
name="edit",
description="Edit code in the current file or the highlighted code",
step=EditHighlightedCodeStep,
),
SlashCommand(
name="config",
description="Customize Continue - slash commands, LLMs, system message, etc.",
step=OpenConfigStep,
),
SlashCommand(
name="comment",
description="Write comments for the current file or highlighted code",
step=CommentCodeStep,
),
SlashCommand(
name="feedback",
description="Send feedback to improve Continue",
step=FeedbackStep,
),
SlashCommand(
name="clear",
description="Clear step history",
step=ClearHistoryStep,
),
SlashCommand(
name="share",
description="Download and share the session transcript",
step=ShareSessionStep,
)
],
context_providers=[
# GitHubIssuesContextProvider(
# repo_name="/",
# auth_token=""
# ),
SearchContextProvider(),
DiffContextProvider(),
URLContextProvider(
preset_urls = [
# Add any common urls you reference here so they appear in autocomplete
]
),
TerminalContextProvider(),
],
)

from continue.

Thanks. I'll try this exact setup on my own machine to see if I can reproduce. The first thing I would check is that you've pulled the model you are using, but I'm quite confident you already know to do this. I'll give an update soon

from continue.

From one of the ollama founders, it sounds like they are having issues with 34b models with long context inputs.

I'm wondering whether the first message you send (when chat history is empty, or right after loading ollama) might work, while later messages don't because of the built up context. Unfortunately I don't have the hardware to test this on my own with a 34b

Another thing would be to test with a smaller model and see if the same issue occurs (I've seen great consistency with 7b models)

from continue.

From one of the ollama founders, it sounds like they are having issues with 34b models with long context inputs.

This is probably true. I see this aswell. Yeah I'll try a smaller model. Any special you recommend?

from continue.

I've been seeing really great results from codellama:13b-instruct, with the instruct part being crucial. Even 7b is decent

from continue.

I'm going to close this issue since it seems mostly resolved, but you're feel free to reopen if something is still wrong. Let me know if you have any more questions about models to use!

from continue.