How to make output consistent about ollama HOT 6 CLOSED

I'm not able to reproduce this using llama2 and mistral with setting seed and temperature through both the API and the Modelfile.

What version of ollama (ollama -v) are you using? Can you also provide your Modelfile?

from ollama.

I'm not able to reproduce this using llama2 and mistral with setting seed and temperature through both the API and the Modelfile.

What version of ollama (ollama -v) are you using? Can you also provide your Modelfile?

ollama -v
ollama version is 0.1.20

cat Modelfile

FROM ./q4_0.bin

TEMPLATE """{{ if .First }}{{ .System }}{{ end }}{{ .Prompt }} [/INST]{{ .Response }} </s><s>[INST] """

SYSTEM "[INST] "
PARAMETER stop "[INST]"
PARAMETER stop "[/INST]"
PARAMETER stop "<<SYS>>"
PARAMETER stop "<</SYS>>"

PARAMETER temperature 0
PARAMETER seed 37
PARAMETER num_ctx 4096

from ollama.

Hi @mxyng, could you please take a look at the Modelfile config I provided when you get a chance?
Thanks!

from ollama.

@Fei-Wang what kind of model is q4_0.bin? The template may be incorrect. It should probably be something like this:

[INST] {{ .System }} {{ .Prompt }} [/INST]

<s> and </s> shouldn't be necessary and {{ .Response }} is (currently) ignored.

from ollama.

Hey @mxyng,

I'm working with q4_0.bin, a finetuned llama2 model, and I've hit two snags:

1. I'm using <s> and </s> as per the guide on Hugging Face (https://huggingface.co/blog/codellama#conversational-instructions). Did I get something wrong?

2. Changing the ModelFile to llama2 hasn't fixed inconsistent outputs. See the screenshot below.

Any ideas?

from ollama.

Closing this as a dupe of #1749

from ollama.