project-baize / baize-chatbot Goto Github PK

When I execute finetune code, it gets such a bug
RuntimeError: module must have its parameters and buffers on device cuda:0 (device_ids[0]) but found one of them on device: cuda:1

I read the technical report, but there wasn't much info about the SDF. How does it work?

Is the intention to release a more detailed paper soon or are you folks considering keeping this as closed?

New to the world of LLMs. I've been playing with the Baize Huggingface demo--it gives great responses!

I'd like to install it on my machine but after a clean install with successful pip install, app.py gives this error: cannot import name 'LlamaForCausalLM' from 'transformers'

Any guidance is appreciated, thank you!

I am now getting an error with app after first shutdown.

I had it all running 10 hours ago.

baize/demo$ python app.py $base_model $lora_model
Traceback (most recent call last):
  File "/app/ai/baize/demo/app.py", line 16, in <module>
    base_model = sys.argv[1]
                 ~~~~~~~~^^^

Upon running the command " python finetune.py 7b 32 0.0002 alpaca"

I got following error.

Traceback (most recent call last):
File "finetune.py", line 171, in
trainer.train()
File "/home/user6/venvs/py38/lib/python3.8/site-packages/transformers/trainer.py", line 1645, in train
return inner_training_loop(
File "/home/user6/venvs/py38/lib/python3.8/site-packages/transformers/trainer.py", line 1929, in _inner_training_loop
tr_loss_step = self.training_step(model, inputs)
File "/home/user6/venvs/py38/lib/python3.8/site-packages/transformers/trainer.py", line 2750, in training_step
loss = self.compute_loss(model, inputs)
File "/home/user6/venvs/py38/lib/python3.8/site-packages/transformers/trainer.py", line 2775, in compute_loss
outputs = model(**inputs)
File "/home/user6/venvs/py38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/user6/venvs/py38/lib/python3.8/site-packages/accelerate/utils/operations.py", line 553, in forward
return model_forward(*args, **kwargs)
File "/home/user6/venvs/py38/lib/python3.8/site-packages/accelerate/utils/operations.py", line 541, in call
return convert_to_fp32(self.model_forward(*args, **kwargs))
File "/home/user6/venvs/py38/lib/python3.8/site-packages/torch/amp/autocast_mode.py", line 14, in decorate_autocast
return func(*args, **kwargs)
File "/home/user6/venvs/py38/lib/python3.8/site-packages/peft/peft_model.py", line 686, in forward
return self.base_model(
File "/home/user6/venvs/py38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/user6/venvs/py38/lib/python3.8/site-packages/accelerate/hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "/home/user6/venvs/py38/lib/python3.8/site-packages/transformers/models/llama/modeling_llama.py", line 688, in forward
outputs = self.model(
File "/home/user6/venvs/py38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/user6/venvs/py38/lib/python3.8/site-packages/accelerate/hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "/home/user6/venvs/py38/lib/python3.8/site-packages/transformers/models/llama/modeling_llama.py", line 570, in forward
layer_outputs = torch.utils.checkpoint.checkpoint(
File "/home/user6/venvs/py38/lib/python3.8/site-packages/torch/utils/checkpoint.py", line 249, in checkpoint
return CheckpointFunction.apply(function, preserve, *args)
File "/home/user6/venvs/py38/lib/python3.8/site-packages/torch/autograd/function.py", line 506, in apply
return super().apply(*args, **kwargs) # type: ignore[misc]
File "/home/user6/venvs/py38/lib/python3.8/site-packages/torch/utils/checkpoint.py", line 107, in forward
outputs = run_function(*args)
File "/home/user6/venvs/py38/lib/python3.8/site-packages/transformers/models/llama/modeling_llama.py", line 566, in custom_forward
return module(*inputs, output_attentions, None)
File "/home/user6/venvs/py38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/user6/venvs/py38/lib/python3.8/site-packages/accelerate/hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "/home/user6/venvs/py38/lib/python3.8/site-packages/transformers/models/llama/modeling_llama.py", line 292, in forward
hidden_states, self_attn_weights, present_key_value = self.self_attn(
File "/home/user6/venvs/py38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/user6/venvs/py38/lib/python3.8/site-packages/accelerate/hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "/home/user6/venvs/py38/lib/python3.8/site-packages/transformers/models/llama/modeling_llama.py", line 194, in forward
query_states = self.q_proj(hidden_states).view(bsz, q_len, self.num_heads, self.head_dim).transpose(1, 2)
AttributeError: 'NoneType' object has no attribute 'view'
0%| | 0/780 [00:01<?, ?it/s]

Can you help me with this?

Hi, thank you for Baize,
Could we please have the v2 dataset?
This one is 2 months old: https://github.com/project-baize/baize-chatbot/tree/main/data

Thank you

I can run the 7b without issue but loading 13b I get the follow error. The error comes up soon as the first message is sent.

Traceback (most recent call last): File "/home/chris/miniconda3/lib/python3.10/site-packages/gradio/routes.py", line 394, in run_predict output = await app.get_blocks().process_api( File "/home/chris/miniconda3/lib/python3.10/site-packages/gradio/blocks.py", line 1075, in process_api result = await self.call_function( File "/home/chris/miniconda3/lib/python3.10/site-packages/gradio/blocks.py", line 898, in call_function prediction = await anyio.to_thread.run_sync( File "/home/chris/miniconda3/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync return await get_asynclib().run_sync_in_worker_thread( File "/home/chris/miniconda3/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread return await future File "/home/chris/miniconda3/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run result = context.run(func, *args) File "/home/chris/miniconda3/lib/python3.10/site-packages/gradio/utils.py", line 549, in async_iteration return next(iterator) File "/vol/storage/checkout/baize/demo/app.py", line 48, in predict for x in sample_decode( File "/vol/storage/checkout/baize/demo/app_modules/utils.py", line 265, in sample_decode outputs = model(input_ids) File "/home/chris/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "/home/chris/miniconda3/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward output = old_forward(*args, **kwargs) File "/home/chris/miniconda3/lib/python3.10/site-packages/peft/peft_model.py", line 579, in forward return self.base_model( File "/home/chris/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "/home/chris/miniconda3/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 687, in forward outputs = self.model( File "/home/chris/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "/home/chris/miniconda3/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 577, in forward layer_outputs = decoder_layer( File "/home/chris/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "/home/chris/miniconda3/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward output = old_forward(*args, **kwargs) File "/home/chris/miniconda3/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 292, in forward hidden_states, self_attn_weights, present_key_value = self.self_attn( File "/home/chris/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "/home/chris/miniconda3/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward output = old_forward(*args, **kwargs) File "/home/chris/miniconda3/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 196, in forward query_states = self.q_proj(hidden_states).view(bsz, q_len, self.num_heads, self.head_dim).transpose(1, 2) File "/home/chris/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "/home/chris/miniconda3/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward output = old_forward(*args, **kwargs) File "/home/chris/miniconda3/lib/python3.10/site-packages/peft/tuners/lora.py", line 591, in forward result = super().forward(x) File "/home/chris/miniconda3/lib/python3.10/site-packages/bitsandbytes/nn/modules.py", line 242, in forward out = bnb.matmul(x, self.weight, bias=self.bias, state=self.state) File "/home/chris/miniconda3/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py", line 488, in matmul return MatMul8bitLt.apply(A, B, out, bias, state) File "/home/chris/miniconda3/lib/python3.10/site-packages/torch/autograd/function.py", line 506, in apply return super().apply(*args, **kwargs) # type: ignore[misc] File "/home/chris/miniconda3/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py", line 317, in forward state.CxB, state.SB = F.transform(state.CB, to_order=formatB) File "/home/chris/miniconda3/lib/python3.10/site-packages/bitsandbytes/functional.py", line 1698, in transform prev_device = pre_call(A.device) AttributeError: 'NoneType' object has no attribute 'device'

Hello, when I run demo/app.py with 7B model, I got this problem 'addmm_impl_cpu_" not implemented for 'Half'. Could you please tell me how to fix it?
This share link expires in 72 hours. For free permanent hosting and GPU upgrades (NEW!), check out Spaces: https://huggingface.co/spaces
Traceback (most recent call last):
File "/home/chenyang/anaconda3/envs/xhd/lib/python3.8/site-packages/gradio/routes.py", line 393, in run_predict
output = await app.get_blocks().process_api(
File "/home/chenyang/anaconda3/envs/xhd/lib/python3.8/site-packages/gradio/blocks.py", line 1069, in process_api
result = await self.call_function(
File "/home/chenyang/anaconda3/envs/xhd/lib/python3.8/site-packages/gradio/blocks.py", line 892, in call_function
prediction = await anyio.to_thread.run_sync(
File "/home/chenyang/anaconda3/envs/xhd/lib/python3.8/site-packages/anyio/to_thread.py", line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "/home/chenyang/anaconda3/envs/xhd/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
File "/home/chenyang/anaconda3/envs/xhd/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 867, in run
result = context.run(func, *args)
File "/home/chenyang/anaconda3/envs/xhd/lib/python3.8/site-packages/gradio/utils.py", line 549, in async_iteration
return next(iterator)
File "app.py", line 43, in predict
for x in greedy_search(input_ids,model,tokenizer,stop_words=["[|Human|]", "[|AI|]"],max_length=max_length_tokens,temperature=temperature,top_p=top_p):
File "/media/hlt/disk/chenyang_space/chenyang_space/xhd_space/baize-main/demo/app_modules/utils.py", line 253, in greedy_search
outputs = model(input_ids)
File "/home/chenyang/anaconda3/envs/xhd/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/chenyang/anaconda3/envs/xhd/lib/python3.8/site-packages/accelerate/hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "/home/chenyang/anaconda3/envs/xhd/lib/python3.8/site-packages/peft/peft_model.py", line 575, in forward
return self.base_model(
File "/home/chenyang/anaconda3/envs/xhd/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/chenyang/anaconda3/envs/xhd/lib/python3.8/site-packages/accelerate/hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "/home/chenyang/anaconda3/envs/xhd/lib/python3.8/site-packages/transformers/models/llama/modeling_llama.py", line 687, in forward
outputs = self.model(
File "/home/chenyang/anaconda3/envs/xhd/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/chenyang/anaconda3/envs/xhd/lib/python3.8/site-packages/accelerate/hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "/home/chenyang/anaconda3/envs/xhd/lib/python3.8/site-packages/transformers/models/llama/modeling_llama.py", line 577, in forward
layer_outputs = decoder_layer(
File "/home/chenyang/anaconda3/envs/xhd/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/chenyang/anaconda3/envs/xhd/lib/python3.8/site-packages/accelerate/hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "/home/chenyang/anaconda3/envs/xhd/lib/python3.8/site-packages/transformers/models/llama/modeling_llama.py", line 292, in forward
hidden_states, self_attn_weights, present_key_value = self.self_attn(
File "/home/chenyang/anaconda3/envs/xhd/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/chenyang/anaconda3/envs/xhd/lib/python3.8/site-packages/accelerate/hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "/home/chenyang/anaconda3/envs/xhd/lib/python3.8/site-packages/transformers/models/llama/modeling_llama.py", line 196, in forward
query_states = self.q_proj(hidden_states).view(bsz, q_len, self.num_heads, self.head_dim).transpose(1, 2)
File "/home/chenyang/anaconda3/envs/xhd/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in call_impl
return forward_call(*args, **kwargs)
File "/home/chenyang/anaconda3/envs/xhd/lib/python3.8/site-packages/accelerate/hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "/home/chenyang/anaconda3/envs/xhd/lib/python3.8/site-packages/peft/tuners/lora.py", line 406, in forward
result = F.linear(x, transpose(self.weight, self.fan_in_fan_out), bias=self.bias)
RuntimeError: "addmm_impl_cpu" not implemented for 'Half'

When asked to write a simple bash script there are many errors in the script and I don't think it "understood" what needs to be written?

Some examples of errors being made:

• "n" is randomly capitalized/lower-cased and causes the script to fail
• spaces are used when defining the variable (also causing the script to fail)
• not sure where those words it's echoing are coming from?
• no markdown/code box
• etc.

Hello

I saw that you released your dataset for everyone to use, so I translated it with OpenAI's model and released it on the HF Hub. I hope it helps others who want to work on Dutch.

You can find the Quora chat set and the Stack Overflow dataset in Dutch. I've also translated the Alpaca Cleaned dataset into Dutch and also converted it into the Baize format. Feel free to add it here or anywhere with a reference to the repository.

Best

Bram

Hello, I would like to know how we can access the 13b int8 version for the demo?

I notice in current report

13B models consumes actually less memory than 7B. Is it a typo?

Hi! It's just a kind reminder that there is a typo of the github link on huggingface. Please check and update.

Hi, I would like to know if we can connect this with Langchain and Llama Index?

If not would love to be able to at some point!

Thanks

Installing the demo | pip install -r requirements.txt

I am installing it with Conda and running it in WSL on a Ubuntu Distro.

Get an error

Building wheel for SentencePiece (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [80 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-311
      creating build/lib.linux-x86_64-cpython-311/sentencepiece
      copying src/sentencepiece/__init__.py -> build/lib.linux-x86_64-cpython-311/sentencepiece
      copying src/sentencepiece/_version.py -> build/lib.linux-x86_64-cpython-311/sentencepiece
      copying src/sentencepiece/sentencepiece_model_pb2.py -> build/lib.linux-x86_64-cpython-311/sentencepiece
      copying src/sentencepiece/sentencepiece_pb2.py -> build/lib.linux-x86_64-cpython-311/sentencepiece
      running build_ext
      /bin/sh: 1: pkg-config: not found
      Cloning into 'sentencepiece'...
      Note: switching to '58f256cf6f01bb86e6fa634a5cc560de5bd1667d'.
      
      You are in 'detached HEAD' state. You can look around, make experimental
      changes and commit them, and you can discard any commits you make in this
      state without impacting any branches by switching back to a branch.
      
      If you want to create a new branch to retain commits you create, you may
      do so (now or later) by using -c with the switch command. Example:
      
        git switch -c <new-branch-name>
      
      Or undo this operation with:
      
        git switch -
      
      Turn off this advice by setting config variable advice.detachedHead to false
      
      ./build_bundled.sh: 19: cmake: not found
      ./build_bundled.sh: 20: cmake: not found
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-install-5zvpbi68/sentencepiece_e20a82f4c647494ebf367f1f3050f679/setup.py", line 136, in <module>
          setup(
        File "/home/user/miniconda3/envs/baize/lib/python3.11/site-packages/setuptools/__init__.py", line 87, in setup
          return distutils.core.setup(**attrs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/user/miniconda3/envs/baize/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
                 ^^^^^^^^^^^^^^^^^^
        File "/home/user/miniconda3/envs/baize/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/home/user/miniconda3/envs/baize/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/home/user/miniconda3/envs/baize/lib/python3.11/site-packages/setuptools/dist.py", line 1208, in run_command
          super().run_command(command)
        File "/home/user/miniconda3/envs/baize/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/home/user/miniconda3/envs/baize/lib/python3.11/site-packages/wheel/bdist_wheel.py", line 325, in run
          self.run_command("build")
        File "/home/user/miniconda3/envs/baize/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/home/user/miniconda3/envs/baize/lib/python3.11/site-packages/setuptools/dist.py", line 1208, in run_command
          super().run_command(command)
        File "/home/user/miniconda3/envs/baize/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/home/user/miniconda3/envs/baize/lib/python3.11/site-packages/setuptools/_distutils/command/build.py", line 132, in run
          self.run_command(cmd_name)
        File "/home/user/miniconda3/envs/baize/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/home/user/miniconda3/envs/baize/lib/python3.11/site-packages/setuptools/dist.py", line 1208, in run_command
          super().run_command(command)
        File "/home/user/miniconda3/envs/baize/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/home/user/miniconda3/envs/baize/lib/python3.11/site-packages/setuptools/command/build_ext.py", line 84, in run
          _build_ext.run(self)
        File "/home/user/miniconda3/envs/baize/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 346, in run
          self.build_extensions()
        File "/home/user/miniconda3/envs/baize/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 468, in build_extensions
          self._build_extensions_serial()
        File "/home/user/miniconda3/envs/baize/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 494, in _build_extensions_serial
          self.build_extension(ext)
        File "/tmp/pip-install-5zvpbi68/sentencepiece_e20a82f4c647494ebf367f1f3050f679/setup.py", line 89, in build_extension
          subprocess.check_call(['./build_bundled.sh', __version__])
        File "/home/user/miniconda3/envs/baize/lib/python3.11/subprocess.py", line 413, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['./build_bundled.sh', '0.1.97']' returned non-zero exit status 127.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for SentencePiece
  Running setup.py clean for SentencePiece
Successfully built llama_index peft transformers
Failed to build SentencePiece
Installing collected packages: tokenizers, SentencePiece, rfc3986, pytz, pydub, mpmath, lit, ffmpy, cmake, websockets, urllib3, uc-micro-py, tzdata, typing-extensions, tqdm, toolz, tenacity, sympy, socksio, sniffio, six, semantic-version, regex, pyyaml, python-multipart, pyrsistent, pypinyin, pyparsing, Pygments, psutil, pillow, packaging, orjson, nvidia-nvtx-cu11, nvidia-nccl-cu11, nvidia-cusparse-cu11, nvidia-curand-cu11, nvidia-cufft-cu11, nvidia-cuda-runtime-cu11, nvidia-cuda-nvrtc-cu11, nvidia-cuda-cupti-cu11, nvidia-cublas-cu11, numpy, networkx, mypy-extensions, multidict, mdurl, markupsafe, markdown2, markdown, latex2mathml, kiwisolver, idna, h11, greenlet, fsspec, frozenlist, fonttools, filelock, entrypoints, cycler, colorama, click, charset-normalizer, attrs, async-timeout, aiofiles, yarl, uvicorn, typing-inspect, SQLAlchemy, requests, python-dateutil, pydantic, nvidia-cusolver-cu11, nvidia-cudnn-cu11, mdtex2html, marshmallow, markdown-it-py, linkify-it-py, jsonschema, jinja2, contourpy, anyio, aiosignal, tiktoken, starlette, pandas, mdit-py-plugins, matplotlib, marshmallow-enum, huggingface-hub, httpcore, duckduckgo_search, aiohttp, transformers, openai, httpx, gradio-client, fastapi, dataclasses_json, altair, langchain, gradio, llama_index, triton, torch, accelerate, peft
  Running setup.py install for SentencePiece ... error
  error: subprocess-exited-with-error
  
  × Running setup.py install for SentencePiece did not run successfully.
  │ exit code: 1
  ╰─> [69 lines of output]
      running install
      /home/user/miniconda3/envs/baize/lib/python3.11/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-311
      creating build/lib.linux-x86_64-cpython-311/sentencepiece
      copying src/sentencepiece/__init__.py -> build/lib.linux-x86_64-cpython-311/sentencepiece
      copying src/sentencepiece/_version.py -> build/lib.linux-x86_64-cpython-311/sentencepiece
      copying src/sentencepiece/sentencepiece_model_pb2.py -> build/lib.linux-x86_64-cpython-311/sentencepiece
      copying src/sentencepiece/sentencepiece_pb2.py -> build/lib.linux-x86_64-cpython-311/sentencepiece
      running build_ext
      /bin/sh: 1: pkg-config: not found
      fatal: destination path 'sentencepiece' already exists and is not an empty directory.
      fatal: destination path 'sentencepiece' already exists and is not an empty directory.
      ./build_bundled.sh: 19: cmake: not found
      ./build_bundled.sh: 20: cmake: not found
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-install-5zvpbi68/sentencepiece_e20a82f4c647494ebf367f1f3050f679/setup.py", line 136, in <module>
          setup(
        File "/home/user/miniconda3/envs/baize/lib/python3.11/site-packages/setuptools/__init__.py", line 87, in setup
          return distutils.core.setup(**attrs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/user/miniconda3/envs/baize/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
                 ^^^^^^^^^^^^^^^^^^
        File "/home/user/miniconda3/envs/baize/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/home/user/miniconda3/envs/baize/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/home/user/miniconda3/envs/baize/lib/python3.11/site-packages/setuptools/dist.py", line 1208, in run_command
          super().run_command(command)
        File "/home/user/miniconda3/envs/baize/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/home/user/miniconda3/envs/baize/lib/python3.11/site-packages/setuptools/command/install.py", line 68, in run
          return orig.install.run(self)
                 ^^^^^^^^^^^^^^^^^^^^^^
        File "/home/user/miniconda3/envs/baize/lib/python3.11/site-packages/setuptools/_distutils/command/install.py", line 698, in run
          self.run_command('build')
        File "/home/user/miniconda3/envs/baize/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/home/user/miniconda3/envs/baize/lib/python3.11/site-packages/setuptools/dist.py", line 1208, in run_command
          super().run_command(command)
        File "/home/user/miniconda3/envs/baize/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/home/user/miniconda3/envs/baize/lib/python3.11/site-packages/setuptools/_distutils/command/build.py", line 132, in run
          self.run_command(cmd_name)
        File "/home/user/miniconda3/envs/baize/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/home/user/miniconda3/envs/baize/lib/python3.11/site-packages/setuptools/dist.py", line 1208, in run_command
          super().run_command(command)
        File "/home/user/miniconda3/envs/baize/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/home/user/miniconda3/envs/baize/lib/python3.11/site-packages/setuptools/command/build_ext.py", line 84, in run
          _build_ext.run(self)
        File "/home/user/miniconda3/envs/baize/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 346, in run
          self.build_extensions()
        File "/home/user/miniconda3/envs/baize/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 468, in build_extensions
          self._build_extensions_serial()
        File "/home/user/miniconda3/envs/baize/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 494, in _build_extensions_serial
          self.build_extension(ext)
        File "/tmp/pip-install-5zvpbi68/sentencepiece_e20a82f4c647494ebf367f1f3050f679/setup.py", line 89, in build_extension
          subprocess.check_call(['./build_bundled.sh', __version__])
        File "/home/user/miniconda3/envs/baize/lib/python3.11/subprocess.py", line 413, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['./build_bundled.sh', '0.1.97']' returned non-zero exit status 127.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> SentencePiece

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

At first, I wrote the same thing in the hugging face community, and then I realized that I should write in Github so that it would be easier for you to reach. So, I'm sorry that I opened another issue here. (And here is the revised version)

Hi, I'm Keldos from ChuanhuChatGPT. It's so nice to hear that you guys have built a demo for your great work based on our project.

In fact, we are also in the process of rapid development. Our code for demo styles is also constantly updated. I have noticed that you committed in hugging face trying to "fix dark mode", but I see nothing happened. Emm..... I have to admit that the CSS I wrote at that time was not so compatible with the principles of gradio, which led to this problem. I also tried to solve this problem in our repo in other ways yesterday, but when I saw your commit, I realized that I should combine the two...

For now, you should write the CSS part like this:

/* bright (default) */
#chuanhu_chatbot {
    background-color: var(--chatbot-color-light) !important;
    color: #000000 !important;
}
[data-testid = "bot"] {
    background-color: #FFFFFF !important;
}
[data-testid = "user"] {
    background-color: #95EC69 !important;
}
/* dark */
.dark #chuanhu_chatbot {
    background-color: var(--chatbot-color-dark) !important;
    color: #FFFFFF !important;
}
.dark [data-testid = "bot"] {
    background-color: #2C2C2C !important;
}
.dark [data-testid = "user"] {
    background-color: #26B561 !important;
}

Yes, you should delete @media (prefers-color-scheme: light) and @media (prefers-color-scheme: dark).

In this way, when the system changes from dark to bright, the chatbot will no longer automatically change to a bright background, but will be consistent with the system framework of gradio. And our settings for dark or bright mode still works when we refresh the web page.

Yesterday, I wrote a JavaScript script to make the whole gradio page adaptable when the system switches between bright and dark mode. But maybe gradio intended to automatically switch from bright to dark, and not automatically switch from dark to bright. I don't know why it want to act like this. If you like, you can check our new JavaScript script (custom.js) (In fact, when I first saw your repository, I found that you even kept this blank js file that I had reserved, which shocked me).

Today I tried to add a toggle for users to override system bright-or-dark mode for the page. It worked. Later it would be pushed and merged into our main branch.

You are welcome to contact me (you can find my email on my github profile page) if you need more help on the demo UI (styles). It's also recommended to check our amazing updates on user interface and interaction. (I am responsible for writing ChangeLog documents, but no one is pressing me, so it may not be written in time. :-))

By the way, I'm glad to tell you that we updated our wiki documents: licenses. Now you can check how to cite our work and how to specify our copyleft information in the source code. But don't worry, it's not necessarily needed. We are already very happy that you have marked the source of the demo code part.

Hope everything is going fine~

Have u encountered the following error? And what's ur torch/transformers/peft vesion?

RuntimeError: Failed to import transformers.trainer because of the following error (look up to see its traceback):
No module named 'torch._six'

If it is running on the server, how do I get it to run on the specified gpu?

https://github.com/project-baize/baize/blob/ccf0bb8485657b7c16a57456bbb835503bac2456/demo/app_modules/utils.py#L278

Was just looking through the code. Doesn't softmax ensure that all the elements add up to 1?

I have tried the 8bit option as well but no change.

It generates tokens slowly and CPU goes high (>80%). GPU jumps up too but always < 20%. So it seems to be CPU hungry instead of GPU.

So by default does it inference on GPU?

from https://huggingface.co/blog/4bit-transformers-bitsandbytes ?

hello，there are 8 gpus on my machine, how to train in parallel?

First, I'd like to thank you very much for this inspiring work!

In your paper, you mentioned "For Baize v1.5, following Vicuna (Chiang et al., 2023), we only compute loss for AI’s responses in the dialogue transcript." I am curious about your implementation and may customize it. I searched the repo but couldn't find it. Could you give me a pointer for that?

Baize support can be seamlessly integrated into cformers (SoTA Transformers with C-backend for fast inference on your CPU).

CUDA SETUP: Detected CUDA version 113
CUDA SETUP: Loading binary /opt/conda/envs/py38/lib/python3.8/site-packages/bitsandbytes/libbitsandbytes_cuda113.so...
Downloading and preparing dataset json/default to /root/.cache/huggingface/datasets/json/default-e59c3670f1657ac9/0.0.0/fe5dd6ea2639a6df622901539cb550cf8797e5a6b2dd7af1cf934bed8e233e6e...
Downloading data files: 100%|██████████| 1/1 [00:00<00:00, 2349.75it/s]
Extracting data files: 100%|██████████| 1/1 [00:00<00:00, 483.88it/s]
Traceback (most recent call last):
File "/opt/conda/envs/py38/lib/python3.8/site-packages/datasets/builder.py", line 1860, in _prepare_split_single
for _, table in generator:
File "/opt/conda/envs/py38/lib/python3.8/site-packages/datasets/packaged_modules/json/json.py", line 113, in _generate_tables
io.BytesIO(batch), read_options=paj.ReadOptions(block_size=block_size)
File "pyarrow/_json.pyx", line 55, in pyarrow._json.ReadOptions.init
File "pyarrow/_json.pyx", line 80, in pyarrow._json.ReadOptions.block_size.set
OverflowError: value too large to convert to int32_t

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "finetune.py", line 51, in
data = load_dataset("json", data_files=DATA_PATH)
File "/opt/conda/envs/py38/lib/python3.8/site-packages/datasets/load.py", line 1791, in load_dataset
builder_instance.download_and_prepare(
File "/opt/conda/envs/py38/lib/python3.8/site-packages/datasets/builder.py", line 891, in download_and_prepare
self._download_and_prepare(
File "/opt/conda/envs/py38/lib/python3.8/site-packages/datasets/builder.py", line 986, in _download_and_prepare
self._prepare_split(split_generator, **prepare_split_kwargs)
File "/opt/conda/envs/py38/lib/python3.8/site-packages/datasets/builder.py", line 1748, in _prepare_split
for job_id, done, content in self._prepare_split_single(
File "/opt/conda/envs/py38/lib/python3.8/site-packages/datasets/builder.py", line 1893, in _prepare_split_single
raise DatasetGenerationError("An error occurred while generating the dataset") from e
datasets.builder.DatasetGenerationError: An error occurred while generating the dataset

这种由于训练数据太大而出现的问题，要怎么解决呢？

Is it possible to train Chinese data directly based on English models-Baize-7B ？if not, why?

Hi, what's the openai cost for creating the self-chat data?

when execute:
"python -m fastchat.model.apply_lora --base huggyllama/llama-7b --target ./model_weights/baize-7b --lora project-baize/baize-lora-7B"
,the error "TypeError: init() got an unexpected keyword argument 'torch_dtype'" focus.

I use this repo to finetune bloomz-7b1-mt with alpaca data (50k conversation) and the results are terrible. It takes 8 hours to train with the same arguments as in how you finetune the llama. What could be the reason of this?

My PC: Windows-11, Python 3.10.10,
CPU i7 gen 12

python app.py decapoda-research/llama-7b-hf project-baize/baize-lora-7B
The tokenizer class you load from this checkpoint is not the same type as the class this function is called from. It may result in unexpected tokenization.
The tokenizer class you load from this checkpoint is 'LLaMATokenizer'.
The class this function is called from is 'LlamaTokenizer'.
Loading checkpoint shards: 100%|█████████████████████████████████████| 33/33 [00:11<00:00, 2.93it/s]
C:\Users\mihaj\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\gradio\deprecation.py:43: UserWarning: You have unused kwarg parameters in Row, please remove them: {'scale': 1}
warnings.warn(
Reloading javascript...
Running on local URL: http://127.0.0.1:7860

To create a public link, set share=True in launch().
Traceback (most recent call last):
File "C:\Users\mihaj\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\gradio\routes.py", line 394, in run_predict
output = await app.get_blocks().process_api(
File "C:\Users\mihaj\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\gradio\blocks.py", line 1075, in process_api
result = await self.call_function(
File "C:\Users\mihaj\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\gradio\blocks.py", line 898, in call_function
prediction = await anyio.to_thread.run_sync(
File "C:\Users\mihaj\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\anyio\to_thread.py", line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "C:\Users\mihaj\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\anyio_backends_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
File "C:\Users\mihaj\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\anyio_backends_asyncio.py", line 867, in run
result = context.run(func, *args)
File "C:\Users\mihaj\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\gradio\utils.py", line 549, in async_iteration
return next(iterator)
File "d:\tmp\ai\baize\demo\app.py", line 43, in predict
for x in greedy_search(input_ids,model,tokenizer,stop_words=["[|Human|]", "[|AI|]"],max_length=max_length_tokens,temperature=temperature,top_p=top_p):
File "d:\tmp\ai\baize\demo\app_modules\utils.py", line 253, in greedy_search
outputs = model(input_ids)
File "C:\Users\mihaj\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "C:\Users\mihaj\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\accelerate\hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "C:\Users\mihaj\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\peft\peft_model.py", line 575, in forward
return self.base_model(
File "C:\Users\mihaj\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "C:\Users\mihaj\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\accelerate\hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "C:\Users\mihaj\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\transformers\models\llama\modeling_llama.py", line 687, in forward
outputs = self.model(
File "C:\Users\mihaj\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "C:\Users\mihaj\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\accelerate\hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "C:\Users\mihaj\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\transformers\models\llama\modeling_llama.py", line 577, in forward
layer_outputs = decoder_layer(
File "C:\Users\mihaj\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "C:\Users\mihaj\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\accelerate\hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "C:\Users\mihaj\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\transformers\models\llama\modeling_llama.py", line 292, in forward
hidden_states, self_attn_weights, present_key_value = self.self_attn(
File "C:\Users\mihaj\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "C:\Users\mihaj\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\accelerate\hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "C:\Users\mihaj\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\transformers\models\llama\modeling_llama.py", line 196, in forward
query_states = self.q_proj(hidden_states).view(bsz, q_len, self.num_heads, self.head_dim).transpose(1, 2)
File "C:\Users\mihaj\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\torch\nn\modules\module.py", line 1501, in call_impl
return forward_call(*args, **kwargs)
File "C:\Users\mihaj\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\accelerate\hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "C:\Users\mihaj\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\peft\tuners\lora.py", line 406, in forward
result = F.linear(x, transpose(self.weight, self.fan_in_fan_out), bias=self.bias)
RuntimeError: "addmm_impl_cpu" not implemented for 'Half'

Will be very helpful to see like a table with requirement for each variation: 7B,13B and so.

decapoda-research/llama-7b-hf has been reported that the weight converted is not suitable to current transformers, please give an exact transformer version.

I am trying to use the Baize API/CLI. I have followed the instructions on the readme. When I run this line:

python3 -m fastchat.model.apply_lora --base huggyllama/llama-7b --target ./model_weights/baize-7b --lora project-baize/baize-lora-7B

I get the following error. Any ideas what's going on?

Thanks!

│ /opt/conda/lib/python3.10/site-packages/peft/peft_model.py:169 in from_pretrained │
│ │
│ 166 │ │ │
│ 167 │ │ # load the config │
│ 168 │ │ config = PEFT_TYPE_TO_CONFIG_MAPPING[ │
│ ❱ 169 │ │ │ PeftConfig.from_pretrained(model_id, subfolder=kwargs.get("subfolder", None) │
│ 170 │ │ ].from_pretrained(model_id, subfolder=kwargs.get("subfolder", None), **kwargs) │
│ 171 │ │ │
│ 172 │ │ if (getattr(model, "hf_device_map", None) is not None) and len( │
│ │
│ /opt/conda/lib/python3.10/site-packages/peft/utils/config.py:110 in from_pretrained │
│ │
│ 107 │ │ │ │ │ pretrained_model_name_or_path, CONFIG_NAME, subfolder=subfolder, **k │
│ 108 │ │ │ │ ) │
│ 109 │ │ │ except Exception: │
│ ❱ 110 │ │ │ │ raise ValueError(f"Can't find '{CONFIG_NAME}' at '{pretrained_model_name │
│ 111 │ │ │
│ 112 │ │ loaded_attributes = cls.from_json_file(config_file) │
│ 113 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: Can't find 'adapter_config.json' at 'project-baize/baize-lora-7B'

interesting paper - https://arxiv.org/abs/2305.07804

Hi,
Do you guys have any plans to make a gptq 4-bit quantized version of your models. That would cut VRAM usage and improve inference speed a lot, without much loss in capabilities. A lot of other llama/alpaca models are doing this.
I'd do it myself but I don't have the kind of RAM needed for a conversion.
Thanks for this great model. Please keep going!

How collect other topic data? such as "china history" topic, how to modify the code, in detail. Thanks!

OSError: Unable to load weights from pytorch checkpoint file for 'model_cache/models--decapoda-research--llama-30b-hf/snapshots/f991780f9362b2fcaefad066cd235058844562b7/pytorch_model-00039-of-00061.bin'. If you tried to load a PyTorch model from a TF 2.0 checkpoint, please set from_tf=True.

The other shard is fine, any suggestion?

When the bot doesn't know/is not sure about the question, it replies with such things as: "I'm sorry, I don't have the knowledge that you ask". How do you guys achieve this? What kind of training data makes the bot respond like that? I'm highly curious about this.

I collect some chinese data about "**云南" like this:

And train follow the readme base on Baize-7B, cost 48 hours, get checkpoints finally.
when I use this checkpoints to run app.py. The AI can speak Chinese, but sometimes it mixes English and Russian, and the answers are not very accurate.

Can you help me analyze the cause? Is it a lack of training data? Or the base model is an English model? Or something else? thanks

I'm trying to run batch generation but greedy_search() seems to work only for single input_ids. Curious why you guys implemented greedy_search()?

Hi, I want to use this code to finetune my own FAQ dataset and make it possible to answer FAQ. From the collect.py, it seems you only use the question to generate the chat dataset.

Do you have any suggestions on how to generate chat dataset with FAQ dataset? is it possible to give Chatgpt the topic(question) and answer, and let it generate chat?

thanks

you can use sharegpt 90k data here
https://huggingface.co/datasets/jeffwan/sharegpt_vicuna/tree/main/sharegpt_90k_raw_dataset

The previous V1 version had limited Chinese ability, and the author said that the Chinese version would be online soon. Now that V2 version is released, does this version include training of Chinese ability? Or is it just an enhancement of its ability and does not involve the training of Chinese ability.

py", line 70, in set_module_8bit_tensor_to_device
new_value = bnb.nn.Int8Params(new_value, requires_grad=False, has_fp16_weights=has_fp16_weights).to(device)
File "/opt/conda/lib/python3.10/site-packages/bitsandbytes/nn/modules.py", line 196, in to
return self.cuda(device)
File "/opt/conda/lib/python3.10/site-packages/bitsandbytes/nn/modules.py", line 160, in cuda
CB, CBt, SCB, SCBt, coo_tensorB = bnb.functional.double_quant(B)
File "/opt/conda/lib/python3.10/site-packages/bitsandbytes/functional.py", line 1616, in double_quant
row_stats, col_stats, nnz_row_ptr = get_colrow_absmax(
File "/opt/conda/lib/python3.10/site-packages/bitsandbytes/functional.py", line 1505, in get_colrow_absmax
lib.cget_col_row_stats(ptrA, ptrRowStats, ptrColStats, ptrNnzrows, ct.c_float(threshold), rows, cols)
File "/opt/conda/lib/python3.10/ctypes/init.py", line 387, in getattr
func = self.getitem(name)
File "/opt/conda/lib/python3.10/ctypes/init.py", line 392, in getitem
func = self._FuncPtr((name_or_ordinal, self))
AttributeError: /opt/conda/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.so: undefined symbol: cget_col_row_stats

Looks like bitsandbytes issue. Should I install it from source?

base_model=decapoda-research/llama-7b-hf
lora_model=project-baize/baize-lora-7B
python app.py $base_model $lora_model

same Top_p, Temperature

even 13b performs poor

hello，there are 8 gpus on my machine, how to train in parallel?

This line builds up the end of the prompt, which should be completed by LLM.

https://github.com/project-baize/baize-chatbot/blob/4fae6c4e550f087958c1f60746f0e6290e6b9f02/preprocess.py#L42C46-L42C46

            if s != "":
                prompt = "The conversation between human and AI assistant.\n"
                s = prompt + s + "[|Human|] "

Shouldn't the line end with [|AI|] and not [|HUMAN|]?