[BMRT][load_bmodel:1573] INFO:Loading bmodel from [../../bmodels/rwkv6-1b5_bf16_1dev.bmodel]. Thanks for your patience...
[BMRT][load_bmodel:1501] INFO:pre net num: 0, load net num: 1
[bmlib_memory][error] bm_alloc_gmem failed, dev_id = 0, size = 0xc0fb4000

du -shb ../../bmodels/rwkv6-1b5_bf16_1dev.bmodel
3244769616      ../../bmodels/rwkv6-1b5_bf16_1dev.bmodel

du -shb ../../../LLM-TPU/bmodels/llama2-7b_int4_1dev.bmodel 
3888418068      ../../../LLM-TPU/bmodels/llama2-7b_int4_1dev.bmodel

root@bm1684:/data/LLM-TPU/models/Llama3/python_demo# python3 pipeline.py -m /data/models/llama3-8b_int8_1dev_512.bmodel -t ../token_config/ --devid 0
None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
Load ../token_config/ ...
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Device [ 0 ] loading ....
[BMRT][bmcpu_setup:436] INFO:cpu_lib 'libcpuop.so' is loaded.
bmcpu init: skip cpu_user_defined
open usercpu.so, init user_cpu_init 
[BMRT][BMProfile:60] INFO:Profile For arch=3
[BMRT][BMProfileDeviceBase:190] INFO:gdma=0, tiu=0, mcu=0
Model[/data/models/llama3-8b_int8_1dev_512.bmodel] loading ....
[BMRT][load_bmodel:1696] INFO:Loading bmodel from [/data/models/llama3-8b_int8_1dev_512.bmodel]. Thanks for your patience...
[BMRT][load_bmodel:1583] INFO:Bmodel loaded, version 2.2+v1.6.beta.0-243-ga948d0acb-20240507
[BMRT][load_bmodel:1585] INFO:pre net num: 0, load net num: 69
[BMRT][load_tpu_module:1674] INFO:loading firmare in bmodel
[BMRT][preload_funcs:1876] INFO: core_id=0, multi_fullnet_func_id=27
[BMRT][preload_funcs:1879] INFO: core_id=0, dynamic_fullnet_func_id=28
Done!

=================================================================
1. If you want to quit, please enter one of [q, quit, exit]
2. To create a new chat session, please enter one of [clear, new]
=================================================================

Question: 你好 

Answer: You are Llama3, a helpful AI assistant.博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博士博

  File "C:\ProgramData\Anaconda3\envs\hf\lib\site-packages\transformers\models\qwen2\modeling_qwen2.py", line 696, in forward
    query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin, position_ids)
  File "C:\ProgramData\Anaconda3\envs\hf\lib\site-packages\transformers\models\qwen2\modeling_qwen2.py", line 172, in apply_rotary_pos_emb
    q_embed = (q * cos) + (rotate_half(q) * sin)
RuntimeError: The size of tensor a (16) must match the size of tensor b (512) at non-singleton dimension 1

def __init__(self, layer_id):
    super().__init__()
    self.layer_id = layer_id
    self.layer = layers[layer_id]
# input_ids, attention_mask, position_ids
def forward(self, hidden_states, position_ids, attention_mask):
    hidden_states, past_kv = self.layer(
        hidden_states,
        attention_mask=attention_mask,
        position_ids=position_ids,
        use_cache=True)
    if past_kv != None:
        present_k, present_v = past_kv
        return hidden_states.float(), present_k.float(), present_v.float()
    else:
        present_k, present_v = torch.tensor([0]),torch.tensor([0])
        return hidden_states.float(), present_k, present_v

    kv_seq_len += past_key_value.get_usable_length(kv_seq_len, self.layer_idx)
AttributeError: 'tuple' object has no attribute 'get_usable_length'

void dump_bf16_tensor(bm_handle_t bm_handle, bm_device_mem_t mem, int offset,
                      int size) {
  std::vector<uint16_t> data(size);
  bm_memcpy_d2s_partial_offset(bm_handle, data.data(), mem, size * 2, offset);
  std::cout << "-------------------------------------" << std::endl;
  fp32 t;
  for (int i = 0; i < size; i++) {
    t.bits = bf16_to_fp32_bits(data[i]);
    std::cout << t.fval << std::endl;
  }
  std::cout << "-------------------------------------" << std::endl;
}

void dump_fp16_tensor(bm_handle_t bm_handle, bm_device_mem_t mem, int offset,
                      int size) {
  std::vector<uint16_t> data(size);
  bm_memcpy_d2s_partial_offset(bm_handle, data.data(), mem, size * 2, offset);
  std::cout << "-------------------------------------" << std::endl;
  fp32 t;
  for (int i = 0; i < size; i++) {
    t.bits = fp16_ieee_to_fp32_bits(data[i]);
    std::cout << t.fval << std::endl;
  }
  std::cout << "-------------------------------------" << std::endl;
}

void dump_fp32_tensor(bm_handle_t bm_handle, bm_device_mem_t mem, int offset,
                      int size) {
  std::vector<float> data(size);
  std::cout << "dump size " << data.size() << std::endl;
  bm_memcpy_d2s_partial_offset(bm_handle, data.data(), mem, size * 4, offset);
  std::cout << "-------------------------------------" << std::endl;
  for (int i = 0; i < size; i++) {
    std::cout << data[i] << std::endl;
  }
  std::cout << "-------------------------------------" << std::endl;
  auto ptr = data.data();
  ptr[0] = ptr[0];
}

void dump_int_tensor(bm_handle_t bm_handle, bm_device_mem_t mem, int offset,
                     int size) {
  std::vector<int> data(size);
  bm_memcpy_d2s_partial_offset(bm_handle, data.data(), mem, size * 4, offset);
  std::cout << "-------------------------------------" << std::endl;
  for (int i = 0; i < size; i++) {
    std::cout << data[i] << std::endl;
  }
  std::cout << "-------------------------------------" << std::endl;
  auto ptr = data.data();
  ptr[0] = ptr[0];
}

root@bm1684:/home/linaro/LLM-TPU# uname -v
#1 SMP Fri Mar 1 05:32:48 CST 2024

root@bm1684:/home/linaro/LLM-TPU# cat /sys/kernel/debug/ion/bm_npu_heap_dump/summary | head -2
Summary:
[0] npu heap size:7717519360 bytes, used:0 bytes        usage rate:0%, memory usage peak 0 bytes
root@bm1684:/home/linaro/LLM-TPU# cat /sys/kernel/debug/ion/bm_vpu_heap_dump/summary | head -2
Summary:
[2] vpu heap size:1073741824 bytes, used:0 bytes        usage rate:0%, memory usage peak 0 bytes
root@bm1684:/home/linaro/LLM-TPU# cat /sys/kernel/debug/ion/bm_vpp_heap_dump/summary | head -2
Summary:
[1] vpp heap size:1073741824 bytes, used:0 bytes        usage rate:0%, memory usage peak 0 bytes

[BMRT][fix_gdma_addr:488] FATAL:gdma dst shouldn't be coeff, origin[0x1071d8000], ctx[0x1071f8000]
llama2: /home/linaro/LLM-TPU/models/Llama2/demo/demo.cpp:311: void LLama2::init(const std::vector<int>&, std::string, std::string, const float&, const float&, const float&, const int&, const int&, const string&, const string&): Assertion `true == ret' failed.
./run_demo.sh: line 26:  1890 Aborted                 ./demo/llama2 --model ../../bmodels/llama2-7b_int4_1dev.bmodel --tokenizer ./support/token_config/tokenizer.model --devid 0

root@bm1684:/home/linaro/LLM-TPU#  ./run.sh --model llama2-7b
+ model_to_demo=(["chatglm2-6b"]="ChatGLM2" ["chatglm3-6b"]="ChatGLM3" ["llama2-7b"]="Llama2" ["qwen-7b"]="Qwen" ["qwen1.5-1.8b"]="Qwen1_5" ["wizardcoder-15b"]="WizardCoder")
+ declare -A model_to_demo
+ parse_args --model llama2-7b
+ [[ 2 -gt 0 ]]
+ key=--model
+ case $key in
+ model=llama2-7b
+ shift 2
+ [[ 0 -gt 0 ]]
+ compare_date=20240110
+ '[' == pcie ']'
./run.sh: line 45: [: ==: unary operator expected
+ '[' = soc ']'
./run.sh: line 48: [: =: unary operator expected
+ '[' '' -lt 20240110 ']'
./run.sh: line 52: [: : integer expression expected
+ echo 'Driver date is , which is up to date. Continuing...'
Driver date is , which is up to date. Continuing...
+ [[ ! -n Llama2 ]]
+ pushd ./models/Llama2
/home/linaro/LLM-TPU/models/Llama2 /home/linaro/LLM-TPU
+ ./run_demo.sh
Bmodel Exists!
llama2 file Exists!
/home/linaro/LLM-TPU/models/Llama2
Demo for Qwen in BM1684X
Init Environment ...
Load ./support/token_config/tokenizer.model ... Done!
Device [ 0 ] loading ....
[BMRT][bmcpu_setup:435] INFO:cpu_lib 'libcpuop.so' is loaded.
bmcpu init: skip cpu_user_defined
open usercpu.so, init user_cpu_init 
[BMRT][BMProfile:59] INFO:Profile For arch=3
[BMRT][BMProfileDeviceBase:190] INFO:gdma=0, tiu=0, mcu=0
Model[../../bmodels/llama2-7b_int4_1dev.bmodel] loading ....
[BMRT][load_bmodel:1573] INFO:Loading bmodel from [../../bmodels/llama2-7b_int4_1dev.bmodel]. Thanks for your patience...
[BMRT][load_bmodel:1501] INFO:pre net num: 0, load net num: 69
[BMRT][fix_gdma_addr:488] FATAL:gdma dst shouldn't be coeff, origin[0x1071d8000], ctx[0x1071f8000]
llama2: /home/linaro/LLM-TPU/models/Llama2/demo/demo.cpp:311: void LLama2::init(const std::vector<int>&, std::string, std::string, const float&, const float&, const float&, const int&, const int&, const string&, const string&): Assertion `true == ret' failed.
./run_demo.sh: line 26:  1890 Aborted                 ./demo/llama2 --model ../../bmodels/llama2-7b_int4_1dev.bmodel --tokenizer ./support/token_config/tokenizer.model --devid 0

root@bm1684:/home/linaro/LLM-TPU# uname -v
#1 SMP Fri Mar 1 05:32:48 CST 2024

python3 web_demo.py --model_path ./llama2-7b_int4_1dev.bmodel --tokenizer_path ../token_config --devid 0 --generation_mode greedy

root@bm1684:/data/LLM-TPU/models/Llama2/python_demo# python3 web_demo.py --model_path ./llama2-7b_int4_1dev.bmodel --tokenizer_path ../token_config --devid 0 --generation_mode greedy
/usr/local/lib/python3.8/dist-packages/gradio_client/documentation.py:103: UserWarning: Could not get documentation group for <class 'gradio.mix.Parallel'>: No known documentation group for module 'gradio.mix'
  warnings.warn(f"Could not get documentation group for {cls}: {exc}")
/usr/local/lib/python3.8/dist-packages/gradio_client/documentation.py:103: UserWarning: Could not get documentation group for <class 'gradio.mix.Series'>: No known documentation group for module 'gradio.mix'
  warnings.warn(f"Could not get documentation group for {cls}: {exc}")
None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
Load ../token_config ...
Device [ 0 ] loading ....
[BMRT][bmcpu_setup:436] INFO:cpu_lib 'libcpuop.so' is loaded.
bmcpu init: skip cpu_user_defined
open usercpu.so, init user_cpu_init 
[BMRT][BMProfile:60] INFO:Profile For arch=3
[BMRT][BMProfileDeviceBase:190] INFO:gdma=0, tiu=0, mcu=0
Model[./llama2-7b_int4_1dev.bmodel] loading ....
[BMRT][load_bmodel:1696] INFO:Loading bmodel from [./llama2-7b_int4_1dev.bmodel]. Thanks for your patience...
[BMRT][load_bmodel:1583] INFO:Bmodel loaded, version 2.2+v1.7.beta.63-g04a8bad4c-20240409
[BMRT][load_bmodel:1585] INFO:pre net num: 0, load net num: 69
[BMRT][load_tpu_module:1674] INFO:loading firmare in bmodel
[BMRT][preload_funcs:1876] INFO: core_id=0, multi_fullnet_func_id=91
[BMRT][preload_funcs:1879] INFO: core_id=0, dynamic_fullnet_func_id=92
Done!
web_demo.py:103: GradioDeprecationWarning: The `style` method is deprecated. Please set these arguments in the constructor instead.
  user_input = gr.Textbox(show_label=False, placeholder="Input...", lines=10).style(
Running on local URL:  http://0.0.0.0:7860

To create a public link, set `share=True` in `launch()`.
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/gradio/routes.py", line 442, in run_predict
    output = await app.get_blocks().process_api(
  File "/usr/local/lib/python3.8/dist-packages/gradio/blocks.py", line 1392, in process_api
    result = await self.call_function(
  File "/usr/local/lib/python3.8/dist-packages/gradio/blocks.py", line 1111, in call_function
    prediction = await utils.async_iteration(iterator)
  File "/usr/local/lib/python3.8/dist-packages/gradio/utils.py", line 346, in async_iteration
    return await iterator.__anext__()
  File "/usr/local/lib/python3.8/dist-packages/gradio/utils.py", line 339, in __anext__
    return await anyio.to_thread.run_sync(
  File "/usr/local/lib/python3.8/dist-packages/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
  File "/usr/local/lib/python3.8/dist-packages/anyio/_backends/_asyncio.py", line 2144, in run_sync_in_worker_thread
    return await future
  File "/usr/local/lib/python3.8/dist-packages/anyio/_backends/_asyncio.py", line 851, in run
    result = context.run(func, *args)
  File "/usr/local/lib/python3.8/dist-packages/gradio/utils.py", line 322, in run_sync_iterator_async
    return next(iterator)
  File "/usr/local/lib/python3.8/dist-packages/gradio/utils.py", line 691, in gen_wrapper
    yield from f(*args, **kwargs)
  File "web_demo.py", line 82, in predict
    for response, history in model.stream_predict(input):
  File "/data/LLM-TPU/models/Llama2/python_demo/pipeline.py", line 172, in stream_predict
    for answer_cur, history in self._generate_predictions(tokens):
  File "/data/LLM-TPU/models/Llama2/python_demo/pipeline.py", line 180, in _generate_predictions
    next_token = self.forward_first(tokens)
AttributeError: 'Llama2' object has no attribute 'forward_first'