AttributeError: 'NoneType' object has no attribute 'shape' about sparsegpt HOT 9 OPEN

I have the same question and I just use the code "
model = AutoModelForCausalLM.from_pretrained('llama2')
c_loss, m_loss = model(examples, labels, observations)",and it all make mistake in library function
只有“c_loss, m_loss = model(examples, labels, observations)”是自己写的代码报错，其他都是库函数报错

from sparsegpt.

same

from sparsegpt.

wondering if this is a transformers version issue

from sparsegpt.

outs[j] = layer(inps[j].unsqueeze(0), attention_mask=attention_mask)[0]
这一行代码没有加位置编码。更改之后是这样的
outs[j] = layer(inps[j].unsqueeze(0), attention_mask=attention_mask, position_ids=cache_position)[0]
@phind-justin @thistleknot

from sparsegpt.

wondering if this is a transformers version issue

hi you can refer the code fo wanda: https://github.com/locuslab/wanda/blob/main/lib/prune.py，modify function of llama.py as prune_sparsegpt()

from sparsegpt.

hi you can refer the code fo wanda: https://github.com/locuslab/wanda/blob/main/lib/prune.py，modify function of llama.py as prune_sparsegpt()

hi you can refer the code fo wanda: https://github.com/locuslab/wanda/blob/main/lib/prune.py , modify function of llama.py as prune_sparsegpt()

from sparsegpt.

I get this same error trying locally.

(venv)$ python llama.py  /data/Llama3-ChatQA-1.5-8B   c4 --sparsity 0.5
...
  File "sparsegpt/venv/lib/python3.11/site-packages/transformers/models/llama/modeling_llama.py", line 110, in forward
    inv_freq_expanded = self.inv_freq[None, :, None].float().expand(position_ids.shape[0], -1, 1)
                                                                    ^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'shape'

from sparsegpt.

@algorithmexplorer cache_position是啥

from sparsegpt.

Hi! If you are running squeeze LLM and have the same issue, this may be useful.
The issue can be attributed to the null position_ids.
First, at llama.py, check class "class Catcher(nn.Module):", in the original code, it is like this:

And position ids are obtained by:

But after I read the code of GPTQ for llama, which runs well with llama. It is written like this:

And

After changing them, I finnaly obtain the position_ids, which is a 2048 size torch.tensor. Please also make ture that the "try" command runs well, because this will affect the size of position_ids (not run will come up with size does not match error(4096 vs. 2048)).

Did not know the real reason, but works for me.

from sparsegpt.