yihonglyu / llama-eval Goto Github PK

Export LLaMA PyTorch to ONNX (~120GB memory):

$ optimum-cli export onnx --model decapoda-research/llama-7b-hf --task causal-lm-with-past --for-ort --device cpu <path to onnx dir>


Verify the exported LLaMA ONNX (~90GB memory):

$ python verify_llama_7b_onnx.py


Benchmark the LLaMA PyTorch (~40GB memory):

$ python llama_7b_pt.py


Benchmark the LLaMA ONNX (~60GB memory):

1. Update model_path to your <path to onnx dir>

2. $ python llama_7b_onnx.py


Optimize the LLaMA ONNX with onnxruntime.transformers.optimizer:

1. $ python -m onnxruntime.transformers.optimizer --input <path to onnx dir>/decoder_model.onnx --output <path to optimized onnx dir>/decoder_model.onnx --num_heads 32 --hidden_size 4096 --model_type gpt2 --use_external_data_format

2. $ python -m onnxruntime.transformers.optimizer --input <path to onnx dir>/decoder_with_past_model.onnx --output <path to optimized onnx dir>/decoder_with_past_model.onnx --num_heads 32 --hidden_size 4096 --model_type gpt2 --use_external_data_format

3. Copy <path to onnx dir> to <path to optimized onnx dir> except:
     - decoder_model.onnx
     - decoder_model.onnx_data
     - decoder_model_merged.onnx
     - decoder_model_merged.onnx_data
     - decoder_with_past_model.onnx
     - decoder_with_past_model.onnx_data

4. Make sure <path to optimized onnx dir> has below:
     - decoder_model.onnx
     - decoder_model.onnx.data
     - decoder_with_past_model.onnx
     - decoder_with_past_model.onnx.data

PS. onnxruntime.transformers.optimizer has issue to optimize decoder_model_merged.onnx


Evaluation of LLaMA ONNX model:

* Per token cost = ORT(for prompt_lengths = 129) - ORT(for prompt_lengths =  1) / (129 - 1)
* Prompt cost = ORT(for prompt_lengths = 1)

PS. LLaMA PyTorch has the same metric