andreyanufr / who_what_benchmark Goto Github PK

Hey @andreyanufr,

Can you make an API in your repo?

Here is the proposal:

• Create a folder named whowhatbench
• Add __init__.py and evaluate.py inside
• Create a class Evaluator that receives a source model in the constructor. Expose it in the __init__.py.
• Add score(optimized_model) method to the Evaluator class that will return the accuracy metric
• Add simple setup.py to your project

Overall it should look like:

from whowhatbench import Evaluator

base_model = Auto/OVModelForCausalLM.from_pretrined(...)
evaluator = Evaluator(base_model)
... # model optimization process
metric = evaluator.score(optimized_model)

The API of the repo allows benchmarking accuracy of many optimization backends including:

• HuggingFace (4-bit via Bytesandbits)
• GPTQ
• LLama.cpp (via bigdl-llm)
• OpenVINO (via optimum)

I suggest creating examples for all four backends and demonstrating capabilities. We can also do a comparison and publish in readme.
You can use optimized versions of llama-7B from here: https://huggingface.co/TheBloke

Can we change API so that it can work with list of prompts instead of JSON files:

prompts = [
"What it PyTorch",
"Who is Mahatma Gandhi",
]
metrics_per_prompt, metrics = evaluator.score(optimized_model, prompts=prompts)

This is more flexible, IMO.