Comments (3)
krrishdholakia
commented on July 28, 2024
cc: @YannDubs
from alpaca_eval.
YannDubs
commented on July 28, 2024
Thanks @krrishdholakia,
Note that the line you linked to is the docstring and the code is actually up to date, but thanks for the heads up I'll update the docstring!
Concerning litellm seems like a great and clean library 💯 ! We already have cohere / claude / openai and want to avoid any dependencies for those. But if you want to send a PR for a new completion function and test it on a model we haven't evaluated yet (e.g. palm-2-chat-bison) we'd love to merge both the completion function and the evaluations.
- Documentation for contributing a completion function
- Documentation for contributing a model
If you open a PR I'd be happy to provide more help on how to go about it!
from alpaca_eval.
YannDubs
commented on July 28, 2024
closing because we updated the docstring.
@krrishdholakia if you end up adding litellm feel free to open a PR and I'll help you there !
from alpaca_eval.
Related Issues (20)
- Alpaca Evaluation Instruction Difficulty used also for Custom Evaluation Dataset HOT 3
- Add Phi 3 models HOT 3
- Huge performance gap when using annotator weighted_alpaca_eval_gpt4_turbo and alpaca_eval_gpt4_turbo_fn HOT 4
- Question on assumption of `model_identity` as a factor for preference on generated outputs. HOT 3
- Unexpected low judge preference for some prompts HOT 1
- Trouble with custom model hosted on OpenAI compatible endpoint HOT 1
- The code for computing instruction difficulty HOT 1
- confused about openai API HOT 1
- Preference doesn't match log_probs in `annotations.json` HOT 1
- Why is max_num_seqs allowed here? HOT 1
- tensor_parallel_size can not work HOT 1
- How to solve the problem of null appearing in the evaluation results? Thank you very much HOT 3
- ERROR:root:Error while parsing completion: HOT 1
- Evaluating a self hosted LLM through an API. HOT 1
- Details on Training GLM for Length-Controlled Winrate HOT 3
- cannot reindex HOT 3
- Discrepancy between alpaca leaderboard and Chatbot arena ELO HOT 1
- ValueError: Trailing data HOT 1
- AssertionError
- Encountering Error about cannot reindex on an axis with duplicate labels HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from alpaca_eval.