nousresearch / open-reasoning-tasks Goto Github PK
View Code? Open in Web Editor NEWA comprehensive repository of reasoning tasks for LLMs (and beyond)
License: Apache License 2.0
A comprehensive repository of reasoning tasks for LLMs (and beyond)
License: Apache License 2.0
Hi,
This is an amazing initiative! The idea of compiling a comprehensive list of potential reasoning tasks for language model evaluation is really valuable. I have a couple of suggestions:
This is about all possible reasoning tasks, and not just tasks associated with datasets or a published paper? If so, I think it would be helpful to mention that in the contributing section.
If the task has an associated dataset or a paper, it would be nice to include those in the schema for reference.
Organizing the tasks in one big markdown file would make it harder to check if a task has already been added. Creating a tasks folder with each task in its own markdown file would make it much easier to check and navigate.
Following the previous structure, maybe we can render those markdown files into a github page for ease of access and search.
I can work on the folder structure, website rendering, and updating the contributing section. I just need your approval before starting to work on it.
I see in quite a few examples, there is essentially a cot for doing math steps. It happens after the final answer is provided. This is much harder for the model to learn than having the cot come before the final answer. It also doesn't help with accuracy as much, because the influence of the final answer is already in the context window before it starts doing the steps for the calculation.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.