nousresearch / open-reasoning-tasks Goto Github PK

Hi,

This is an amazing initiative! The idea of compiling a comprehensive list of potential reasoning tasks for language model evaluation is really valuable. I have a couple of suggestions:

1. This is about all possible reasoning tasks, and not just tasks associated with datasets or a published paper? If so, I think it would be helpful to mention that in the contributing section.

2. If the task has an associated dataset or a paper, it would be nice to include those in the schema for reference.

3. Organizing the tasks in one big markdown file would make it harder to check if a task has already been added. Creating a tasks folder with each task in its own markdown file would make it much easier to check and navigate.

4. Following the previous structure, maybe we can render those markdown files into a github page for ease of access and search.

I can work on the folder structure, website rendering, and updating the contributing section. I just need your approval before starting to work on it.

I see in quite a few examples, there is essentially a cot for doing math steps. It happens after the final answer is provided. This is much harder for the model to learn than having the cot come before the final answer. It also doesn't help with accuracy as much, because the influence of the final answer is already in the context window before it starts doing the steps for the calculation.