SciAssess is a comprehensive benchmark designed to evaluate the proficiency of Large Language Models (LLMs) in scientific literature analysis. It focuses on assessing LLMs' abilities in memorization, comprehension, and analysis within the context of scientific literature, covering a wide range of scientific fields such as general chemistry, organic materials, and alloy materials. SciAssess provides a rigorous and thorough assessment of LLMs, supporting the ongoing development of LLM applications in scientific literature analysis.

For more details, please refer to our paper: SciAssess: Benchmarking LLM Proficiency in Scientific Literature Analysis.

To use SciAssess, first clone the repository:

git clone https://github.com/sci-assess/SciAssess.git
cd SciAssess

Install the required dependencies:

pip install -e .
pip install Levenshtein
pip install munkres
pip install rdkit

login the wandb:

wandb login

Additional Considerations:

In some task evaluations, we use models from sentence-transformers on Hugging Face. Please ensure that you can connect to Hugging Face. If you are unable to connect, you might consider manually downloading the corresponding models and updating the model import path in ./sciassess/Implement/utils/metrics.py to reflect the location where you have placed the models.

Due to copyright restrictions, we are unable to directly distribute the original PDF of the article. You will need to download the corresponding PDF according to the instructions in README and store it in SciAssess_library/pdfs.

All articles involved in this evaluation are listed in doi.txt. You need to download the corresponding PDFs according to the DOIs and store them in SciAssess_library/pdfs.

Each PDF should be named as doi.pdf, with '/' in the DOI replaced by '_', e.g., an article with DOI 10.1002/adfm.202008332 should be named as 10.1002_adfm.202008332.pdf and placed in SciAssess_library/pdfs.

Some articles' supporting information is also evaluated. These articles' DOIs are listed in si_doi.txt. You need to download the corresponding PDFs and store them in SciAssess_library/pdfs, named as doi_si.pdf.

If you want to evaluate your own model, you need to configure your model's registration information and implementation in sciassess/Registry/completion_fns and sciassess/Implement/completion_fns, respectively. See openai/evals:completion-fns.md for configuration instructions.

Note that most evaluations depend on the article PDFs, so you may need to process the input PDFs within your model's method. The PDF file path will be passed in the __call__ function through kwargs['file_name'], and you need to handle this parameter and process the PDF in the __call__ function. See openai_with_pdf.py for an example based on PyPDF and GPT.

After completing the model configuration, run the following command to evaluate your model:

bash run_sciassess.sh your_model_name

Replace your_model_name with the name of your model (default: gpt3.5).

Remember to export your OpenAI API key as an environment variable:

export OPENAI_API_KEY=your_openai_api_key

Here is an example of evaluating a Qwen2-7B-instruct model locally:

Firstly, utilize the vllm to set up an openAI-like server of Qwen2-7B-instruct modell,and set your OPENAI_API_BASE and OPENAI_API_KEY.

such as :

export OPENAI_API_KEY="EMPTY"
export OPENAI_API_KEY="http://localhost:8000/v1"

(How to utilize the vllm to set up an openAI-like server of Qwen2-instruct-7B model? please surf the :https://huggingface.co/Qwen/Qwen2-7B-Instruct)

Secondly,download the pdfs and put it all to the ./SciAssess/SciAssess_library/pdfs

Thirdly, in the ./SciAssess/sciassess/Implement/completion_fns create a file called "qwen2.py"

and paste the code to the "qwen2.py":

import logging
from typing import List,Dict,Any,Union
from evals.api import CompletionResult
from .base_completion_fn import BaseCompletionFn
from openai import OpenAI
import time
from sciassess.Implement.completion_fns.utils import call_without_throw
from .utils import extract_text

openai_api_key="EMPTY"
openai_api_base = "http://localhost:8000/v1"
logger = logging.getLogger(__name__)

client= OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)

class Qwen2CompletionResult(CompletionResult):
    def __init__(self,response:str) -> None:
        self.response = response

    def get_completions(self) -> List[str]:
        return [self.response.strip()] if self.response else ["Unknown"]
    
class Qwen2CompletionFn(BaseCompletionFn):
    def __init__(self,**kwargs):
        super().__init__(**kwargs)
        self.model="Qwen2-7B-Instruct"

    def get_completions(self, messages, **kwargs) -> str:
        chat_response= client.chat.completions.create(
            model=self.model,
            messages=messages,
        )
        response = chat_response.choices[0].message.content
        return  Qwen2CompletionResult(response)        

    @call_without_throw
    def __call__(self, prompt: Union[str, list[dict]], **kwargs: Any):
        if isinstance(prompt, str):
            messages = [{"role": "user", "content": prompt}]
        else:
            messages = prompt
        if "file_name" in kwargs:
            attached_file_content = "\nThe file is as follows:\n\n" + "".join(extract_text(kwargs["file_name"], self.pdf_parser))
            #wyc-temp-change
            attached_file_content = attached_file_content[:2048]   
            kwargs.pop('file_name')
        else:
            attached_file_content = ""

        messages[-1]['content'] += attached_file_content
        return self.get_completions(messages=messages, **kwargs) 

Fourthly,in the ./SciAssess/sciassess/Registry/completion_fns,create a file called "qwen2.yaml"

and here is the code:

qwen2:
    class: sciassess.Implement.completion_fns.qwen2:Qwen2CompletionFn

Finally, in the project root directory :

bash run_sciassess.sh qwen2

And here we also provide a fine-tuning example,please refer to readme.md

0.9.0 (2024-03-17) Beta version first released

0.9.1 (2024-03-28) Fix critical bugs. Now the code is executable.

0.9.2 (2024-04-06) Optimize the metric of multiple choice questions.

0.9.3 (2024-04-07) Merge mmlu college chemistry and high school chemistry.

Remove abstract2title and research_question_extraction due to uncertainty of model grading.

1.0.0 (2024-04-08) Official version released

We welcome contributions to the SciAssess benchmark. If you have any suggestions or improvements, please feel free to open an issue or create a pull request.

If you use SciAssess in your research, please cite our paper:

@misc{cai2024sciassess,
      title={SciAssess: Benchmarking LLM Proficiency in Scientific Literature Analysis}, 
      author={Hengxing Cai and Xiaochen Cai and Junhan Chang and Sihang Li and Lin Yao and Changxin Wang and Zhifeng Gao and Yongge Li and Mujie Lin and Shuwen Yang and Jiankun Wang and Yuqi Yin and Yaqi Li and Linfeng Zhang and Guolin Ke},
      year={2024},
      eprint={2403.01976},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}