OSF Preprint Digest is a command-line tool that retrieves preprints from the Open Science Framework (OSF) generating concise summaries using the BART-large-CNN model. I was surprised to see that there was (to my knowledge at least) no readily available tool that summarises published articles in this way, so I decided to make it myself 🙂

• The tool allows the user to specify the date interval for abstracts (i.e., the period from the request), as well as the discipline.
• API calls are separately made for both the abstract and the author list, which are then combined.
• It filters the preprints to include only those with English abstracts.

Clone the repository:

git clone https://github.com/sohaamir/osf_digest.git

Change to the project directory:

cd osf_digest

Install the required dependencies:

pip install -r requirements.txt

which will install the following:

requests
tqdm
unidecode
langdetect
transformers
torch
python-dotenv

Install the CLI tool from the project root:

pip install -e .

To run the script, you need to set up the necessary API tokens, which we can then load into the script using dotenv.

To do this:

• Create a file named .env in the project root directory.
• Open the .env file and add the following lines:

OSF_TOKEN=your_osf_token
HF_TOKEN=your_hf_token

Replace your_osf_token with your OSF API token and your_hf_token with your Hugging Face API token. Both are freely available on the respective websites, you just need to make an account.

Your folder structure should look (something) like this:

├── LICENSE
├── README.md
├── cli
│   ├── __init__.py
│   ├── __pycache__
│   │   ├── __init__.cpython-311.pyc
│   │   └── cli.cpython-311.pyc
│   ├── cli.py
│   └── instructions.md
├── data
│   ├── csv
│   │   └── 2024-05-25_16-11-31_Psychiatry_Cognitive Neuroscience_7day_preprints.csv
│   └── json
│       ├── 2024-05-25_16-11-31_Cognitive Neuroscience_7day_preprints.json
│       └── 2024-05-25_16-11-31_Psychiatry_7day_preprints.json
├── osf_digest.egg-info
│   ├── PKG-INFO
│   ├── SOURCES.txt
│   ├── dependency_links.txt
│   ├── entry_points.txt
│   ├── requires.txt
│   └── top_level.txt
├── output
│   └── digests
│       └── 2024-05-25_16-13-48_Psychiatry_Cognitive Neuroscience_7day_summary.csv
├── request_preprints.py
├── requirements.txt
└── setup.py

(I have decided to include some example outputs with the folder, but feel free to delete them. The actual CLI code is contained within /cli/cli.py but the 'barebones' Python code is also provided in the request_preprints.py script in the repository's root.)

We firstly import a number of external modules/packages:

import requests
import json
from datetime import datetime, timedelta
from tqdm import tqdm
from unidecode import unidecode
from concurrent.futures import ThreadPoolExecutor, as_completed
import os
import time
from langdetect import detect
import csv
from transformers import BartTokenizer, BartForConditionalGeneration, pipeline
import torch
from dotenv import load_dotenv

• requests to get the information from OSF
• datetime to set the length of time for the request to cover
• tqdm for progress bars when pulling the information
• unidecode to sort out pesky characters (most commonly in author's names)
• concurrent.futures to speed things up
• langdetect to remove non-English abstracts
• transformers and torch to run the BART model
• dotenv to load the environmental variables (our OSF and Hugging Face tokens)

To run the OSF Preprint Digest tool, use the following command:

osf_digest --disciplines <discipline1> <discipline2> ... --days <days> --pagesize <pagesize> --max_length <max_length> --min_length <min_length>

Arguments

--disciplines: List of disciplines to retrieve preprints from (default: ['Psychiatry'])
--days: Number of days to summarize since today (default: 7, range: 0-365)
--pagesize: Number of preprints to retrieve per request (default: 100)
--max_length: Maximum length of the generated summary (default: 170)
--min_length: Minimum length of the generated summary (default: 30)

Example

osf_digest --disciplines "Psychiatry" "Neuroscience" --days 14 --pagesize 50 --max_length 200 --min_length 50

This command will retrieve preprints from the "Psychiatry" and "Neuroscience" disciplines published in the last 14 days, with a page size of 50 preprints per request. The generated summaries will have a maximum length of 200 and a minimum length of 50.

All disciplines on OSF are supported, please visit the OSF website for a complete list. You can see the discipline list for a preprint on it's webpage.

The OSF Preprint Digest tool generates the following output files:

• JSON files containing the retrieved preprints, saved in the data/json directory (a single example):

{
        "id": "j726r",
        "title": "How visual experience shapes body representation",
        "authors": [
            "Shahzad, Iqra",
            "Occelli, Valeria",
            "Giraudet, Eleonore",
            "Azanon, Elena",
            "Longo, Matthew",
            "Moruaux, Andre",
            "Collignon, Olivier"
        ],
        "abstract": [
            "We do not have a veridical representation of our body in our mind. For instance,",
            "tactile distances of equal measure along the medial-lateral axis of our limbs are generally perceived",
            "as larger than those running along the proximal-distal axis. This anisotropy in tactile distances reflects",
            "distortions in body-shape representation, such that the body parts are perceived as wider than they",
            "are. While the origin of such anisotropy remains unknown, it has been suggested that visual",
            "experience could partially play a role in its manifestation. To causally test the role of",
            "visual experience on body shape representation, we investigated tactile distance perception in sighted and early",
            "blind individuals comparing medial-lateral and proximal-distal tactile distances of stimuli presented on the ventral and",
            "dorsal part of the forearm, wrist, and hand. Overestimation of distances in the medial-lateral over",
            "proximal-distal body axes were found in both sighted and blind people, but the magnitude of",
            "the anisotropy was significantly reduced in the forearms of blind people. We conclude that tactile",
            "distance perception is mediated by similar mechanisms in both sighted and blind people, but that",
            "visual experience can modulate the tactile distance anisotropy."
        ],
        "date_published": "2024-04-20T16:07:35.153014",
        "license": "563c1cf88c5e4a3877f9e96c",
        "disciplines": [
            "Psychiatry"
        ],
        "tags": [
            "blindness",
            "tactile distance anisotropy",
            "touch",
            "vision"
        ]
    },

• A CSV file containing the preprint summaries, saved in the data/csv directory:

• A CSV file containing the discipline-wise summaries, saved in the output/digests directory (the summaries generated by the BART model):

This project is licensed under the MIT License. See the LICENSE file for more information.

Contributions are welcome! If you find any issues or have suggestions for improvements, please open an issue or submit a pull request.

This tool utilizes the Open Science Framework (OSF) API for retrieving preprints. The preprint summaries are generated using the BART-large-CNN model from Hugging Face.

If you have any questions or need further assistance, please contact me at [email protected] or open a thread!