This repo holds code and data for the publication:

Tal August, Lauren Kim, Katharina Reinecke, and Noah Smith ''Writing Strategies for Science Communication: Data and Computational Analysis'', Conference on Empirical Methods in Natural Language Processing (EMNLP) 2020.

Begin by creating a new virtual environment and installing required packages (here using conda)

conda env create -f environment.yml

And activate it

conda activate sci_articles

The urls of all scraped articles are in data/cleaned_article_urls.csv and all annotations made on these articles are in data/annotations.csv

The code for the scraper (including the parameters used to run the scrapy spider) are in the scraper/ directory

If you want the full text for each article, you can run the scraper with the parameters of the original paper. In the scraper/ directory, run python runSpiders.py all <FILEPATH>.jsonl for wherever you want to store the files.

Note that this might take a few days to scrape all the data.

We use the RoBERTa models from the Huggingface transformers library, pretraining and finetuning details are in the paper. Many of the scripts we use are adapted from the Huggingface example scripts, please refer to these for more details, specifically the scripts in examples/text-classification/.

If you are interested in getting access to the final finetuned models for classifying the writing strategies, feel free to reach out!

For the annotation interface, see the repo: https://github.com/talaugust/scientific_article_annotation/.