This repo holds code and data for the publication:
Tal August, Lauren Kim, Katharina Reinecke, and Noah Smith ''Writing Strategies for Science Communication: Data and Computational Analysis'', Conference on Empirical Methods in Natural Language Processing (EMNLP) 2020.
Begin by creating a new virtual environment and installing required packages (here using conda)
conda env create -f environment.yml
And activate it
conda activate sci_articles
The urls of all scraped articles are in data/cleaned_article_urls.csv
and all annotations made on these articles are in data/annotations.csv
The code for the scraper (including the parameters used to run the scrapy spider) are in the scraper/
directory
If you want the full text for each article, you can run the scraper with the parameters of the original paper. In the scraper/
directory, run python runSpiders.py all <FILEPATH>.jsonl
for wherever you want to store the files.
Note that this might take a few days to scrape all the data.
We use the RoBERTa models from the Huggingface transformers library, pretraining and finetuning details are in the paper. Many of the scripts we use are adapted from the Huggingface example scripts, please refer to these for more details, specifically the scripts in examples/text-classification/
.
If you are interested in getting access to the final finetuned models for classifying the writing strategies, feel free to reach out!
For the annotation interface, see the repo: https://github.com/talaugust/scientific_article_annotation/.