paperfetcher / paperfetcher Goto Github PK

View Code? Open in Web Editor NEW

9.0 2.0 2.0 6.05 MB

Pip-installable Python package to automate handsearching and citation searching for systematic reviews.

Home Page: https://paperfetcher.github.io

License: MIT License

Python 97.81% Makefile 0.73% Shell 0.54% Batchfile 0.92%

automation meta-analysis systematic-review data-retrieval

paperfetcher's Introduction

paperfetcher

About

Paperfetcher is a Python package to automate handsearching and citation searching (snowballing) for systematic reviews. Paperfetcher works with Python 3.7+.

To learn more about Paperfetcher, visit paperfetcher.github.io.

Usage

To get started, browse the following Getting Started Jupyter notebooks (also in the examples/ directory in this repository):

Once you understand the workflow, read the Module Documentation to learn more about paperfetcher's various modules, classes and functions.

Installation

Installation with pip

Install pip
Install paperfetcher using pip:

pip install paperfetcher

Installation from source

Clone this repository
In the repository directory, run

python setup.py install

or, if you have pip installed

pip install .

Running tests (if installed from source)

You can run tests to make sure that paperfetcher is working correctly on your system.

To run integration tests, in the repository directory, run

cd tests/tests_integration
pytest

To run unit tests, in the repository directory, run

cd tests/tests_unit
pytest

The Team

Paperfetcher was developed by Akash Pallath at the University of Pennsylvania and Qiyang Zhang at the Johns Hopkins University.

paperfetcher's People

Contributors

Stargazers

Watchers

Forkers

apallath xantocx

paperfetcher's Issues

Update integration tests

annrev-conmatphys is now indexed => integration test fails.
Fix: update expected sum of results.

Convert CrossrefSearch methods to class methods.

Allow setting crossref API key dynamically

Notebook utils

Write notebook utils classes for:

handsearch -> DOIDataset -> choose format
handsearch -> CitationsDataset (choose fields) -> choose format
snowballsearch -> DOIDataset -> choose format

Allow searches with no keywords and no dates

These parameters should be optional => handle None types properly

Error in parsing returned results

File "/Users/apallath/opt/miniconda3/lib/python3.7/site-packages/paperfetcher/handsearch.py", line 248, in __call__
    self.results += (batch['items'])
TypeError: list indices must be integers or slices, not str

Occurs infrequently in large handsearches. TODO: generate a minimal reproducible example.

Snowball search refactoring/bug fixes

Distinguishing between forward citation results between multiple DOIs

In the situation where I run multiple DOIs on a forward citation search with a RIS output, there is no way to determine which citation results belong to which DOI. Could be solved by including the Initially search DOI in the RIS output data for each of the forward citations. Thanks!

Fwd/back COCI snowball

Examples

Examples for:

handsearch with no keywords (augment current example notebook)
snowballsearch
notebook utils

Corrections to README, docs, and examples notebooks.

Readme updates.
Examples MD updates.
Docs missing region updates.

Possible R version of paperfetcher

Hey - I'm working on evidence synthesis in R, and there's an important gap where paperfetcher sits in the R package ecosystem. I'm the organiser for the Evidence Synthesis Hackathon, and I'd love to suggest paperfetcher as a hackathon project to translate the great work you've done into an R version - would you be keen to collaborate? Thanks so much!

Store large searches on disk instead of in-memory.

Can either:
a) write to database

sqlite? This should work for Streamlit cloud as well: search data doesn't need to be persistent.

b) dump data periodically to temporary file?

Evaluate options and implement. This will require changes to the codebase.

Fix dependencies

Add new dependencies to requirements.txt

RIS dataset

Implement RIS dataset as a list of dictionaries.
Use CrossRef for content negotiation from DOI to RIS.
Option to add an extra field.
rispy for I/O.