EPFL Spring 2018 Semester Project

Supervised by Matteo Romanello, Digital Humanities Laboratory, EPFL

Citation matching in the humanities publication is not an easy task. This project adopts a step-by-step approach to match references in the Linked Books dataset. Due to the large scale of data available, the search space is required to be reduced to efferently find the match.

• The approach starts with matching partial references with full references in one document based on some general rules.

• Secondly, with the help of hash functions, the full references are grouped together into a global candidate group.

• Finally, semi-supervised learning method is applied to address the problem of lacking ground truth and classify within the candidate group whether two citations point to the same document.

• Manually annotating the global candidate group is needed for the evaluation of this approach.

The reposotory is organized as following:

• material: presentation sildes and report

• code: main code and notebook

  • Local Clustering:
    • Local Clustering.ipynb
  • Hash Matching:
    • getfullref.py: get "full" references
    • hashfunction.py: definition of hash function
    • hashgenerate.py: generate hashes for all full references
    • hashsplit.py: change the structure of reference and its hashes
    • hashmatch.py: match references based on hashes
    • Hash Stat.ipynb: show some statistics of returned references of each hash
    • Hash Sample.ipynb: apply hash black list and select samples for annotating
  • Global Matching:
    • Global Matching.ipynb
  • Support Modules:
    • dbmodel.py: modules for database
    • readref.py: read and store reference into desired structure

• Colavizza, Giovanni, Matteo Romanello, and Frédéric Kaplan. "The references of references: a method to enrich humanities library catalogs with citation data." International Journal on Digital Libraries (2017): 1-11.
• Fedoryszak, Mateusz, et al. "Methodology for evaluating citation parsing and matching." Intelligent Tools for Building a Scientific Information Platform. Springer, Berlin, Heidelberg, 2013. 145-154.
• Fedoryszak, Mateusz, and Łukasz Bolikowski. "Efficient Blocking Method for a Large Scale Citation Matching." D-Lib Magazine 20, no. 11/12 (2014).
• Fedoryszak, Mateusz, Dominika Tkaczyk, and Łukasz Bolikowski. "Large scale citation matching using Apache Hadoop." arXiv preprint arXiv:1303.6906 (2013).