hornc / plotmarc Goto Github PK

Visually characterise a bibliographic collection in terms of publication dates and available bibliographic identifier coverage

License: GNU General Public License v3.0

Python 100.00%

marc21 bibliographic-data venn-diagram

plotmarc's Introduction

plotMARC

A command line tool to visually characterise a bibliographic collection in terms of publication dates and available bibliographic identifier coverage.

Basic usage

From a directory containing one or more binary MARC21 format for bibliographic data files (with extension .mrc) representing a bibliographic collection, run:

./plotMARC.py

The script will process the MARC files using pymarc, and produce a single <directoryname>.png image containing a 3-way Venn diagram displaying the number of records with the following bibliographic identifiers:

ISBN
OCN (OCLC number)
LCCN (Library of Congress number)

and a histogram showing the publication dates in the bib records.

Example output

Summary for Test Collection:
Record counts for bibliographic identifiers present in this collection:
Total:  248632	100.00%
ISBN:   175593	 70.62%
LCCN:   130780	 52.60%
OCN :    82801	 33.30%
No Id:   42263	 17.00%

Requirements

Install these using pip:

 pip install -r requirements.txt

License

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

plotmarc's People

Contributors

Stargazers

Watchers

plotmarc's Issues

Revisit date bar chart

Does

https://stackoverflow.com/questions/58183804/matplotlib-histogram-with-equal-bars-width

give any good suggestions for improving the publication date chat?

(moving from a code comment) Date chart seems ok for now.

Improve MARC publication date extraction

Currently the code only looks at MARC 260$c Publication, Distribution, etc. (Imprint) for publication date.

Use this as the preferred source of publication date, but if not present, check for and use 264$c Production, Publication, Distribution, Manufacture, and Copyright Notice, if it exists.

Normalising / parsing OCLC numbers

https://knowledge.exlibrisgroup.com/Voyager/Knowledge_Articles/Basic_information_on_the_OCLC_control_number

Normalising LCCNs

https://www.loc.gov/marc/lccn-namespace.html#:~:text=A%20normalized%20LCCN%20is%20a,first%20character%20must%20be%20alphabetic.

Display a No_ID region on the 3Venn plot

Display a non-overlapping, to-scale, region next to the Venn diagram to indicate how many records had none of the three bibids.

add an option to suppress pymarc warnings

Suggestion:
add a -q option to hide warnings from pymarc Reader.

Seeing the warnings once is sometimes helpful, but once you are satisfied that the data is good enough, the constant stream of warnings is distracting....

Provide a summary of material types within a collection

Types of interest:

Monographs
Serials
Maps
Sheet Music
Other

(possibly AV music / video?)

To get this breakdown, it is likely that more than one field/subfield will need to be checked. For the purposes of stats calculation, it makes sense to place each record into only one of the above categories.

MVP: text only summary output. Think about how to graphically display this (if at all) later -- depending on how useful it is.

accept (an optional) filename from CLI to process one specific MARC file

default to *.mrc if no file specified

Purpose:

Large multi-file MARC collections may have different date ranges or materials located in different files. It will often be helpful to know which files have the most recent material etc

Extension: for mutliple file collections; process each file individually, and also combine the results in a way that only processes each file once.

Text based output format for bibid categories and date counts

Provide a text based output for exporting data.

Current code outputs:

Dates: [0, 1400, 1760, 1770, 1780, 1790, 1800, 1810, 1820, 1830, 1840, 1850, 1860, 1870, 1880, 1890, 1900, 1910, 1920, 1930, 1940, 1950, 1960, 1970, 1980, 1990, 2000, 2010] [34636, 0, 1, 1, 2, 5, 18, 22, 95, 146, 188, 325, 341, 462, 596, 828, 875, 956, 1294, 1666, 2456, 5346, 12925, 30085, 45280, 76277, 33109, 697]
No dates: 34636
No IDs: 294
Venn Categories: [294, 893, 351, 3362, 55381, 61284, 17013, 110054]

Needs something more usable.

TSV for dates, for copy-paste into a spreadsheet?
Venn categories; a format which can be passed directly back into matplotlib_venn or other Venn diagramming tools?

Improvements

OCLCn -> OCN
label y-axis: 'Records'
chart title: Publication Dates
chart title: Record BibIds
Overall title - name of partner (directory - title cased)
~~Display No-ID portion~~ (Moved to separate issue: #3)
EARLY to be clearer 1400-1700 >1400

Filter 035 fields for OCLC ids

Currently the code is treating any MARC 035 field as an OCLC number, which is inaccurate.

Filter out 035s which do not contain OCLC numbers.

See
https://github.com/internetarchive/openlibrary/blob/39962157c12465d050e43458444aba9a59661c8d/openlibrary/catalog/marc/parse.py Line 125 ~ for some regex ideas.