Some subject_people have the form { type: "/type/text", value: String } instead of just plain strings.

entities-affected: 403
examples:
  - url: https://openlibrary.org/works/OL55497W.json
mongo-queries:
  - name: Affected works
    query: """db.works.find({ subject_people: {$type: "object"} }, {key: 1, subject_people: 1, _id: 0})"""
dump: ol_dump_works_2017-11-30

• Generate list of work keys to update
• Fix affected works
• Create issue for analyzing/removing references in internetarchive/openlibrary

Example:

ol.Work.get('OL3087688W').editions

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Python/2.7/site-packages/olclient/openlibrary.py", line 124, in editions
    for ed in editions]
  File "/Library/Python/2.7/site-packages/olclient/openlibrary.py", line 288, in _ol_edition_json_to_book_args
    for author in data.pop('authors', [])]
  File "/Library/Python/2.7/site-packages/olclient/openlibrary.py", line 445, in get
    bio=data.pop('bio', {}).get('value', u''),
AttributeError: 'unicode' object has no attribute 'get'

The page in question is: https://openlibrary.org/works/OL3087688W/editions.json
1 edition with author field:
"/authors/OL482817A"

Some authors have bios of format "bio": "text" while others have, and the client expects, the format:

"bio": {"type": "/type/text", "value": "Bio text"}

Not sure why there is a variation in the data, there seem to be lots of both styles.

The merges from 2013 need to be undone. They've conflated a novelist with a statistician and affected thousands of works.

https://openlibrary.org/authors/OL24370A/Kathleen_Morgan

@LeadSongDog writes:
The statistician is https://viaf.org/viaf/73431623
The pulp romance author is https://viaf.org/viaf/9176228

This issue originally opened here by @tfmorris:
internetarchive/openlibrary#712

Open Library recently switched from using Open Library username + password -> Open Library email + password -> Internet Archive email + password.

However, the openlibrary-client currently still uses Open Library username + password for login (there shouldn't still be a way to login or change OL account credentials).

https://github.com/internetarchive/openlibrary-client/blob/master/olclient/openlibrary.py#L60
/account/login endpoint has to be modified on openlibrary to return a valid cookie when IA creds provided.

When Edition.get has no results, it returns an 'empty' edition with an error field on it. This edition can be (and was) saved, and is now on OpenLibrary: https://openlibrary.org/books/None

Input:
ol.Edition.get(isbn='foobar')
Output:

<class 'olclient.openlibrary.OpenLibrary.Edition.<locals>.Edition'
{'_work': None,
'work_olid': None,
'olid': 'None',
'description': None,
'notes': None,
'identifiers': {'wikidata': ['Q2071569']},
'title': None,
'subtitle': None,
'pages': None,
'authors': [],
'publisher': None,
'publish_date': None,
'cover': None,
'created': {'type': '/type/datetime', 'value': '2018-02-25T22:48:09.562592'},
'last_modified': {'type': '/type/datetime', 'value': '2018-02-25T22:48:09.562592'},
'latest_revision': 1,
'error': 'notfound',
'type': {'key': '/type/edition'},
'revision': 1}>

https://openlibrary.org/people/AliceKirk
I came across this during the author merge cleanup because Alice created dozens of duplicate author records for the same person.

Originally I thought this and the repetition in books e.g.
https://openlibrary.org/search?q=title%3A+%22Innovative+Design+Guidebook+for+Game+Changers%22&mode=everything

were just innocent mistakes, but after looking a little further I realized all the entries are just designed to turn OpenLibrary into a giant link farm for BookBoon.

In some cases, records were actively vandalized, like these two where works were retitled and reauthored:

• https://openlibrary.org/works/OL2994057W/Long-Term_Assets?b=5&a=2&_compare=Compare&m=diff
• https://openlibrary.org/works/OL6830628W/Careering_Ahead?b=4&a=2&_compare=Compare&m=diff

Originally posted by @tfmorris: internetarchive/openlibrary#711

To work with the new save_many(), Authors need a save() method.

ol --configure shouldn't require --email. It should just ask (like it does for password). Also, the config file should save the email.

Should be a 3 line fix -- in cli.py and config.py

30 in works

grep -c "Thing.url" ol_dump_works_2017-09-30.txt 
30

example: /works/OL16268456W

A continuous integration server such as Travis CI would be a very useful bit of infrastructure (and free).

entities-affected: 1136
examples:
  - url: https://openlibrary.org/works/OL67142W.json?v=2
mongo-queries:
  - name: Affected works
    query: """db.getCollection('works').find({ works: {$exists: true} })"""
dump: ol_dump_works_2017-98-31

Based on the above example, it looks like these were introduced by WorkBot ~2009. In the above, example, the revision is given the comment "add subject: Piano", so these should be added to subjects.
https://openlibrary.org/works/OL67142W?m=history

Looked through the main codebase, and can find no reference of the works property on a Work, so they should be safe to correct.

• Generate list of work keys to change
• Add all works to subjects

Proposal:

Default User Agent string:

olclient/v.v.v (<operator contact, defaults to IA login email?>)

e.g.
User-Agent: olclient/0.0.17 ([email protected])

Extension; Provide the ability to use a specific named bot:
namedBot/v.v olclient/v.v.v (<contact address/url>)

Example that returns a JSON object that can then be passed to one of the save methods:

    def get_redirect(self, from_olid, to_olid):
        ''' from OLID, to OLID
        '''
        assert olid.get_type(from_olid) == olid.get_type(to_olid)
        data = {
            'key': olid.full_key(from_olid),
            'location': olid.full_key(to_olid),
            'type': { 'key': '/type/redirect' }
        }
        return data

@mekarpeles , I've realised the main reason I've held off adding these sort of methods is because I wasn't sure how best to handle the marshalling between raw json (which I'm using because it is easy), and the existing OpenLibrary classes, which inherit from the common Entity classes in common.py .

Now that I've reminded myself, I'll think on it a bit more. Maybe we can get together and discuss the direction we want to take the client, and bring Robin in too so he gets some more context?

entities-affected: 10
examples:
  - url: https://openlibrary.org/works/OL167326W.json
mongo-queries:
  - name: Affected works
    query: """db.works.find({ subjects: {$type: "object"} })"""
dump: ol_dump_works_2017-11-30

• Generate list of work keys to update
• Fix affected works
• Create issue for analyzing/removing references in internetarchive/openlibrary to subject objects

It's a pretty common practice to follow different license strategies for server and client code with client libraries, example code, etc being permissively licensed to encourage their use in applications.

I'd like to suggest that this client library be switched to a BSD license.

Work's add_subject, add_subjects, and rm_subjects methods reorder the pre-existing subjects on modification by converting the subjects to a set:

Just a thought, but a diff function between entries (Works, Editions, Authors) that look like they should be the same would be helpful in deciding which one is better, or what fields need to be merged.

In general I imagine there are three outcomes:

1. One record is clearly better, and the other can be safely discarded (i.e less good record's attributes are a subset of the better record)
2. There is an obvious way to merge data cleanly (copying missing attributes from one to the other) to make a more complete record
3. There are differences/conflicts within the same attributes and no way to decide programatically what to do, so human editing is required ... or maybe they really are different entities?

First step is to perform the diff though.

After running pip install openlibrary-client I can't seem to import from olclient:

# python
Python 2.7.13 (default, Jan 19 2017, 14:48:08) 
[GCC 6.3.0 20170118] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from olclient.openlibrary import OpenLibrary
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jeff/.virtualenvs/env_aaaaarg/local/lib/python2.7/site-packages/olclient/__init__.py", line 17, in <module>
    from .openlibrary import OpenLibrary
  File "/home/jeff/.virtualenvs/env_aaaaarg/local/lib/python2.7/site-packages/olclient/openlibrary.py", line 13, in <module>
    import requests
  File "/home/jeff/.virtualenvs/env_aaaaarg/local/lib/python2.7/site-packages/requests/__init__.py", line 52, in <module>
    from .packages.urllib3.contrib import pyopenssl
  File "/home/jeff/.virtualenvs/env_aaaaarg/local/lib/python2.7/site-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 54, in <module>
    import OpenSSL.SSL
  File "/home/jeff/.virtualenvs/env_aaaaarg/local/lib/python2.7/site-packages/OpenSSL/__init__.py", line 8, in <module>
    from OpenSSL import rand, crypto, SSL
  File "/home/jeff/.virtualenvs/env_aaaaarg/local/lib/python2.7/site-packages/OpenSSL/SSL.py", line 112, in <module>
    SSL_ST_INIT = _lib.SSL_ST_INIT
AttributeError: 'module' object has no attribute 'SSL_ST_INIT'
>>> 

Even running "pip list" produces this same error. I'm guessing pyOpenSSL 16.1.0 isn't playing nice with libssl 1.1.0e? I am running Debian testing.

If I clone the repository, change the version to 16.2.0 in requirements.txt and run pip install ., then openlibrary-client works perfectly.

Can this dependency be upgraded without causing problems for other users?

2982 authors have a website properties. These should be moved to links

grep -c '"website":' /storage/openlibrary/ol_dump_authors_2017-09-30.txt 
2982

Implement ol.save_many([ list of OL objects], comment)

see https://github.com/hornc/catharbot/blob/3d909284c251c7948c97bcc952a371670c25d3d8/catharbot/catharbot.py#L57 for a raw json implementation.

It seems to me that an interface for further data analytics would enhance
our experience with Archive.org.
I can tackle this by using complex networks and text mining techniques,
because I have developed research on these topics.
Immediate ideas are:
*) Derive network by users interactions and vocabulary usage and deliver some graphical interfaces for exploring these structures.
*) Counting of words, tags, terms and user activity.

I understand it might be late for coping with GSoC, but if you find it suitable,
I might apply.
My apologies for not making this contact earlier, but I handed my doctorate
dissertation a few days ago and could not concentrate as needed until now.
Some info about my research and software development efforts are gathered here:
https://pastebin.com/iNNuN4fy7
Anyway, this topic might be of use for the Archive.org community as a whole and for
developments outside GSoC, but I did not find where to post this.

Best Regards!
Renato Fabbri

Edits such as https://openlibrary.org/books/OL7251755M/Charlotte_Temple?b=5&a=4&_compare=Compare&m=diff
should be low-hanging fruit for data cleanup.

It would be great if openlibrary-client could easily be extended to import new book metadata records from 3rd party partners (like BetterWorldBooks APIs, Onix Feeds, MARCs)

A good first step would be a small utility which can take an ISBN and lookup/pull information from other sources. This would be very helpful supplement for Openlibrary.org's "add a book" feature.

Currently not found authors and works returns an object:

>>> ol.get('OLnotA')
<class 'olclient.openlibrary.Author' {'bio': None, 'identifiers': {}, 'name': u'', 'olid': u'OLnotA', 'error': u'notfound'}>

>>> ol.get('OLnotW')
<class 'olclient.openlibrary.Work' {'_editions': [], 'key': u'/works/OLnotW', 'olid': 'OLnotW', 'error': u'notfound'}>

While a not found Edition returns None:

I suggest it should raise a "Not found" exception of some sort.

Similar to #29

Some works have

"authors": [{"type": "/type/author_role" ...
=> 8,280,196

vs

"authors": [{"type": {"key": "/type/author_role"} ..
=> 7,743,184

This means the current works schema fails validation on about half of all works :(

The UI appears to save the "type: key" format

Experiment, taking https://openlibrary.org/works/OL10002760W.json which has
"authors": [{"type": "/type/author_role", "author": {"key": "/authors/OL3968169A"}}]
and saving without manual changes via the UI... aftering saving
"authors": [{"type": {"key": "/type/author_role"}, "author": {"key": "/authors/OL3968169A"}}]

so type: key was added by the UI.

Actions

• Confirm whether "type: key" is the best way to represent work authors
• ensure the Work.schema is correct
• update remaining works to use the correct authors format

The Open Library client mentioned in the README is an angular module. This has two major disadvantages -

1. Developers can't use it with other frameworks like React or Vue. They can't even use it with Vanilla Js or jQuery.
2. No cli support. To use client on the command line, devs have to use the python client. For those developers using Nodejs as their backend, this can be a deal breaker.

It would be much better if a client is made on Node.js first which can provide the option to use the tool on the command line as well as in their Nodejs script.
I would love to make this client

Have we gone through the process of searching all authors with works and seeing which are likely duplicates?

Delete ~55K bad author records with names matching the regex grep -E " Cd[a-zA-Z0-9]{2,4} " and related music CDs (ie 58565 less the 3415 that @hornc already deleted)
cc: @tfmorris, @mikemaehr, @bencomp, @hornc

Moved from internetarchive/openlibrary#152 (comment)

Only 968 authors have this property, but there are many that are identified as organisations, entity_type: org this seems like a useful distinction. Shall we make more use of this property and come up with some guidelines?

grep -c '"entity_type":' /storage/openlibrary/ol_dump_authors_2017-09-30.txt 
968

Are the tests passing for everyone else?

Binary MARC isn't really my strong suit, but it looks like the delimiter character (\x1f or ^_) is encoded using its graphical representation ($) in the example binary .mrc files, which causes the test comparisons to fail when the conversion returns the actual binary delimiter.

The tests pass if I change the test data, but I'm a little leery of doing that until I understand what changed elsewhere that caused this difference. I'm testing on a Python 3 branch, so there's a small chance that something to do with Python 3 or its impact on pymarc is to blame.

edition = ol.get(u'OL17922113M')

if the server responds with a 503 (or presumably other error codes?)
edition is None without any error or notification that something has gone wrong.

~763K editions (out of 25M) have a description field, 97% of those have "description": {"type": "/type/text", "value": "The description"}, and just 3% have simply "description": "The description". The OL UI displays both, but SOMETIMES saves the simpler "description": "string" kind, and converts the format on any save (and the diffs DO NOT show up on the history view). There is a notes property that has the same issue.

Need to fix the issue in OL to avoid creating more!

The README says:

This process will ask for a username and password and save them in ~/.config/ol.ini (or whichever config location the user has specified). In the next version, the password will not be stored; instead the account will be authenticated and the username and resulting cookie (and not the password) will be stored in the config

This is a security risk, so I think it's time for "the next version."

The Dewey Decimal Class and LoC Classification is often mistakenly found at edition records when it properly belongs to the entire corresponding work. The same could also be said for Goodreads and LibraryThing identifiers.

For example see https://openlibrary.org/works/OL3907500W and its various editions such as https://openlibrary.org/books/OL4562383M.

Previously discussed at internetarchive/openlibrary#497

Addressing this could massively improve the ease of finding duplicate works, particularly when they are in different languages.

Editions currently have two properties to represent contributions:

"contributions": ["Alberto Gonzalez Troyano (Translator)"]

grep -c '"contributions":' /storage/openlibrary/ol_dump_editions_2017-09-30.txt 
7,662,302

vs.

"contributors": [{"role": "Foreword", "name": "Harry Vipond"}]

grep -c '"contributors":' /storage/openlibrary/ol_dump_editions_2017-09-30.txt 
34272

My opinion:
Contributors {role: , name:} is better / more structured than contributions, which may as well be a notes field.
But best of all is the author_role type found on works which provided an unambiguous link to the author record.

see #44 for an issue around author_roles

I'd like to propose using author_role on editions and converting the contributions / contributors properties, but that will be a big job and needs consensus and more thought on how to go about it.

Port over the EDIT functionality of https://github.com/internetarchive/openlibrary/blob/master/openlibrary/api.py (which allows one to edit the metadata of Editions, Works, etc) into the openlibrary-client

There are several thousand edition records tor "Books on Tape" and similar publishers needing edits such as this:
https://openlibrary.org/books/OL10601512M?b=7&a=6&_compare=Compare&m=diff

For each
-the "Part m of n" should move to the subtitle
-the type of book should change from audio cassette or audio disc to audiobook on cassette or audiobook on disc
-the linked work changes to the record (of those for the same author and the shortened title) which has the most editions

The assumption that things written for Python 2.7 will work on Python 3.x turns out to be untrue (and, philosophically, some would say that anything that's not covered by tests can be assumed to not work).

Last run ~2012. Seemed to be having problems:

NYT bestseller bot is adding, then immediately replacing, tags:

http://openlibrary.org/works/OL16010721W/Lost_in_Shangri-la?m=history

more: http://openlibrary.org/people/nyt_bestsellers_bot

see: internetarchive/openlibrary#54

A dumpbin is a Wholesale-to-Retail package for convenient handling of one, two, or three dozen copies in the bookstore. In several hundred cases from 2008 or 2009 we imported Amazon records (since 404ed) that treated dumpbins as books. These should all be made to vanish:
https://openlibrary.org/search?q=title%3A+%22dumpbin%22

Attached are lists of 101851 deleted author IDs and 270619 redirected author IDs from the 2018-03-31 dump.

The deleted authors should be removed from the index. The directed ones, we probably still want to be able to look up by ID, but we should make sure that they're not indexed as a target anywhere (perhaps a task for a separate ticket?).

deleted-author-ids.txt
redirect-author-ids.txt

Should allow loading/saving config file from custom path

See: internetarchive/openlibrary#896 / #73

Similar to internetarchive/openlibrary#152 (# many Compact Cassette catalog ids have been embedded in author record name fields.
Please run this oh-so-similar check to quantify the current size of the problem:
egrep -Ec " Cs[a-zA-Z0-9]{2,4} " ol_dump_authors_2018-02-28.txt

cc: @LeadSongDog

12142 authors have a comment property in their data. The examples I have seen have value = "initial import"

grep -c '"comment":' /storage/openlibrary/ol_dump_authors_2017-09-30.txt 
12142

Works and Editions have some properties in common, e.g. datetimes, links, authors. These should be moved to a single location. olclient/schemata/shared_definitions.schema.json

Relative $refs can be used, but the python jsonschema will need to have the resolver configured to know where to look for the relative paths, see python-jsonschema/jsonschema#98 (comment)

Searching Authors for '.com' gives 20K results, most are already deleted spam authors (starting at page 5):

https://openlibrary.org/search/authors?q=.com&mode=everything&page=5

I can manually get them removed from Solr using the admin interface and the script I was using to update the removed 'Cd' authors: https://gist.github.com/hornc/230682dece335b998cb1b30498517251

These were not being removed from Solr once deleted, unfortunately I am still seeing deleted authors remaining in Solr after deletion even after internetarchive/openlibrary#861

Merged authors converted into redirects are now being removed from the solr index though.

• Update Documentation to adding a new book to Open Library Client
• Check documentation of others and add relevant code examples to use the Open Library Client in the Wiki.

entities-affected: 7418
examples:
  - url: https://openlibrary.org/works/OL15558210W.json
mongo-queries:
  - name: Affected works
    query: """db.getCollection('works').find({ number_of_editions: {$exists: true} })"""
dump: ol_dump_works_2017-98-31

This property appears on some works, but does not appear to be in sync with the actual state of the work. Should be removed.

Looks like was originally introduced by EdwardBot in some edition merging code: https://github.com/internetarchive/openlibrary/search?utf8=%E2%9C%93&q=number_of_editions

Those files should (probably?) also be updated, although they do appear inactive.

• Generate list of work keys to update
• Remove all instances of number_of_editions property
• Create issues for removing all references in internetarchive/openlibrary to number_of_editions

Should be able to search by a solr query (i.e. title:foo OR title:bar OR isbn:*)

It's both kludgy and non-portable, so it'd be better if there were a Python solution (I'll research alternatives).