bookops-cat / overload Goto Github PK

User should be able to specify if a note that holdings have not been updated should be added to initials field:
NYPL: 901 $aW2Sbot$bCATBL$hOCLC holdings not updated
BPL: 947 $aW2Sbot$hOCLC holdings not updated

Another option would be to add this note automatically if user selects not to update holdings checkbox, or, by default check a box for the note when holdings are not being updated - this way it can always be unchecked and skipped if a particular workflow demands that.

add a custom logging adapter that will allow to pass to Loggly a properly formatted traceback (traceback in its own field)

If no 300 tag in retrieved from Worldcat record, W2S crashes.

new location code for branches: cn
investigate matching mechanism for this location

currently the text in help windows splits words between lines
fix it in overload_help.py

see GHSA-7fcj-pq9j-wh2r

the vulnerability does not affect Overload, but good to update anyway

create detailed scenarios for Midwest bibs with desired outcome
bpl situation may be even more difficult to untangle
skip these bibs from preprocessing initially

When modifying existing template:

Exception in Tkinter callback
Traceback (most recent call last):
File "C:\Program Files\Python27\lib\lib-tk\Tkinter.py", line 1547, in call
return self.func(*args)
...
File "C:\Program Files\Python27\lib\lib-tk\Tkinter.py", line 2634, in get
return self.tk.call(self._w, 'get', first)
TclError: bad listbox index "": must be active, anchor, end, @x,y, or a number

Displaying title from Sierra and from retrieved record from Worldcat together will allow quick identification if correct record has been found.

use keyring to store credentials in Windows credentials manager

enhancement to generate PVR CSV report separately from creating mrc file so as to separate out books before saving file

as in subject

BPL leased non-fic with vendor call # None and inhouse call # None not displayed

It would be helpful to include at least source of data for W2S process in the initials tag of the output MARC records. This could help in investigating what went wrong during the processing - which workflow was used : order or ISBN only, etc.

remove the 04 standard division from the call number

Does call number for WL graphic novel includes format prefix "GRAPHIC"?

bur51 - Business Center
any other new codes for NYPL?

A while back Google changed access token response for their authorization service. The response lists refresh_token param twice making it too long for Windows Credentials Manager to store.
The best would be to move away from oauth2client to a newer, Python 3 library such as google-auth or oauthlib. That would require though refactoring Overload to Python 3 - a large undertaking, especially considering its reliance on PyZ3950 which is Py2 only.

In the meantime, the easiest way to deal with access token response is to delete one of the refresh_token params from credentials object in the oauth2client.contrib.keyring_storage in the locked_put method.

def locked_put(self, credentials):
    """Write Credentials to file.

    Args:
        credentials: Credentials, the credentials to store.
    """

    # google updated their API auth server response, which duplicates
    # the refresh_token param, which makes credential string too long
    # for Windows credential manager
    # this is a walkaround the problem
    del credentials.token_response["refresh_token"]

    keyring.set_password(self._service_name, self._user_name,
                         credentials.to_json())

It seems NYPL duplicate report includes incorrect data. Looks like the cause is how the report is compiled and not problem with matching, Platform, or results analysis. Still, rule out of other possibilities.

Display W2S_MULTI_ORD and W2S_SKIPPED_ORD. At the moment they are recorded in temp folder, but not shown to a user.

Update Overload to add appropriate 910 tag to each NYPL record processed based on the order fixed code.
910 $a RL
910 $a BL

verify all fields are applied correctly

see here:
{ "app": "overload", "asciTime": "2019-03-19 14:03:43,855", "fileName": "pvf_gui.py", "lineNo": "2374", "levelName": "ERROR", "message": "Unhandled error: Traceback (most recent call last):\n File \u0027v0.5.3/pvf/pvf_gui.py\u0027, line 2350, in process\n File \u0027v0.5.3/pvf/manager.py\u0027, line 234, in run_processing\n File \u0027v0.5.3/bibs/bibs.py\u0027, line 712, in repr\nUnicodeEncodeError: \u0027ascii\u0027 codec can\u0027t encode character u\u0027/xb0\u0027 in position 15: ordinal not in range(128)\n" }

Could use tabs to allow user evaluation and selection of all retrieved from Worldcat matches. Tabs could include an internal metrics showing ranking of quality of each record. Such ranking could ignore level coding in the fixed fields and focus instead on a presence of other elements (number of subject headings, classification, sources of cataloging, etc.)

Consider improvement to call number creation when MM as first location (example mm2al).

FTP:
{ "app": "overload", "asciTime": "2019-11-26 10:45:42,612", "fileName": "ftp_manager.py", "lineNo": "263", "levelName": "ERROR", "message": "Download from FTP failed: host\u003dftp.ybp.com, file on remote\u003d19901466.mrc, destination path\u003dE:/migration-invoices/19901466.mrc, transfer type\u003dbinary. Error: [Errno 2] No such file or directory: u\u0027E:/migration-invoices//19901466.mrc\u0027" } | { "app": "overload", "asciTime": "2019-11-26 10:45:42,612", "fileName": "ftp_manager.py", "lineNo": "263", "levelName": "ERROR", "message": "Download from FTP failed: host\u003dftp.ybp.com, file on remote\u003d19901466.mrc, destination path\u003dE:/migration-invoices/19901466.mrc, transfer type\u003dbinary. Error: [Errno 2] No such file or directory: u\u0027E:/migration-invoices//19901466.mrc\u0027" }

When trying to change the dst directory:
{ "app": "overload", "asciTime": "2019-11-26 10:45:42,612", "fileName": "ftp_manager.py", "lineNo": "263", "levelName": "ERROR", "message": "Download from FTP failed: host\u003dftp.ybp.com, file on remote\u003d19901466.mrc, destination path\u003dE:/migration-invoices/19901466.mrc, transfer type\u003dbinary. Error: [Errno 2] No such file or directory: u\u0027E:/migration-invoices//19901466.mrc\u0027" }

PVF:
overload-2019-11-27 07:53:57,078-pvf_gui.py-2395-ERROR-40-Unhandled error: Traceback (most recent call last):\n File 'C:/Users/[.,.]/Documents/GitHub/overload/overload/pvf/pvf_gui.py', line 2371, in process\n self.progbar, self.current_process)\n File 'C:/Users/[...]/Documents/GitHub/overload/overload/pvf/manager.py', line 513, in run_processing\n write_marc21(fh_new, bib)\n File 'C:/Users/[...]/Documents/GitHub/overload/overload/bibs/bibs.py', line 27, in write_marc21\n writer.close()\nUnboundLocalError: local variable 'writer' referenced before assignment\n

bookops-worldcat is not compatible with python 2, but it can be adapted for overload project.

This is only a temporary solution. The goal, in near future, should be to update entire code of the overload to python 3, but this requires careful planning. The biggest obstacle is overload's reliance on PyZ3950 module, available only in py2 and not actively maintained (no chance for py3 update).

Prevent this
=082 04$a158.1 A
=049 \$aNYPP
=091 \$a158.1 A$cA
=100 1$aAhlquist, Diane,$eauthor.

ISBN parser in GetBib should ignore empty lines

https://google-auth.readthedocs.io/en/latest/oauth2client-deprecation.html
to
https://google-auth.readthedocs.io/en/latest/

Develop a workflow for NYPL RL materials where source of data are ISBNs alone (no order records in Sierra). Use provided default coding for ReCap.

for WC 2 Sierra, where there is no order record, there should be an option to enter the information to insert into items for the whole batch
i.e.
loc=input("Enter LOC code:")
itype=input("Enter Item type:")

Add automatically a "Processed by" internal note to every order.
Internal note in template should create second internal note.

as for NYPL

remove key from the request url when logging

Normalize Quotation marks to accommodate Sierra export of ISBN's

Change vednor to vendor
line 371
def populate_sierra_data(self, widget, data):
l1 = " {}\n".format(data["title"])
l2 = " bib #: {}, ord #: {} Vednor note: {}\n".format(
data["sierraId"], data["oid"], data["venNote"],
)

Remove terms from thesauri unsupported in our catalog (foreign language, etc.) from bibs being accepted.

Unmapped locations terminate processing - instead they should be skipped and rejected from output file.

Improve Platform/Z3950 logging data to ease debugging and troubleshooting. In particular, provide response with endpoint/keyword in one entry for easy searching. At the moment request entry includes endpoint/keyword, but response entry lacks this data making it cumbersome to view events when they are mixed with other library app loggers.

example
ʻAnānī, Khalīl

• solution: tie agency to load table template in Sierra, add command line to specify the template

respect local call number policies

Enhancement
add option to manually insert call number in w2s record report

Worldcat queries should include all available identifiers. Currently if the source of data is Sierra export, W2S will attempt to extract OCLC #, LC #, ISBN, etc. and query Worldcat in this order, stopping if a hit was found. Instead, W2S should keep searching, retrieve results for each query, dedup results, order them, and present to user for selection under separate tab. Each records should have info how it was retrieved, i.e. by OCLC #, ISBN, etc.

bug in quarter assignment

nypl_calnum.py
if order_data
lines 86-99
elif order_data.callLabel == "bil":
subfields.extend(["f", "BILINGUAL"])

NYPL does not use BILINGUAL as a classmark

Change the color of the 082 to darkgoldenrod from chocolate so it it stands out better from call number in tomato

Line 452 in gui_report
)
widget.tag_config(
"highlight_brown",
font=("tahoma", "10", "bold"),
foreground="darkgoldenrod",
)

validators should log types of validation errors and data that fails
this will provide information to improve the validation (weeding out false positives)

respect local call numbers