library-data-platform / ldlite Goto Github PK

Python 3.10.6

In querying /item-storage/items, I am receiving this error output while using LDLite to get the data from my institutions Folio tenent (see attachment). I am using the sample script from the ldlite git hub to get the data and create the tables: https://github.com/library-data-platform/ldlite/blob/main/examples/folio_demo.py.

I haven't been able to get the item records or anything else after that point in the script. I have tried breaking up the queries and table creation parts into multiple cell in my Jupyter notebook and have managed to get some of the tables created but it stalls out on items (see attachment).

item-storage_items_errorOutput.txt

Test_LDLite2_script.pdf

I was trying to run a copy of a notebook that worked fine previously (I think back in September 2022). The current version of the notebook is on Google Colaboratory, and it installs the latest LDLite (currently 0.0.35) with !pip install --upgrade ldlite.

It all starts okay. Connecting to FOLIO Snapshot worked fine, and I successfully completed several queries and selections. But when I try to set up a db cursor so I can follow the example for executing a SQL query (cur = db.cursor()), I get the following error:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
[<ipython-input-63-9a71221bf8ff>](https://localhost:8080/#) in <module>
----> 1 cur = db.cursor()

AttributeError: 'NoneType' object has no attribute 'cursor'

If instead I change the database connection to db = ld.experimental_connect_db_sqlite(filename='ldlite-sqlite2.db'), I get an error when I try to query the LDLite instance. That is, this command

_ = ld.query(table='users', path='/users', query='cql.allRecords=1 sortby id')

yields

ldlite: querying: /users

reading           

scanning           

transforming           
---------------------------------------------------------------------------
OperationalError                          Traceback (most recent call last)
[/usr/local/lib/python3.8/dist-packages/ldlite/_jsonx.py](https://localhost:8080/#) in _transform_json(db, dbtype, table, total, quiet, max_depth)
    416                 table_j = table + '__t' if i == 0 else table + '__t' + str(i + 1)
--> 417                 _transform_data(dbtype, '', cur2, [(1, table_j)], jdict, newattrs, 1, row_ids, max_depth, {})
    418             if not quiet:

7 frames
OperationalError: no such column: False

During handling of the above exception, another exception occurred:

RuntimeError                              Traceback (most recent call last)
[/usr/local/lib/python3.8/dist-packages/ldlite/_jsonx.py](https://localhost:8080/#) in _transform_json(db, dbtype, table, total, quiet, max_depth)
    422             pbar.close()
    423     except (RuntimeError, psycopg2.Error, sqlite3.OperationalError, duckdb.CatalogException) as e:
--> 424         raise RuntimeError('running JSON transform: ' + str(e))
    425     finally:
    426         cur.close()

RuntimeError: running JSON transform: no such column: False

In trying to build an automated harvesting script for our production tenant, I've repeatedly encountered an issue where a query (particularly to the inventory-storage endpoints) will hang during the "reading" phase and require a keyboard interrupt to exit. Perhaps we could add a default timeout or allow specifying a requests timeout (and number of retries) as an argument to the query method?

Reporting for a colleague at Five Colleges:

When upgrading LDLite, he was unable to do transformations. He fixed this with a downgrade to .31 and python 3.83.

"I'm running a windows system, with python 3.88. LDlite is running in a virtual environment. I did the standard pip -m .... to update.
didn't work so i created a new venv, also didn't work but that one was python 3.10. Tried it in pipev too. Unfortunately I've since reinstalled with .31 in a python 3.83 venv, which is now working fine, i can't replicate it"

Error Message:
seen this ldplite error before?

RuntimeError Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_42136/1165802898.py in
----> 1 ld.query(table='p_users', path='/users', query='cql.allRecords=1 sortby id')
c:\Users\aneslin\Documents\python\ldplite\ldp\lib\site-packages\ldlite_init_.py in query(self, table, path, query, json_depth, limit, transform)
399 newattrs = {}
400 if json_depth > 0:
--> 401 jsontables, jsonattrs = _transform_json(self.db, self.dbtype, table, count, self._quiet, json_depth)
402 newtables += jsontables
403 newattrs = jsonattrs
c:\Users\aneslin\Documents\python\ldplite\ldp\lib\site-packages\ldlite_jsonx.py in _transform_json(db, dbtype, table, total, quiet, max_depth)
393 cur2 = db.cursor()
394 while True:
--> 395 row = cur.fetchone()
396 if row is None:
397 break
RuntimeError: Invalid Input Error: Attempting to execute an unsuccessful or closed pending query result
Error: