Buffer manager exception: Failed to claim a frame: Python API: Bulk Load kuzu==0.0.12 v kuzu==0.2.0 about kuzu HOT 5 OPEN

hi @saschamcdonald thanks for sharing this. will take a look into it.

from kuzu.

Hi @saschamcdonald , thanks for reporting this! We've made significant changes to the rel storage since v0.1.0, and that should be why the behaviour is inconsistent between these two versions, though the exception is not expected. We'd love to look into the exception and fix it. Is it possible for you to share the parquet file with us?

from kuzu.

re share the parquet file with us?: Unfortunately I have a GDPR issue as the data could be personally identifiable. From review of the changes between versions of KuzuDB as referenced in the raised issue, it could be that ingress of a large parquet file ( relative to a client's available memory ) into kuzu v 0.2.0, is potentially not chunking ingress in terms of memory spill to disc as effectively as kuzu v 0.0.12 via it's default settings. I'll try and create test data over the next couple of weeks to offer a repeatable test data set for the team. I think memory to disc spill is potentially the issues. In the interim is there a debug level I can set to capture more info for the team? - brb.

from kuzu.

Hi @saschamcdonald , thanks for the info. Is it possible to share the rel table schema and some statistical information of the dataset, like number of nodes and number of rels and also some degree distribution info? We can try to reproduce this locally on our side.

I'll try and create test data over the next couple of weeks to offer a repeatable test data set for the team.

That would be much appreciated!

In the interim is there a debug level I can set to capture more info for the team?

Unfortunately, we currently don't have a way to collect more debugging info without compiling from source and running inside a debugger.

from kuzu.

@ray6080 Here is a small repo containing my code that generates test data and offers environment repeatability for the issue and hopefully useful for the team:
https://github.com/saschamcdonald/ch_06_kuzudb_tests

from kuzu.