H2O assign bug with multi-clusters about h2o-3 HOT 4 CLOSED

Wendy Wong commented: [https://support.h2o.ai/a/tickets/102968|https://support.h2o.ai/a/tickets/102968|smart-link] -> More details of investigations and logs

When running

{quote}self.raw_frame = h2o.assign(self.raw_frame, 'raw_frame'){quote}

on a H2O Cluster with 6 nodes of 8gb for a dataset of 3.1gb a connection error occurs:

h2o.exceptions.H2OConnectionError: Unexpected HTTP error: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

Right before the assign, the data is loaded and sorted with the following code:

{quote}temp_raw = h2o.import_file(path=os.path.join(self.config['input_dir'], self.config['data_frame_file']),{quote}

{quote}destination_frame="temp_raw", col_names=col_names, col_types=dict_types_import,{quote}

{quote}skipped_columns=skipped_indices, na_strings=["Infinity", "-Infinity", "NaN"]){quote}

{quote}[http://logging.info|http://logging.info|smart-link]("ID of temp_raw " + temp_raw.frame_id){quote}

{quote}http://logging.info|http://logging.info|smart-link{quote}

{quote}self.raw_frame = temp_raw.sort([self.config['col_name_id']]){quote}

{quote}[http://logging.info|http://logging.info|smart-link]("ID of temp_raw " + temp_raw.frame_id){quote}

{quote}http://logging.info|http://logging.info|smart-link{quote}

When removing the sort, the assign succeeds. When using the exact same code (so with the sort) on a data set of 1.6gb, the assign also succeeds. And when running the sort+assign on a single node cluster of 48gb, the code also succeeds.

When replacing the assign by another function or property on the self.raw_frame, e.g. nrows, frame_id these calls also fail.

In annex the output of the script we are running. We added some [http://logging.info|http://logging.info|smart-link] statements regarding the H2O frames present in the H2O backend, which you can see with statements as:[2022-09-15 12:57:25] [INFO ] – ['cols_frame', 'temp_raw']

from h2o-3.

Wendy Wong commented: Yuliia has generated the dataset that will reproduce the error. It is located in s3://h2o-public-test-data/bigdata/laptop/sort_merge_tests/pubdev_8866_mem_leak_data.csv

from h2o-3.

Wendy Wong commented: This one needs to be done by Q2 of 2023.

from h2o-3.

JIRA Issue Details

Jira Issue: PUBDEV-8937
Assignee: Wendy Wong
Reporter: Wendy Wong
State: In Progress
Fix Version: N/A
Attachments: N/A
Development PRs: N/A

from h2o-3.