Merge is slower than expected and loads more than expected into memory. about delta-rs HOT 5 OPEN

It needs to scan the entire table if you don't use partitioning, if you do partition then you need to give an explicit partition predicate to reduce the amount of partitions you read

from delta-rs.

Thanks for the reply @ion-elgreco , why does it need to scan the entire table into memory before it starts writing data? Is this just a lack of optimization, or is there something fundamental to what merge is doing that prevents this kind of optimization?

from delta-rs.

Thanks for the reply @ion-elgreco , why does it need to scan the entire table into memory before it starts writing data? Is this just a lack of optimization, or is there something fundamental to what merge is doing that prevents this kind of optimization?

It needs to scan the entire table because it needs to find out which rows the merge into condition applies to.

from delta-rs.

It needs to scan the entire table because it needs to find out which rows the merge into condition applies to.

I understand this. However, it doesn't need to hold the entire table in memory while it is performing the merge. It could do this in a streaming fashion – this is more or less what you get out of the box with datafusion.

from delta-rs.

To answer my own questions here:

why does it need to scan the entire table into memory before it starts writing data

MergeBarrier holds on to all records for a particular file until either a delete, update, or insert is encountered, or until the input data is fully exhausted. This means that in workloads with a large input set, where merge does not typically have deletes, updates, and inserts, the entire dataset will usually be buffered in memory.

This could be avoided if we somehow explicitly told DataFusion to fully exhaust one file at a time so that data could be flushed.

I can imagine using partitioning to break the merge into many operations so that all the data is not pulled into memory at once. But in my case, I'd probably rather put the effort towards not using Merge.

from delta-rs.