Memory pressure and perforamnce on string heavy parquet about parquetsharp HOT 2 CLOSED

PR #119 shoud alleviate this problem.

It tracks the presence of dictionary encoding and assumes it's a good proxy for choosing whether to cache duplicated ByteArray entries or not. This is currently only enabled for string as they are immutable. But the same logic can easily be applied to byte[] (probably behind an optional flag, as they are not immutable and could confuse the user if he modifies these arrays).

Local testing has shown that if dictinary encoding is disabled, Parquet C++ returns different ByteArray entries even if the string is identical. But that with dictinary encoding enabled, the entries are properly merged (which makes sense). Testing equality of ByteArray is quite cheap (a pointer and length), so I expect this to be a winner in most scenarios.

from parquetsharp.

Released in 2.2.0-beta1.

from parquetsharp.