Write statistics about odbc2parquet HOT 14 CLOSED

This behavior is fixed in odbc2parquet 4.1.2 as it uses parquet 50.

from odbc2parquet.

I assumed the parquet-rs crate would do so by default. Seems I am wrong. What is a good way to test statistics have been written?

from odbc2parquet.

Note to self: https://docs.rs/parquet/latest/parquet/file/properties/struct.WriterProperties.html#method.statistics_enabled

Happy to change the default, still wonder how to test this, though.

from odbc2parquet.

Extracting statistics from a file written with odbc2parquet worked for me. To do this I used a piece of Python code kindly supplied by Uwe on Stack Overflow:

import pyarrow.parquet as pq

pq_file = pq.ParquetFile("tmp.par")
# Get metadata for the i-th RowGroup
rg_meta = pq_file.metadata.row_group(0)
# Get the "max" statistic for the k-th column
max_of_col = rg_meta.column(0).statistics.max
print(max_of_col)

Now, I wonder, what makes you think odbc2parquet does not write statistics? Are the statistics I saw calculated on demand?

from odbc2parquet.

Interestingly this seems to be the case with a single specific table only:

I can reproduce this by fetching a single column from a SQL Server database.

I'll try to create a reproducer for you next week.

from odbc2parquet.

Minimal reproducer on my SQL Server instance:

CREATE TABLE tmp (x CHAR(1))
INSERT INTO tmp (x) VALUES ('a')

Fetch with SELECT * FROM tmp.

from odbc2parquet.

I can reproduce this, but I wonder if this is an issue. It parquet-rs seems to generate statistics for e.g. integer columns, but not for text. Explicitly enabling writing statistics in the writer properties does not change it. So if this is an issue, I think I would need to send it upstream. I wonder however if I am missing another flag I could set or if this could even be considered sensible behavior for text columns.

from odbc2parquet.

Am I also right in assuming it is specifically about the min/max statistic?

from odbc2parquet.

Yes. This is important because it helps query engines skip row groups that are irrelevant.

Am I right to assume that this is the implementation used by this tool? I can open a ticket. https://github.com/apache/arrow-rs

from odbc2parquet.

Am I right to assume that this is the implementation used by this tool? I can open a ticket. https://github.com/apache/arrow-rs

Yes this is the correct repository to raise an issue. odbc2parquet uses the parquet crate. The repository in question also is home to the arrow crate (which you might have guessed).

In case it could be of help, here is the point in odbc2parquets code there the writer properties are instantiated:

https://github.com/pacman82/odbc2parquet/blob/8415a466da726ccbca6dadf7102238ab74d59534/src/query/parquet_writer.rs#L48C10-L48C28

I rely very much on the defaults, but setting to produce statistics explicitly did not change behavior.

from odbc2parquet.

Feel free to link this issue to the upstream ticket.

from odbc2parquet.

I wonder if it's the same as apache/arrow-rs#5162

from odbc2parquet.

This might very well be the case. I think text is written as "physical" type bytes with logical type UTF-8

from odbc2parquet.

The ticket was unrelated. My ticket: apache/arrow-rs#5270

from odbc2parquet.