Feature request: Progress bar for full table copies about odbc2parquet HOT 6 CLOSED

odbc2parquet 3.1.0 is released which emits a log message on info level stating the number of currently fetched rows.

from odbc2parquet.

Currently it is entirely possible to use odbc2parquet in situations there the first files has been written while even the data source is not sure how many rows the query will yield. odbc2parquet has no idea what the maximum numbers of row are. I am not confident that every data source would support count_big or that it would always be cheap to call.

Emitting the total number of rows fetched so far on the other hand is easy. I see nothing that would speak against that.

from odbc2parquet.

Yeah, the count thing would only make sense with a special "full table copy" mode, where you're not passing a query but a table name.

from odbc2parquet.

I have thought a bit about how to do a progress bar, but there is no good way to retrieve the total number of rows without making additional assumptions about the data source in question. The age old tradeoff between usefulness and re-usability.

However I could consider making the core functions of this CLI tool available as the library, allowing you to build a specialized CLI tool for your use-case on top. I would only extend the mental effort if this is something you would consider, though. Otherwise I am also fine with closing the issue and calling it a day.

Best, Markus

from odbc2parquet.

Ok! I don't think I would use that library given that I'm much more fluent in Python and there are various easy to use ways to fetch data from SQL Server and write to Parquet in Python.

from odbc2parquet.

Thanks for the feedback, closing the issue.

from odbc2parquet.