Comments (29)
chrisfw
commented on July 17, 2024
1
Hi Markus,
I tested the new version and I successfully queried and created a parquet
file from the entire history dataset. Thank you very much for implementing
this workaround - it is much appreciated!
Regards,
Chris
On Thu, Feb 10, 2022 at 8:32 PM Chris Whelan ***@***.***>
wrote:
… Thanks for work on the new release and the status update Markus. That is
disheartening about the Oracle 64 bit int insert behavior! You would think
they would fix these types of longstanding issues, but sadly they don't
seem to care.
Regards,
Chris
On Thu, Feb 10, 2022 at 5:07 PM Markus Klein ***@***.***>
wrote:
> Released version 0.6.22 with the --driver-does-not-support-64bit-integers
> flag. Not closing the issue, because I haven't done the helpful error
> message yet. However the flag itself I can introduce and test without
> having Oracle part of the test setup, so I figured I unblock your use case
> right now.
>
> Tested the insert path with a 64Bit. Expectedly the Oracle driver also
> failed, but without emitting any error at all, which I find a bit
> disheartening, on how to best tackle that.
>
> —
> Reply to this email directly, view it on GitHub
> <#143 (comment)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABOZ7PKLEAQMOXIGHVYWRULU2QZJJANCNFSM5NKLN6AA>
> .
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
>
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
from odbc2parquet.
pacman82
commented on July 17, 2024
Hello @chrisfw thanks for opening this issue, and providing the detailed log output. The datatype (-25) in the error message likely relates to BigInt, which is a hint, that the driver does not like the fact, that the tool is trying to fetch the columns with scale 0 as 64Bit integers. It seems the driver does at least not support this conversion for the SQL DataType, maybe the buffer can stay i64. Don't know yet.
I won't have time to investigate this deeper until the weekend, however in the meanwhile it would be helpful to verify the hypothesis, by trying to do this query without the columns those buffer description have an i64 in the kind attribute (see log output).
Ps. Am I right to assume you run this under windows?
Cheers, Markus
from odbc2parquet.
chrisfw
commented on July 17, 2024
Hi Markus,
Thanks for your prompt response. I will make the change to the query to
exclude those columns as you suggest and let you know if it works. Yes,
you are correct in your assumption that I am running it on Windows.
Regards,
Chris
…On Wed, Feb 2, 2022 at 1:56 AM Markus Klein ***@***.***> wrote:
Hello @chrisfw <https://github.com/chrisfw> thanks for opening this
issue, and providing the detailed log output. The datatype (-25) in the
error message likely relates to BigInt, which is a hint, that the driver
does not like the fact, that the tool is trying to fetch the columns with scale
0 as 64Bit integers. It seems the driver does at least not support this
conversion for the SQL DataType, maybe the buffer can stay i64. Don't know
yet.
I won't have time to investigate this deeper until the weekend, however in
the meanwhile it would be helpful to verify the hypothesis, by trying to do
this query without the columns those buffer description have an i64 in
the kind attribute (see log output).
Ps. Am I right to assume you run this under windows?
Cheers, Markus
—
Reply to this email directly, view it on GitHub
<#143 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABOZ7PNP5Q3VVXISZFWTUNDUZDIQRANCNFSM5NKLN6AA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you were mentioned.Message ID:
***@***.***>
from odbc2parquet.
chrisfw
commented on July 17, 2024
Hi Markus,
I tried what you suggested and there is no longer an error message, but the
generated parquet file is very small and the process completes in a second
or two, so it is not working properly given the known size of the table.
Please let me know what else I can do to help troubleshoot.
Thanks,
Chris
E:\history>odbc2parquet -vvv query --column-compression-default gzip -c
"Driver={Oracle in
instantclient_19_13};Dbq=WMPTST;Uid=redacted;Pwd=redacted;"
test.parquet "select
CIHIS_COMP,CIHIS_APPLICATION,CIHIS_DATE,CIHIS_SEQ,CIHIS_ELEMENT,CIHIS_TYPE_CODE,CIHIS_READ_SEQ,CIHIS_TRANS_TYPE,CIHIS_BILL_DATE,CIHIS_AMT,CIHIS_FACTOR,CIHIS_MANUAL,CIHIS_ADJDT,CIHIS_TYPE,CIHIS_COUNT,CIHIS_UT_TAX1_AMT,CIHIS_UT_TAX2_AMT,CIHIS_BANK,CIHIS_RATE,CIHIS_TAX_CD,CIHIS_TAX_TYPE,CIHIS_MTAX_KIND,CIHIS_BDOWN,CIHIS_BUDGET_CD,CIHIS_FROM_DATE,CIHIS_CLASS_TYPE,CIHIS_CLASS_NUM,CIHIS_PRO,CIHIS_POSTMARK_DT,CIHIS_TX_CODE,CIHIS_DELINQ_FLAG,CIHIS_DISPUTE_FLAG,CIHIS_ACTION,CIHIS_DESC,CIHIS_USE,CIHIS_ACCT_TP,CIHIS_TAXABLE_SALES,CIHIS_METERED_ITEM_RATE,CIHIS_NONTAXABLE_SALES,CIHIS_TAX_INSTALLMENT,CIHIS_TSS_BOD,CIHIS_MIN_ADD,FILLER_CD1,FILLER_CD2,CIHIS_OYR,CIHIS_TO_DATE,CIHIS_ITEM_SEQ,CIHIS_PPAC_TYPE,CIHIS_DEFER_DUE_DT,CIHIS_SERV_VOLT,CIHIS_SERVICE,CIHIS_BUDPAY,CIHIS_BILL_CONTRACT,CIHIS_BLOCK_AMT1,CIHIS_BLOCK_AMT2,CIHIS_CRB_PCT,CIHIS_CRB_NUM,CIHIS_BALANCE_AMOUNT,CIHIS_WORKORDER_APPLICATION
from fcihistv"
2022-02-02T07:11:12-05:00 - DEBUG - ODBC Environment created.
2022-02-02T07:11:12-05:00 - WARN - State: 01000, Native error: 0, Message:
[Microsoft][ODBC Driver Manager] The driver doesn't support the version of
ODBC behavior that the application requested (see SQLSetEnvAttr).
2022-02-02T07:11:12-05:00 - WARN - State: 01004, Native error: 0, Message:
[Oracle][ODBC]String data, right truncated.doesn't support the version of
ODBC behavi
2022-02-02T07:11:12-05:00 - DEBUG - ODBC column description for column 1:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 67, 79, 77, 80],
data_type: Decimal { precision: 2, scale: 0 }, nullability: Nullable }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC buffer description for column 1:
BufferDescription { nullable: true, kind: I32 }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC column description for column 2:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 65, 80, 80, 76, 73, 67,
65, 84, 73, 79, 78], data_type: Decimal { precision: 2, scale: 0 },
nullability: Nullable }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC buffer description for column 2:
BufferDescription { nullable: true, kind: I32 }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC column description for column 3:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 68, 65, 84, 69],
data_type: Decimal { precision: 8, scale: 0 }, nullability: Nullable }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC buffer description for column 3:
BufferDescription { nullable: true, kind: I32 }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC column description for column 4:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 83, 69, 81], data_type:
Decimal { precision: 7, scale: 0 }, nullability: Nullable }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC buffer description for column 4:
BufferDescription { nullable: true, kind: I32 }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC column description for column 5:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 69, 76, 69, 77, 69, 78,
84], data_type: Decimal { precision: 4, scale: 0 }, nullability: Nullable }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC buffer description for column 5:
BufferDescription { nullable: true, kind: I32 }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC column description for column 6:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 84, 89, 80, 69, 95, 67,
79, 68, 69], data_type: Decimal { precision: 4, scale: 0 }, nullability:
Nullable }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC buffer description for column 6:
BufferDescription { nullable: true, kind: I32 }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC column description for column 7:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 82, 69, 65, 68, 95, 83,
69, 81], data_type: Decimal { precision: 4, scale: 0 }, nullability:
Nullable }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC buffer description for column 7:
BufferDescription { nullable: true, kind: I32 }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC column description for column 8:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 84, 82, 65, 78, 83, 95,
84, 89, 80, 69], data_type: Decimal { precision: 2, scale: 0 },
nullability: Nullable }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC buffer description for column 8:
BufferDescription { nullable: true, kind: I32 }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC column description for column 9:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 66, 73, 76, 76, 95, 68,
65, 84, 69], data_type: Decimal { precision: 9, scale: 0 }, nullability:
Nullable }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC buffer description for column 9:
BufferDescription { nullable: true, kind: I32 }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC column description for column 10:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 65, 77, 84], data_type:
Decimal { precision: 11, scale: 2 }, nullability: Nullable }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC buffer description for column 10:
BufferDescription { nullable: true, kind: Text { max_str_len: 13 } }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC column description for column 11:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 70, 65, 67, 84, 79, 82],
data_type: Decimal { precision: 9, scale: 6 }, nullability: Nullable }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC buffer description for column 11:
BufferDescription { nullable: true, kind: Text { max_str_len: 11 } }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC column description for column 12:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 77, 65, 78, 85, 65, 76],
data_type: Decimal { precision: 2, scale: 0 }, nullability: Nullable }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC buffer description for column 12:
BufferDescription { nullable: true, kind: I32 }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC column description for column 13:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 65, 68, 74, 68, 84],
data_type: Decimal { precision: 9, scale: 0 }, nullability: Nullable }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC buffer description for column 13:
BufferDescription { nullable: true, kind: I32 }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC column description for column 14:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 84, 89, 80, 69],
data_type: Decimal { precision: 2, scale: 0 }, nullability: Nullable }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC buffer description for column 14:
BufferDescription { nullable: true, kind: I32 }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC column description for column 15:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 67, 79, 85, 78, 84],
data_type: Decimal { precision: 7, scale: 0 }, nullability: Nullable }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC buffer description for column 15:
BufferDescription { nullable: true, kind: I32 }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC column description for column 16:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 85, 84, 95, 84, 65, 88,
49, 95, 65, 77, 84], data_type: Decimal { precision: 11, scale: 2 },
nullability: Nullable }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC buffer description for column 16:
BufferDescription { nullable: true, kind: Text { max_str_len: 13 } }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC column description for column 17:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 85, 84, 95, 84, 65, 88,
50, 95, 65, 77, 84], data_type: Decimal { precision: 11, scale: 2 },
nullability: Nullable }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC buffer description for column 17:
BufferDescription { nullable: true, kind: Text { max_str_len: 13 } }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC column description for column 18:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 66, 65, 78, 75],
data_type: Decimal { precision: 4, scale: 0 }, nullability: Nullable }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC buffer description for column 18:
BufferDescription { nullable: true, kind: I32 }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC column description for column 19:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 82, 65, 84, 69],
data_type: Decimal { precision: 4, scale: 0 }, nullability: Nullable }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC buffer description for column 19:
BufferDescription { nullable: true, kind: I32 }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC column description for column 20:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 84, 65, 88, 95, 67, 68],
data_type: Decimal { precision: 2, scale: 0 }, nullability: Nullable }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC buffer description for column 20:
BufferDescription { nullable: true, kind: I32 }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC column description for column 21:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 84, 65, 88, 95, 84, 89,
80, 69], data_type: Decimal { precision: 2, scale: 0 }, nullability:
Nullable }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC buffer description for column 21:
BufferDescription { nullable: true, kind: I32 }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC column description for column 22:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 77, 84, 65, 88, 95, 75,
73, 78, 68], data_type: Decimal { precision: 2, scale: 0 }, nullability:
Nullable }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC buffer description for column 22:
BufferDescription { nullable: true, kind: I32 }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC column description for column 23:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 66, 68, 79, 87, 78],
data_type: Decimal { precision: 2, scale: 0 }, nullability: Nullable }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC buffer description for column 23:
BufferDescription { nullable: true, kind: I32 }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC column description for column 24:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 66, 85, 68, 71, 69, 84,
95, 67, 68], data_type: Decimal { precision: 2, scale: 0 }, nullability:
Nullable }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC buffer description for column 24:
BufferDescription { nullable: true, kind: I32 }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC column description for column 25:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 70, 82, 79, 77, 95, 68,
65, 84, 69], data_type: Decimal { precision: 9, scale: 0 }, nullability:
Nullable }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC buffer description for column 25:
BufferDescription { nullable: true, kind: I32 }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC column description for column 26:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 67, 76, 65, 83, 83, 95,
84, 89, 80, 69], data_type: Decimal { precision: 4, scale: 0 },
nullability: Nullable }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC buffer description for column 26:
BufferDescription { nullable: true, kind: I32 }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC column description for column 27:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 67, 76, 65, 83, 83, 95,
78, 85, 77], data_type: Decimal { precision: 4, scale: 0 }, nullability:
Nullable }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC buffer description for column 27:
BufferDescription { nullable: true, kind: I32 }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC column description for column 28:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 80, 82, 79], data_type:
Decimal { precision: 2, scale: 0 }, nullability: Nullable }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC buffer description for column 28:
BufferDescription { nullable: true, kind: I32 }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC column description for column 29:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 80, 79, 83, 84, 77, 65,
82, 75, 95, 68, 84], data_type: Decimal { precision: 9, scale: 0 },
nullability: Nullable }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC buffer description for column 29:
BufferDescription { nullable: true, kind: I32 }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC column description for column 30:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 84, 88, 95, 67, 79, 68,
69], data_type: Decimal { precision: 2, scale: 0 }, nullability: Nullable }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC buffer description for column 30:
BufferDescription { nullable: true, kind: I32 }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC column description for column 31:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 68, 69, 76, 73, 78, 81,
95, 70, 76, 65, 71], data_type: Varchar { length: 1 }, nullability:
Nullable }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC buffer description for column 31:
BufferDescription { nullable: true, kind: WText { max_str_len: 2 } }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC column description for column 32:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 68, 73, 83, 80, 85, 84,
69, 95, 70, 76, 65, 71], data_type: Varchar { length: 1 }, nullability:
Nullable }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC buffer description for column 32:
BufferDescription { nullable: true, kind: WText { max_str_len: 2 } }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC column description for column 33:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 65, 67, 84, 73, 79, 78],
data_type: Decimal { precision: 2, scale: 0 }, nullability: Nullable }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC buffer description for column 33:
BufferDescription { nullable: true, kind: I32 }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC column description for column 34:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 68, 69, 83, 67],
data_type: Varchar { length: 20 }, nullability: Nullable }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC buffer description for column 34:
BufferDescription { nullable: true, kind: WText { max_str_len: 40 } }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC column description for column 35:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 85, 83, 69], data_type:
Decimal { precision: 9, scale: 0 }, nullability: Nullable }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC buffer description for column 35:
BufferDescription { nullable: true, kind: I32 }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC column description for column 36:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 65, 67, 67, 84, 95, 84,
80], data_type: Varchar { length: 1 }, nullability: Nullable }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC buffer description for column 36:
BufferDescription { nullable: true, kind: WText { max_str_len: 2 } }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC column description for column 37:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 84, 65, 88, 65, 66, 76,
69, 95, 83, 65, 76, 69, 83], data_type: Decimal { precision: 11, scale: 2
}, nullability: Nullable }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC buffer description for column 37:
BufferDescription { nullable: true, kind: Text { max_str_len: 13 } }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC column description for column 38:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 77, 69, 84, 69, 82, 69,
68, 95, 73, 84, 69, 77, 95, 82, 65, 84, 69], data_type: Decimal {
precision: 4, scale: 0 }, nullability: Nullable }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC buffer description for column 38:
BufferDescription { nullable: true, kind: I32 }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC column description for column 39:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 78, 79, 78, 84, 65, 88,
65, 66, 76, 69, 95, 83, 65, 76, 69, 83], data_type: Decimal { precision:
11, scale: 2 }, nullability: Nullable }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC buffer description for column 39:
BufferDescription { nullable: true, kind: Text { max_str_len: 13 } }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC column description for column 40:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 84, 65, 88, 95, 73, 78,
83, 84, 65, 76, 76, 77, 69, 78, 84], data_type: Decimal { precision: 2,
scale: 0 }, nullability: Nullable }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC buffer description for column 40:
BufferDescription { nullable: true, kind: I32 }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC column description for column 41:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 84, 83, 83, 95, 66, 79,
68], data_type: Decimal { precision: 11, scale: 2 }, nullability: Nullable }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC buffer description for column 41:
BufferDescription { nullable: true, kind: Text { max_str_len: 13 } }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC column description for column 42:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 77, 73, 78, 95, 65, 68,
68], data_type: Decimal { precision: 11, scale: 2 }, nullability: Nullable }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC buffer description for column 42:
BufferDescription { nullable: true, kind: Text { max_str_len: 13 } }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC column description for column 43:
ColumnDescription { name: [70, 73, 76, 76, 69, 82, 95, 67, 68, 49],
data_type: Decimal { precision: 2, scale: 0 }, nullability: Nullable }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC buffer description for column 43:
BufferDescription { nullable: true, kind: I32 }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC column description for column 44:
ColumnDescription { name: [70, 73, 76, 76, 69, 82, 95, 67, 68, 50],
data_type: Decimal { precision: 2, scale: 0 }, nullability: Nullable }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC buffer description for column 44:
BufferDescription { nullable: true, kind: I32 }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC column description for column 45:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 79, 89, 82], data_type:
Decimal { precision: 4, scale: 0 }, nullability: Nullable }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC buffer description for column 45:
BufferDescription { nullable: true, kind: I32 }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC column description for column 46:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 84, 79, 95, 68, 65, 84,
69], data_type: Decimal { precision: 9, scale: 0 }, nullability: Nullable }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC buffer description for column 46:
BufferDescription { nullable: true, kind: I32 }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC column description for column 47:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 73, 84, 69, 77, 95, 83,
69, 81], data_type: Decimal { precision: 7, scale: 0 }, nullability:
Nullable }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC buffer description for column 47:
BufferDescription { nullable: true, kind: I32 }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC column description for column 48:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 80, 80, 65, 67, 95, 84,
89, 80, 69], data_type: Decimal { precision: 4, scale: 0 }, nullability:
Nullable }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC buffer description for column 48:
BufferDescription { nullable: true, kind: I32 }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC column description for column 49:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 68, 69, 70, 69, 82, 95,
68, 85, 69, 95, 68, 84], data_type: Decimal { precision: 9, scale: 0 },
nullability: Nullable }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC buffer description for column 49:
BufferDescription { nullable: true, kind: I32 }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC column description for column 50:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 83, 69, 82, 86, 95, 86,
79, 76, 84], data_type: Decimal { precision: 9, scale: 0 }, nullability:
Nullable }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC buffer description for column 50:
BufferDescription { nullable: true, kind: I32 }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC column description for column 51:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 83, 69, 82, 86, 73, 67,
69], data_type: Decimal { precision: 4, scale: 0 }, nullability: Nullable }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC buffer description for column 51:
BufferDescription { nullable: true, kind: I32 }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC column description for column 52:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 66, 85, 68, 80, 65, 89],
data_type: Decimal { precision: 11, scale: 2 }, nullability: Nullable }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC buffer description for column 52:
BufferDescription { nullable: true, kind: Text { max_str_len: 13 } }
2022-02-02T07:11:12-05:00 - DEBUG - ODBC column description for column 53:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 66, 73, 76, 76, 95, 67,
79, 78, 84, 82, 65, 67, 84], data_type: Varchar { length: 10 },
nullability: Nullable }
2022-02-02T07:11:13-05:00 - DEBUG - ODBC buffer description for column 53:
BufferDescription { nullable: true, kind: WText { max_str_len: 20 } }
2022-02-02T07:11:13-05:00 - DEBUG - ODBC column description for column 54:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 66, 76, 79, 67, 75, 95,
65, 77, 84, 49], data_type: Decimal { precision: 11, scale: 2 },
nullability: Nullable }
2022-02-02T07:11:13-05:00 - DEBUG - ODBC buffer description for column 54:
BufferDescription { nullable: true, kind: Text { max_str_len: 13 } }
2022-02-02T07:11:13-05:00 - DEBUG - ODBC column description for column 55:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 66, 76, 79, 67, 75, 95,
65, 77, 84, 50], data_type: Decimal { precision: 11, scale: 2 },
nullability: Nullable }
2022-02-02T07:11:13-05:00 - DEBUG - ODBC buffer description for column 55:
BufferDescription { nullable: true, kind: Text { max_str_len: 13 } }
2022-02-02T07:11:13-05:00 - DEBUG - ODBC column description for column 56:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 67, 82, 66, 95, 80, 67,
84], data_type: Decimal { precision: 9, scale: 6 }, nullability: Nullable }
2022-02-02T07:11:13-05:00 - DEBUG - ODBC buffer description for column 56:
BufferDescription { nullable: true, kind: Text { max_str_len: 11 } }
2022-02-02T07:11:13-05:00 - DEBUG - ODBC column description for column 57:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 67, 82, 66, 95, 78, 85,
77], data_type: Decimal { precision: 2, scale: 0 }, nullability: Nullable }
2022-02-02T07:11:13-05:00 - DEBUG - ODBC buffer description for column 57:
BufferDescription { nullable: true, kind: I32 }
2022-02-02T07:11:13-05:00 - DEBUG - ODBC column description for column 58:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 66, 65, 76, 65, 78, 67,
69, 95, 65, 77, 79, 85, 78, 84], data_type: Decimal { precision: 11, scale:
2 }, nullability: Nullable }
2022-02-02T07:11:13-05:00 - DEBUG - ODBC buffer description for column 58:
BufferDescription { nullable: true, kind: Text { max_str_len: 13 } }
2022-02-02T07:11:13-05:00 - DEBUG - ODBC column description for column 59:
ColumnDescription { name: [67, 73, 72, 73, 83, 95, 87, 79, 82, 75, 79, 82,
68, 69, 82, 95, 65, 80, 80, 76, 73, 67, 65, 84, 73, 79, 78], data_type:
Decimal { precision: 2, scale: 0 }, nullability: Nullable }
2022-02-02T07:11:13-05:00 - DEBUG - ODBC buffer description for column 59:
BufferDescription { nullable: true, kind: I32 }
2022-02-02T07:11:13-05:00 - INFO - Memory usage per row is 1015 bytes. This
excludes memory directly allocated by the ODBC driver.
2022-02-02T07:11:13-05:00 - INFO - Batch size set to 65535 rows.
E:\history>dir
Volume in drive E is Samsung_T5
Volume Serial Number is 249C-CC44
Directory of E:\history
02/01/2022 03:55 PM <DIR> .
02/01/2022 03:55 PM <DIR> ..
02/02/2022 07:11 AM 1,665 test.parquet
1 File(s) 1,665 bytes
2 Dir(s) 681,636,134,912 bytes free
On Wed, Feb 2, 2022 at 6:46 AM Chris Whelan ***@***.***>
wrote:
… Hi Markus,
Thanks for your prompt response. I will make the change to the query to
exclude those columns as you suggest and let you know if it works. Yes,
you are correct in your assumption that I am running it on Windows.
Regards,
Chris
On Wed, Feb 2, 2022 at 1:56 AM Markus Klein ***@***.***>
wrote:
> Hello @chrisfw <https://github.com/chrisfw> thanks for opening this
> issue, and providing the detailed log output. The datatype (-25) in the
> error message likely relates to BigInt, which is a hint, that the driver
> does not like the fact, that the tool is trying to fetch the columns with scale
> 0 as 64Bit integers. It seems the driver does at least not support this
> conversion for the SQL DataType, maybe the buffer can stay i64. Don't know
> yet.
>
> I won't have time to investigate this deeper until the weekend, however
> in the meanwhile it would be helpful to verify the hypothesis, by trying to
> do this query without the columns those buffer description have an i64
> in the kind attribute (see log output).
>
> Ps. Am I right to assume you run this under windows?
>
> Cheers, Markus
>
> —
> Reply to this email directly, view it on GitHub
> <#143 (comment)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABOZ7PNP5Q3VVXISZFWTUNDUZDIQRANCNFSM5NKLN6AA>
> .
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
>
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
from odbc2parquet.
pacman82
commented on July 17, 2024
Is there a free version of the database you are using available, so we can include it in the test suite? I'd very much like to have a reproducing example?
from odbc2parquet.
chrisfw
commented on July 17, 2024
Hi Markus,
Yes, I believe you can download Oracle XE for Windows directly from
Oracle's website once you set up a free account.
Regards,
Chris
…On Wed, Feb 2, 2022 at 7:52 AM Markus Klein ***@***.***> wrote:
Is there a free version of the database you are using available, so we can
include it in the test suite? I'd very much like to have a reproducing
example?
—
Reply to this email directly, view it on GitHub
<#143 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABOZ7PLDV56BBTJVCEAGV4TUZESIJANCNFSM5NKLN6AA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you were mentioned.Message ID:
***@***.***>
from odbc2parquet.
pacman82
commented on July 17, 2024
For reference:
- https://github.com/microsoft/ODBC-Specification/blob/b7ef71fba508ed010cd979428efae3091b732d75/Windows/inc/sqlext.h#L592
SQL_BIGINTis-5SQL_SIGNED_OFFESTis-20
from odbc2parquet.
pacman82
commented on July 17, 2024
Setting up an Oracle is quite involved, was hoping for a ready made container easy to integrate into CI 😅 .
Anyhow, I think I now get at least the error messeg. Contrary to what it says, it is not about the SQL Data Type, but about the C Buffer Type we are binding. As seen in the linked reference above. Note that the Constant is declared in a header file name sqlext.h, meaning that it is an extension to the standard.
So far however 64Bit integers were supported by every database I encounterd. However a search revealed this stack overflow question.
Seemingly indicating that Oracle does not support bindig variables to 64 Bit integer buffers. Please note that the question is kinda old. Are you running the neweset version? Maybe Oracle added support in the meanwhile. So far however it seems the biggest improvement is that they added the constant -25 to the error message, giving users who browse C headers and add constantst an idea of what might be going on.
Regards what can be done in terms of this tool. I think it is a reasonable default behaviour to query Numeric(1..=18,0) as i64. It's also very sensible to let the driver do the conversion. Yet I could think about adding a flag, allowing to query those as text, and convert them from text to i64 within the tool itself. However setting a magic flag after interpreting an obscure error message is not the kind of UX im aiming for.
As a workaroud you could think a about casting every Numeric column with scale zero and precision greater than 9 (so starting at 10) into a Varchar. That way these would be fetched as text. You would have to post process the resulting parquet file though, to turn these back into integer colums, as they would be treated as text.
The sudden stoppage of the tool is worrysome. It can mean one of two things. The Oracle ODBC driver does causes an invalid memory access, double free or other kind of segmentation fault. Or one of the invariants I thought always to be true during writing the code, turns out to be violated.
Since I still have trouble with my test setup, I would kindly ask you to urn the debug build of the tool. Instruction of how to achieve that will follow shortly, but I now have to leave my PC so my GF can use it for gaming.
Cheers, Markus
from odbc2parquet.
pacman82
commented on July 17, 2024
Hello @chrisfw ,
In case you find the time to help out diagnosing this thing.
To run obtain a debug build of the tool, you would need a rust toolchain.
https://rustup.rs/ contains instruction of how to install one. It's just executing the script, and at one point you need to download either Visual Studio, or the Visual Studio Build tools to get the msvc compiler backend.
Checking out the repository and typing cargo build will result in the debug executable present in the target\debug subdirectory.
I could also just mail, or upload the executable to you, if you give me some coordinates.
Cheers, Markus
from odbc2parquet.
chrisfw
commented on July 17, 2024
Hi Markus,
Thanks for following up on this issue. FYI, the version of Oracle I am using is 19c which is fairly recent.
I will work on creating the debug version of the tool and testing the problematic table this week. I’ll let you know the results.
Thanks for your thoughts on possible workarounds, but I’m really hoping that a better solution will present itself after some further debugging. Oracle is one of the most heavily utilized databases in the world, so it would be a shame if a workaround is required in order to utilize the tool with it.
Regards,
Chris
… On Feb 6, 2022, at 9:36 AM, Markus Klein ***@***.***> wrote:
Hello @chrisfw ,
In case you find the time to help out diagnosing this thing.
To run obtain a debug build of the tool, you would need a rust toolchain.
https://rustup.rs/ contains instruction of how to install one. It's just executing the script, and at one point you need to download either Visual Studio, or the Visual Studio Build tools to get the msvc compiler backend.
Checking out the repository and typing cargo build will result in the debug executable present in the target\debug subdirectory.
I could also just mail, or upload the executable to you, if you give me some coordinates.
Cheers, Markus
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
You are receiving this because you were mentioned.
from odbc2parquet.
pacman82
commented on July 17, 2024
Hello Chris,
Thanks for your thoughts on possible workarounds, but I’m really hoping that a better solution will present itself after some further debugging. Oracle is one of the most heavily utilized databases in the world, so it would be a shame if a workaround is required in order to utilize the tool with it.
Very much thinking the same thing.
With regards to the conversion:
The error state reported by the driver should be HY003 IMHO and not HY004. (Meaning it's the buffer type which can't be bound. If it were about the SQL Type it should have said -5.). Also the error should be returned on SQLBindParameter instead of SQLFetch.
Maybe that's worth an upstream bug ticket.
Anyhow, with regards to the conversion, the solution might be to query the NUMERIC columns as text, and perform the conversion within the tool, instead of relying on the driver. This switch of behaviour could be activated with a flag on the command line. However, it would be a better UX if odbc2parquet would detect the error and tried again automatically with the adapted behaviour. Preconditions for the better UX are however:
- We can test the error handling code paths for Oracle on CI
- First call to
SQLFetchdid not change the state of the cursor, so we can just try again with a different set of bound buffers, without losing data.
Early termination of the tool so far is a mystery. Curious on what kind of stack trace I'll see.
FYI: $env:RUST_BACKTRACE="1" sets the environment in power shell, so we see more, if it fails with the debug version of the tool.
Cheers, Markus
from odbc2parquet.
chrisfw
commented on July 17, 2024
Thanks for following up Markus. I like your idea for having the tool detect the problem and do the necessary conversion.
It will likely be a few days before I am able to test with the debug version, but I will do so at the earliest opportunity.
One thought I had that may be a possibility for you is to simply create a free oracle cloud infrastructure account and spin up a free tier oracle autonomous database. If that is problematic, I may be able to create one for you to use temporarily for testing. Let me know your thoughts on this idea when you have a chance.
Regards,
Chris
… On Feb 7, 2022, at 3:07 AM, Markus Klein ***@***.***> wrote:
Hello Chris,
Thanks for your thoughts on possible workarounds, but I’m really hoping that a better solution will present itself after some further debugging. Oracle is one of the most heavily utilized databases in the world, so it would be a shame if a workaround is required in order to utilize the tool with it.
Very much thinking the same thing.
With regards to the conversion:
The error state reported by the driver should be HY003 IMHO and not HY004. (Meaning it's the buffer type which can't be bound. If it were about the SQL Type it should have said -5.). Also the error should be returned on SQLBindParameter instead of SQLFetch.
Maybe that's worth an upstream bug ticket.
Anyhow, with regards to the conversion, the solution might be to query the NUMERIC columns as text, and perform the conversion within the tool, instead of relying on the driver. This switch of behaviour could be activated with a flag on the command line. However, it would be a better UX if odbc2parquet would detect the error and tried again automatically with the adapted behaviour. Preconditions for the better UX are however:
We can test the error handling code paths for Oracle on CI
First call to SQLFetch did not change the state of the cursor, so we can just try again with a different set of bound buffers, without losing data.
Early termination of the tool so far is a mystery. Curious on what kind of stack trace I'll see.
FYI: $env:RUST_BACKTRACE="1" sets the environment in power shell, so we see more, if it fails with the debug version of the tool.
Cheers, Markus
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
You are receiving this because you were mentioned.
from odbc2parquet.
pacman82
commented on July 17, 2024
Hello Chris,
One thought I had that may be a possibility for you is to simply create a free oracle cloud infrastructure account and spin up a free tier oracle autonomous database.
Nice idea. I'll look into that. Especially for CI. I did manage to build an oracle XE locally by the way. Yet even my humple attempts to create a table failed with a syntax error. At least I managed to figure out the connection string and did successfully connect.
Same here, it will be quite a few days before I can look deeper into this again.
In terms of experiments (only of course if you find the time):
There are four types of columns in your query.
- Character columns which will be queried as text.
- Numeric columns with scale zero which fit into a 32Bit integer.
- Numeric columns with scale zero which fit into a 64Bit integer.
- Numeric columns with scale != 0, which are queried as text.
I wonder if there is a subset of columns, for which the tool would work?
Cheers, Markus
from odbc2parquet.
pacman82
commented on July 17, 2024
Yet even my humple attempts to create a table failed with a syntax error.
Writing this I just realized that round brackets are magic in bash. 🤦
from odbc2parquet.
chrisfw
commented on July 17, 2024
Thanks for the update and info Markus. Sounds good. FYI, if you aren’t already aware Oracle SQL Developer and DBeaver are free gui tools that allow visual table creation and also allow you to easily generate the ddl associated with the various db objects which can then easily be utilized in ci/cd.
Regards,
Chris
… On Feb 8, 2022, at 11:57 AM, Markus Klein ***@***.***> wrote:
Yet even my humple attempts to create a table failed with a syntax error.
Writing this I just realized that round brackets are magic in bash. 🤦
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
You are receiving this because you were mentioned.
from odbc2parquet.
chrisfw
commented on July 17, 2024
Hi again Markus,
I had an opportunity to build the debug version of the exe and test, but I
actually discovered that the "problem" I thought I was seeing was not
actually a problem. There was an empty copy of the table in the connection
user's schema which was effectively hiding the real target tables, so the
tool was not crashing - there just wasn't any data in the table it was
accessing. Once I qualified the table name in the query with the actual
target schema, I was able to run the query limited to the column types
supported by the tool and the parquet file is generated with data in it.
So now we are left only with the original Oracle driver 64bit int mapping
problem. Please let me know if there is anything else I can do to help
troubleshoot.
Regards,
Chris
On Tue, Feb 8, 2022 at 1:33 PM Chris Whelan ***@***.***>
wrote:
… Thanks for the update and info Markus. Sounds good. FYI, if you aren’t
already aware Oracle SQL Developer and DBeaver are free gui tools that
allow visual table creation and also allow you to easily generate the ddl
associated with the various db objects which can then easily be utilized in
ci/cd.
Regards,
Chris
On Feb 8, 2022, at 11:57 AM, Markus Klein ***@***.***>
wrote:
Yet even my humple attempts to create a table failed with a syntax error.
Writing this I just realized that round brackets are magic in bash. 🤦
—
Reply to this email directly, view it on GitHub
<#143 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABOZ7PKVXSBDS6QYOXHHLADU2FDPBANCNFSM5NKLN6AA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you were mentioned.Message ID:
***@***.***>
from odbc2parquet.
pacman82
commented on July 17, 2024
Thanks Chris,
Boy, that's a relief. Well the mapping issue is well understood. You can even read in the official documentation that 64Bit Integers are not supported. The public Oracle Forums are mourning this since decades it seems. Workaround is also understood, query these as string and convert. I'll add a flag to the tool. I am a bit undecided if I should just check for Oracle in the database name, or give the user a better error message (with a hint to the flag) if he/she encounters an error.
Cheers, Markus
from odbc2parquet.
chrisfw
commented on July 17, 2024
Sounds good Markus. I think giving the user a better error message with
the hint of the flag usage would be fine unless you can always reliably
detect that it is Oracle through the DSN info (questionable?) or the
provided Driver info (which seems like it would always have the string) .
Please let me know what you decide to do.
Thanks for all of your assistance.
Regards,
Chris
…On Tue, Feb 8, 2022 at 5:14 PM Markus Klein ***@***.***> wrote:
Thanks Chris,
Boy, that's a relief. Well the mapping issue is well understood. You can
even read in the official documentation that 64Bit Integers are not
supported. The public Oracle Forums are mourning this since decades it
seems. Workaround is also understood, query these as string and convert.
I'll add a flag to the tool. I am a bit undecided if I should just check
for Oracle in the database name, or give the user a better error message
(with a hint to the flag) if he/she encounters an error.
Cheers, Markus
—
Reply to this email directly, view it on GitHub
<#143 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABOZ7POG7QFRWRFQ3ZDNMYDU2GIWDANCNFSM5NKLN6AA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you were mentioned.Message ID:
***@***.***>
from odbc2parquet.
pacman82
commented on July 17, 2024
I should just check for Oracle in the database name
I ment to write driver name.
However I think I use the flag, so if any other ODBC drivers out there, don't like 64 Bit integers it can also be used for them.
Cheers, Markus
from odbc2parquet.
chrisfw
commented on July 17, 2024
Hi Markus,
The flag approach sounds like the right call here. I look forward to
testing it out when a build with the flag is available. Thanks a lot!
Regards,
Chris
…On Wed, Feb 9, 2022 at 3:14 AM Markus Klein ***@***.***> wrote:
I should just check for Oracle in the database name
I ment to write driver name.
However I think I use the flag, so if any other ODBC drivers out there,
don't like 64 Bit integers it can also be used for them.
Cheers, Markus
—
Reply to this email directly, view it on GitHub
<#143 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABOZ7PIZTVHZX2LCIKQCMSLU2IO77ANCNFSM5NKLN6AA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you were mentioned.Message ID:
***@***.***>
from odbc2parquet.
pacman82
commented on July 17, 2024
Released version 0.6.22 with the --driver-does-not-support-64bit-integers flag. Not closing the issue, because I haven't done the helpful error message yet. However the flag itself I can introduce and test without having Oracle part of the test setup, so I figured I unblock your use case right now.
Tested the insert path with a 64Bit. Expectedly the Oracle driver also failed, but without emitting any error at all, which I find a bit disheartening, on how to best tackle that.
from odbc2parquet.
chrisfw
commented on July 17, 2024
Thanks for work on the new release and the status update Markus. That is
disheartening about the Oracle 64 bit int insert behavior! You would think
they would fix these types of longstanding issues, but sadly they don't
seem to care.
Regards,
Chris
…On Thu, Feb 10, 2022 at 5:07 PM Markus Klein ***@***.***> wrote:
Released version 0.6.22 with the --driver-does-not-support-64bit-integers
flag. Not closing the issue, because I haven't done the helpful error
message yet. However the flag itself I can introduce and test without
having Oracle part of the test setup, so I figured I unblock your use case
right now.
Tested the insert path with a 64Bit. Expectedly the Oracle driver also
failed, but without emitting any error at all, which I find a bit
disheartening, on how to best tackle that.
—
Reply to this email directly, view it on GitHub
<#143 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABOZ7PKLEAQMOXIGHVYWRULU2QZJJANCNFSM5NKLN6AA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you were mentioned.Message ID:
***@***.***>
from odbc2parquet.
pacman82
commented on July 17, 2024
It seems history (unsuprisingly) repeats itself here: blue-yonder/turbodbc#142
I'll go add the better error message upstream in odbc-api, this way useres of arrow-odbc and arrow-odbc-py, won't be left to interpret that all by themeselfes.
from odbc2parquet.
chrisfw
commented on July 17, 2024
Well, at least the situation is already well-understood and well in hand. Thanks for nothing Oracle.
… On Feb 11, 2022, at 12:45 PM, Markus Klein ***@***.***> wrote:
It seems history (unsuprisingly) repeats itself here: blue-yonder/turbodbc#142
I'll go add the better error message upstream in odbc-api, this way useres of arrow-odbc and arrow-odbc-py, won't be left to interpret that all by themeselfes.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
You are receiving this because you were mentioned.
from odbc2parquet.
pacman82
commented on July 17, 2024
Hello @chrisfw,
I have created a free tier account in the oracle cloud; I created a database; I also have an ADMIN user with a password. Now I am trying to figure out, how the ODBC connection string would look like. Managed to setup Oracle locally and test, but for the CI I would prefer the cloud solution. Mostly because Oracle does not seem to have ready made containers on Docker Hub.
Pretty sure I can get this done on my own, but I thought I might save some time asking you, since you seem to have experience with that.
Thanks for nothing Oracle.
To be fair, it is very unlikely that it is ill will on their side. It is a hard problem to structure a big organization, so a group of people both understands the buisness value of something technical like a good ODBC driver and has the autonomy, the resources and skills to act on it.
Stuff like good error messages and 64Bit support seldomly make it into a sales call, but in a world there developers have increasing autonomy to choose the tech they use, it starts to matter. However much we suffer, my guess is, there is a person in Oracle who has the skills, and will to improve that driver, but just can't convince his/her manager to spend time on that. I am also guessing that person feels the pain more than we do.
from odbc2parquet.
chrisfw
commented on July 17, 2024
Hi Markus,
I have been working with an on-premise database for my testing with the
connection string below. I have only used JDBC connections to an Oracle
cloud autonomous db and it required some additional steps, e.g. setting up
the Oracle wallet and if I recall changes to sqlnet.ora and tnsnames.ora.
These steps are probably also required for ODBC.
I may be able to get you a docker image of 19c. We use Docker heavily in
our workflow and build our app and db in containers. I will see what I can
do to get you an empty db as our ansible workflow generates our dbs
prepopulated with our application's schema so I will need to get you one
without that step being executed.
Yes, I understand it isn't a matter of ill-will on Oracle's part, but
unfortunately we have spent way too much time and money working on the
Oracle database platform. Don't get me wrong, it is a great database - but
it is the most expensive software you could ever imagine and purchasing an
Enterprise Edition license does not unlock all of the features. It only
gives you some and the ability to purchase others. Every feature that I
have wanted to take advantage of as a developer I discover is yet another
additional package or component that must be purchased separately and is
licensed per CPU for some exorbitant cost, e.g. $17,000/Core. Oracle
licensing costs put us in a situation where we cannot fully take advantage
of the elasticity of the cloud because adding CPUs to a VM that hosts an
Oracle database is an extremely expensive endeavor. As a result we have to
limit the number of CPUs we can provision not because we want to, but
because it is too costly in terms of Oracle licensing not to limit them.
So, yes I would agree with you regarding being better off than the
engineers who are stuck working for the profit-center that is Oracle.
…On Sat, Feb 12, 2022 at 5:24 AM Markus Klein ***@***.***> wrote:
Hello @chrisfw <https://github.com/chrisfw>,
I have created a free tier account in the oracle cloud; I created a
database; I also have an ADMIN user with a password. Now I am trying to
figure out, how the ODBC connection string would look like. Managed to
setup Oracle locally and test, but for the CI I would prefer the cloud
solution. Mostly because Oracle does not seem to have ready made containers
on Docker Hub.
Pretty sure I can get this done on my own, but I thought I might save some
time asking you, since you seem to have experience with that.
Thanks for nothing Oracle.
To be fair, it is very unlikely that it is ill will on their side. It is a
hard problem to structure a big organization, so a group of people both
understands the buisness value of something technical like a good ODBC
driver and has the autonomy, the resources and skills to act on it.
Stuff like good error messages and 64Bit support seldomly make it into a
sales call, but in a world there developers have increasing autonomy to
choose the tech they use, it starts to matter. However much we suffer, my
guess is, there is a person in Oracle who has the skills, and will to
improve that driver, but just can't convince his/her manager to spend time
on that. I am also convinced that person suffers more than we do.
—
Reply to this email directly, view it on GitHub
<#143 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABOZ7PIUKVFRNKUNG4PC6Y3U2YYODANCNFSM5NKLN6AA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you were mentioned.Message ID:
***@***.***>
from odbc2parquet.
pacman82
commented on July 17, 2024
I may be able to get you a docker image of 19c.
FYI: I've been able to build a docker image using this repository https://github.com/oracle/docker-images .
Yet I find checking this repository out. Calling the script, to build the docker container, to involved for CI. I just want a ready image with an empty database and a premade user, which I can just docker pull.
However publishing the one I have to docker hub, I'm to scared to do, because I do not want to get in hot water with regards to licensing.
I'll be okay with either just calling docker build on some nice docker file for the CI, or using the ATP instance I spun up on Oracle Cloud, whichever seems less effort.
Preferably still the dockerfile, because there is less friction with tests, if somebody wants to fork and run tests.
from odbc2parquet.
chrisfw
commented on July 17, 2024
Hi Markus,
Ok, good to know. I totally understand your hesitancy with regard to
Oracle licensing as I feel the same way.
Thanks again for all of your (and other contributors') work on this tool
and the changes you made to get this working with the Oracle driver for me!
Regards,
Chris
…On Mon, Feb 14, 2022 at 8:49 AM Markus Klein ***@***.***> wrote:
I may be able to get you a docker image of 19c.
FYI: I've been able to build a docker image using this repository
https://github.com/oracle/docker-images .
Yet I find checking this repository out. Calling the script, to build the
docker container, to involved for CI. I just want a ready image with an
empty database and a premade user, which I can just docker pull.
However publishing the one I have to docker hub, I'm to scared to do,
because I do not want to get in hot water with regards to licensing.
I'll be okay with either just calling docker build on some nice docker
file for the CI, or using the ATP instance I spun up on Oracle Cloud,
whichever seems less effort.
Preferably still the dockerfile, because there is less friction with
tests, if somebody wants to fork and run tests.
—
Reply to this email directly, view it on GitHub
<#143 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABOZ7PI5FTWVFGTUIKQURP3U3D4ULANCNFSM5NKLN6AA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you were mentioned.Message ID:
***@***.***>
from odbc2parquet.
pacman82
commented on July 17, 2024
Closing this issue, because I'll probably won't find the time to include Oracle in the test setup in a sensible way. However, the tool will emit a hint for Oracle users regardless.
Line 253 in de06683
from odbc2parquet.
Related Issues (20)
- Issue with MySQL JSON columns HOT 8
- Reserved Column Names not Supported HOT 1
- Feature Request - Support column encryption in the generated parquet file HOT 4
- JobName as .sql file in config file HOT 4
- Parquet format version support HOT 9
- Feature suggestion: connect to URL `postgresql://username:pass@host/database` HOT 1
- What permissions are needed? - State: 42501, Native error: 1, Message: ERROR: permission denied HOT 4
- StarRocks parquet file import of parquet file generated by odbc2parquet fails with encoding error HOT 11
- Memory allocation with column-length-limit HOT 11
- Build for alpine HOT 8
- file-size-threshold generates wrong size files HOT 1
- --no-empty-file option doesn't work properly when row-groups-per-file should devide result into few files HOT 6
- MSSQL nvarchar - missing column in output file HOT 2
- Feature request: Progress bar for full table copies HOT 6
- Data source must return valid UTF16 in wide character buffer: Utf16Error HOT 4
- Write statistics HOT 14
- Make zstd the default compression HOT 4
- Build release assets for Ubuntu ARM64 as well HOT 11
- Exporter adding trailing zero's in when exporting from PostgreSQL Numeric dtype HOT 5
- thread 'main' panicked at src/query/date.rs:60:87: called `Option::unwrap()` on a `None` value HOT 7
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from odbc2parquet.