Charsets about firebirdex HOT 15 CLOSED

I did some testing on my database with charset: :cp1252 and it all seems to work perfectly (also with text in BLOB fields).
Great job!!
Are you planning to integrate it into master?

from firebirdex.

merged

from firebirdex.

I have seen your help wanted label, but I'm afraid most of the work will have to be done in efirebirdsql and although I can read some Erlang I don't know Erlang enough to actually create a PR for that.
I have seen that in efirebridsql_op.erl there is -define(CHARSET, "UTF8"). and you are using this CHARSET in the op_attach function for the isc_dpb_lc_ctype parameter. I hope it will be just a question of creating an extra connection parameter 'charset' that can be passed from Firebirdex to efirebirdsql and when this parameter is present to use it, and when it is not present to fall back to the default UTF8.
Do you see a way how I could help with my limited Erlang knowledge?

from firebirdex.

Yes, it looks like we'll have to modify efirebirdsql.
I'll think about it, but I don't think I can do it right away.
I'll wait until someone sends me a pull request ;-)

from firebirdex.

I'm a but puzzled, maybe you can help me out.
When I do a efirebirdsql:execute in an erlang shell I get the following data back:
[[{<<"LL_VOORNAAM">>,<<"GÜLCAN">>},
{<<"LL_NAAM">>,<<"YAZGÜN">>}]]

LL_VOORNAAM and LL_NAAM are the fieldnames for first name and last name, but as you can see they contain some non-ascii characters. But Erlang & efirebirdsql seem to interprete them correctly.

When I do the same query with Firebirdex.query in Elixir I get as result:
%{
"ll_naam" => <<89, 65, 90, 71, 220, 78>>,
"ll_voornaam" => <<71, 220, 76, 67, 65, 78>>
}
where Firebirdex does not give me back the first name and last name as a string.

Also when I use efirebirdsql from within Elixir I get the same:
{"LL_VOORNAAM", <<71, 220, 76, 67, 65, 78>>},
{"LL_NAAM", <<89, 65, 90, 71, 220, 78>>}

I don't understand what is going on. Can you help?

from firebirdex.

I think I understand what is happening.
I am trying to come up with a specific solution.

For example, if the driver itself converts codecs.
We would use https://hex.pm/packages/codepagex.
Is there a better library?

https://hex.pm/packages/iconv requires iconv to be installed, which I would like to avoid if possible

I would also like to know how to convert it in Firebird server, but it may not be possible in Firebird 2.5

from firebirdex.

I have used https://github.com/eugenehr/erlyconv before. It's an erlang library.

I don't understand however why efirebirdsql is interpreting the characters correctly, but not anymore when efirebirdsql passes the strings to Firebirdex.

I would also like to know how to convert it in Firebird server, but it may not be possible in Firebird 2.5
I thought that is what the isc_dpb_lc_ctype was doing when attaching to a database. Currently in efirebirdsql_op you are defaulting to UTF8. That is why I'm so puzzled by the fact I'm getting the non ascii characters correctly in Erlang from a WIN1252 encoded database using the default UTF8 charset.

from firebirdex.

Thanks

I think that the client should convert encode.

I looked here for consideration
https://elixirforum.com/t/sharing-with-the-community-text-transcoding-libraries/17962

Codepagex has too many features, but I thought erlyconv would be just right for this project.

I plan to first modify efirebirdsql to pass charset, then modify firebirdex to use erlyconv for charset conversion.

from firebirdex.

I found this post.
https://stackoverflow.com/questions/70548448/how-can-i-encode-decode-shift-jis-in-elixir

erlyconv can't handle cp932, so I'm going to use codepagex

from firebirdex.

I only checked with erlyconv cp1252, so if it does not support cp932 codepagex is the way to go.

from firebirdex.

@hermanius
I fixed it in the charset_param branch.
Please check if it works properly.

from firebirdex.

Do you need to create a new release tag, because a mix deps.get still downloads the old version 0.3.2?

from firebirdex.

published 0.3.3 and tagged
https://hex.pm/packages/firebirdex/0.3.3

from firebirdex.

hmm, strange. When trying to use 0.3.3 in a project I get the following error message when doing a query with charset: :cp1252 as a connection option.

[error] GenServer #PID<0.844.0> terminating
** (DBConnection.ConnectionError) client #PID<0.967.0> stopped: ** (Codepagex.Error) Unknown encoding :"VENDORS/MICSFT/WINDOWS/CP1252"
(codepagex 0.1.6) lib/codepagex.ex:519: Codepagex.from_string!/4
(firebirdex 0.3.3) lib/firebirdex/connection.ex:62: Firebirdex.Connection.handle_prepare/3
(db_connection 2.4.2) lib/db_connection/holder.ex:354: DBConnection.Holder.holder_apply/4
(db_connection 2.4.2) lib/db_connection.ex:1349: DBConnection.prepare/4
(db_connection 2.4.2) lib/db_connection.ex:1342: DBConnection.run_prepare/4
(db_connection 2.4.2) lib/db_connection.ex:1354: DBConnection.run_prepare_execute/5
(db_connection 2.4.2) lib/db_connection.ex:1459: DBConnection.run/6
(db_connection 2.4.2) lib/db_connection.ex:595: DBConnection.parsed_prepare_execute/5
(db_connection 2.4.2) lib/db_connection.ex:587: DBConnection.prepare_execute/4
(firebirdex 0.3.3) lib/firebirdex.ex:23: Firebirdex.query/4

from firebirdex.

#11 (comment)

from firebirdex.