Hungarian dictionary has wrong character encoding about dictionaries HOT 7 CLOSED

Do you know the encoding? The dic file seems to be UTF8, but I’m not Hungarian so I can’t assert the aff file 🤷‍♂️ Could you help?

from dictionaries.

Yes, the encoding is UTF-8 on the files, my original post wasn't clear. I should've written, that somewhere in the encoding process, something was messed up, and the characters were encoded into UTF-8, while still being wrong.

I'm not sure what could be done, other than manually going in and correcting every line, with the proper characters. The problem with that, is that not even I'm sure what's supposed to be correct, even though I'm native Hungarian, because the characters are wrong, I'm not always sure what word is supposed to be there.

It's stated with the hun dictionary, that you used magyarispell. Maybe your code could be re-generated from that. I've checked out magyarispell, the characters look fine in that project. The Hungarian language uses the ISO-8859-2 encoding, sometimes called as Latin-2, (or Windows 1250, if using windows).

But I don't know what the code generation process was in your project, because magyarispell uses a directory structure by topics and files with different grammar usages. But maybe during generation, the initial files were read with the wrong encoding, and then using the wrong characters, it was encoded into UTF-8.

from dictionaries.

Oh I got that, sorry! You can clone the repo and do npm test, that should provide the files in source/hungarian/. The tarbal is from here. The script that encodes to UTF8 is here. It sets both to ISO8859-2, but that doesn’t seem to work.
I locally tried UTF-8 for the dic file, and ISO8859-2 for the aff file, but it still seems wrong? 🤔

from dictionaries.

@alexaka1 Could you look into the encoding issue as layout out in my previous comment? I don’t know Hungarian so I can’t do it myself.

from dictionaries.

@alexaka1 Ping!

from dictionaries.

Sorry I didn't answer sooner. Partly because I was busy with school and work, and also because I gave up on spellchecking.

If you've tried it out, and didn't work again, then I really don't know what's going on. I'm not familiar with how char encoding works, just when to stop that something is wrong. Maybe the OS locale messes it up, even though it shouldn't, but I don't know.

I'll have a look at it though when I have the time.

from dictionaries.

Don’t worry, school’s more important 😉

I’ll close this for now. I can’t fix it, I need someone who knows Hungarian to come in. It would be awesome if you’d be up for it tho, or someone else!

from dictionaries.