unicode support? about ksmbd HOT 14 CLOSED

Maybe, You should change Default NLS option to utf8 in kernel config to get hangul ability.
Have you ever tried to change it ?

from ksmbd.

ok will try this, i just made sure the utf8 kernel module is build and present, was assuming it would be loaded and used as needed.

from ksmbd.

Just tested this on a Windows computer. Windows shows the cifsd files as utf8, which is cannot handle. Screenshot attached for an example.

The filenames must be converted to UTF-16 to work properly (and no, accessing them does not work).

edit: tested in Linux. If mounted with defaults, same behaviour. If mounted with iocharset=utf8, it works properly.

The openwrt device's kernel config has CONFIG_NLS_DEFAULT="utf-8"

from ksmbd.

Turns out utf-8 should be utf8. Now it works properly on both Windows and Linux.

from ksmbd.

Yes, DEFAULT_NLS should be utf8. Maybe we need to improve this.

-ss

from ksmbd.

[1-26.7588] nls->charset : macturkish, charset : utf-8
[1-26.7588] nls->charset : macroman, charset : utf-8
[1-26.7627] nls->charset : macromanian, charset : utf-8
[1-26.7677] nls->charset : macinuit, charset : utf-8
[1-26.7724] nls->charset : maciceland, charset : utf-8
[1-26.7773] nls->charset : macgreek, charset : utf-8
[1-26.7820] nls->charset : macgaelic, charset : utf-8
[1-26.7868] nls->charset : maccyrillic, charset : utf-8
[1-26.7917] nls->charset : maccroatian, charset : utf-8
[1-26.7967] nls->charset : maccenteuro, charset : utf-8
[1-26.8017] nls->charset : macceltic, charset : utf-8
[1-26.8064] nls->charset : utf8, charset : utf-8
[1-26.8108] nls->charset : koi8-ru, charset : utf-8
[1-26.8154] nls->charset : koi8-u, charset : utf-8
[1-26.8199] nls->charset : koi8-r, charset : utf-8
[1-26.8245] nls->charset : iso8859-15, charset : utf-8
[1-26.8293] nls->charset : iso8859-14, charset : utf-8
[1-26.8342] nls->charset : iso8859-13, charset : utf-8
[1-26.8391] nls->charset : iso8859-9, charset : utf-8
[1-26.8439] nls->charset : cp1255, charset : utf-8
[1-26.8484] nls->charset : iso8859-7, charset : utf-8
[1-26.8532] nls->charset : iso8859-6, charset : utf-8
[1-26.8580] nls->charset : iso8859-5, charset : utf-8
[1-26.8628] nls->charset : iso8859-4, charset : utf-8
[1-26.8676] nls->charset : iso8859-3, charset : utf-8
[1-26.8724] nls->charset : iso8859-2, charset : utf-8
[1-26.8772] nls->charset : iso8859-1, charset : utf-8
[1-26.8820] nls->charset : ascii, charset : utf-8
[1-26.8864] nls->charset : cp1251, charset : utf-8
[1-26.8909] nls->charset : cp1250, charset : utf-8
[1-26.8955] nls->charset : cp950, charset : utf-8
[1-26.8999] nls->charset : cp949, charset : utf-8
[1-26.9043] nls->charset : cp936, charset : utf-8
[1-26.9088] nls->charset : euc-jp, charset : utf-8
[1-26.9133] nls->charset : cp932, charset : utf-8
[1-26.9177] nls->charset : cp874, charset : utf-8
[1-26.9222] nls->charset : cp869, charset : utf-8
[1-26.9266] nls->charset : cp866, charset : utf-8
[1-26.9311] nls->charset : cp865, charset : utf-8
[1-26.9355] nls->charset : cp864, charset : utf-8
[1-26.9400] nls->charset : cp863, charset : utf-8
[1-26.9444] nls->charset : cp862, charset : utf-8
[1-26.9488] nls->charset : cp861, charset : utf-8
[1-26.9533] nls->charset : cp860, charset : utf-8
[1-26.9577] nls->charset : cp857, charset : utf-8
[1-26.9622] nls->charset : cp855, charset : utf-8
[1-26.9666] nls->charset : cp852, charset : utf-8
[1-26.9710] nls->charset : cp850, charset : utf-8
[1-26.9755] nls->charset : cp775, charset : utf-8
[1-26.9799] nls->charset : cp737, charset : utf-8
[1-26.9844] nls->charset : cp437, charset : utf-8
[1-26.9888] nls->charset : default, charset : utf-8
[3-27.0047] nls->charset : macturkish, charset : utf-8
[3-27.0048] nls->charset : macroman, charset : utf-8
[3-27.0087] nls->charset : macromanian, charset : utf-8
[3-27.0136] nls->charset : macinuit, charset : utf-8
[3-27.0183] nls->charset : maciceland, charset : utf-8
[3-27.0232] nls->charset : macgreek, charset : utf-8
[3-27.0279] nls->charset : macgaelic, charset : utf-8
[3-27.0327] nls->charset : maccyrillic, charset : utf-8
[3-27.0377] nls->charset : maccroatian, charset : utf-8
[3-27.0426] nls->charset : maccenteuro, charset : utf-8
[3-27.0476] nls->charset : macceltic, charset : utf-8
[3-27.0524] nls->charset : utf8, charset : utf-8
[3-27.0567] nls->charset : koi8-ru, charset : utf-8
[3-27.0614] nls->charset : koi8-u, charset : utf-8
[3-27.0659] nls->charset : koi8-r, charset : utf-8
[3-27.0704] nls->charset : iso8859-15, charset : utf-8
[3-27.0753] nls->charset : iso8859-14, charset : utf-8
[3-27.0802] nls->charset : iso8859-13, charset : utf-8
[3-27.0851] nls->charset : iso8859-9, charset : utf-8
[3-27.0898] nls->charset : cp1255, charset : utf-8
[3-27.0944] nls->charset : iso8859-7, charset : utf-8
[3-27.0992] nls->charset : iso8859-6, charset : utf-8
[3-27.1040] nls->charset : iso8859-5, charset : utf-8
[3-27.1088] nls->charset : iso8859-4, charset : utf-8
[3-27.1135] nls->charset : iso8859-3, charset : utf-8
[3-27.1183] nls->charset : iso8859-2, charset : utf-8
[3-27.1231] nls->charset : iso8859-1, charset : utf-8
[3-27.1279] nls->charset : ascii, charset : utf-8
[3-27.1324] nls->charset : cp1251, charset : utf-8
[3-27.1369] nls->charset : cp1250, charset : utf-8
[3-27.1414] nls->charset : cp950, charset : utf-8
[3-27.1458] nls->charset : cp949, charset : utf-8
[3-27.1503] nls->charset : cp936, charset : utf-8
[3-27.1547] nls->charset : euc-jp, charset : utf-8
[3-27.1593] nls->charset : cp932, charset : utf-8
[3-27.1637] nls->charset : cp874, charset : utf-8
[3-27.1681] nls->charset : cp869, charset : utf-8
[3-27.1726] nls->charset : cp866, charset : utf-8
[3-27.1770] nls->charset : cp865, charset : utf-8
[3-27.1815] nls->charset : cp864, charset : utf-8
[3-27.1859] nls->charset : cp863, charset : utf-8
[3-27.1904] nls->charset : cp862, charset : utf-8
[3-27.1948] nls->charset : cp861, charset : utf-8
[3-27.1992] nls->charset : cp860, charset : utf-8
[3-27.2037] nls->charset : cp857, charset : utf-8
[3-27.2081] nls->charset : cp855, charset : utf-8
[3-27.2126] nls->charset : cp852, charset : utf-8
[3-27.2170] nls->charset : cp850, charset : utf-8
[3-27.2214] nls->charset : cp775, charset : utf-8
[3-27.2259] nls->charset : cp737, charset : utf-8
[3-27.2303] nls->charset : cp437, charset : utf-8
[3-27.2348] nls->charset : default, charset : utf-8

These are nls table lookup messages. I can not see "utf-8" charset in kernel nls table.
I think that only utf8 charset exists in linux kernel. and I think that it is typo if openWRT kernel config has utf-8 as default config. So it is correct to change CONFIG_NLS_DEFAULT="utf8" in your kernel config.
Thanks!

from ksmbd.

It's iso8859-1 by default. utf-8 was a typo on my part. It will probably be changed to utf8 by default once cifsd gets merged.

from ksmbd.

rsync: mkstemp "/home/mangix/test/AccurateRip/Pop/2NE1/2nd Mini Album/.01 - 내가 제일 잘 나가 (I Am the Best).flac.YFQzKt" failed: No such file or directory (2)
kcifsd: check_invalid_char:143: found invalid character : 0x3f
Okay, I see. problem filename was hangul, So should use utf8 in kernel.
If you fix it, can close this issue now ?
Thanks for your feedback!

from ksmbd.

Hm.. I have a idea. I will try to add new patch to use utf8 in only cifsd regardless of your default nls(iso8859-1).

from ksmbd.

ok can try the next set of changes, since german special chars (aäöü) have the same problem on Windows 10 they are incorrectly displayed and folders are not accessible.

from ksmbd.

Note that samba has a "unix charset" setting for this.

from ksmbd.

Yep, I checked it. So tried to set utf8 at default in cifsd. I pushed the patch to use utf8 for cifsd regardless of default kernel nls config. please check updated master branch.
Thanks for your feedback!

from ksmbd.

@namjaejeon @neheb ok the switch to utf8 worked for me and now german umlauts are displayed correctly and i can enter the folders. I assume this also fixed neheb's problem?

from ksmbd.

Yup!

from ksmbd.