cls / libutf Goto Github PK

The following function definitions are missing from utf.h:

char *utfecpy(char *s1, char *es1, char *s2);
Rune *runestrecpy(Rune *s1, Rune *es1, Rune *s2);

This is as per #3, neither commits referenced in that issue add the relevant definitions to utf.h.

tl;dr - The behavior of isalpharune treats CJK Ideographs as non alpha.

CJK ideographs are specified in the UnicodeData.txt (9.0.0) as:

12018 4E00;<CJK Ideograph, First>;Lo;0;L;;;;;N;;;;;
12019 9FD5;<CJK Ideograph, Last>;Lo;0;L;;;;;N;;;;;

However, the result of the generation (via awk) marks these as singles. This means that any query of a rune in that range (other than these two) will return false to isalpharune().

My best guess is that the awk script wants ranges to have hex codes in the same range and these have nothing in common there. (4E00 vs 9FD5)

There should be a test suite to check for UTF-8 corner cases, to replace the current utftest executable, which can only check a couple of the functions in Libutf.

Libutf should provide the following functions, which are required for compatibility with Plan 9's libutf:

char *utfecpy(char *s1, char *es1, char *s2);
Rune *runestrecpy(Rune *s1, Rune *es1, Rune *s2);