Add full support for UTF-16 / Unicode about regex-engine HOT 3 OPEN

PureBasic's string functions use UCS-2 encoding in Unicode mode according to the official documentation. But PureBasic uses the API functions of the operating systems for displaying the strings and these all (Windows, Linux and macOS) interpret the PureBasic string as UTF-16, so programs written in PureBasic can display all Unicode characters.

I'm not quite sure about that, because internally PB will treat all strings as UCS-2 (i.e. 16-bit fixed-width encoding) so if you pass to PB a string in UTF-16 that encodes beyond the range supported by UCS-2 then PB is most likely going to corrupt it since it doesn't cater for variable-width encoding (like UTF-16 does). The problem would affect any string manipulation command, but possibly also under-the-hood string processing.

The whole reason why PB went for UCS-2 is to keep things simple, i.e. have a fixed correspondence of one-character = two-bytes — an assumption that doesn't hold true in UTF-16. This might affect correct counting of characters in string when the UTF-16 string contains a 32-bit Unicode character.

So, although it's true that any UCS-2 string will be correctly handled by OS functions that accept UTF-16 strings, the reverse might not be true — i.e. that "PureBasic can display all Unicode characters".

If that's an important assumption, it might be better to test it out first.

from regex-engine.

I'm not quite sure about that, because internally PB will treat all strings as UCS-2 (i.e. 16-bit fixed-width encoding) so if you pass to PB a string in UTF-16 that encodes beyond the range supported by UCS-2 then PB is most likely going to corrupt it since it doesn't cater for variable-width encoding (like UTF-16 does). The problem would affect any string manipulation command, but possibly also under-the-hood string processing.

I am aware of that. By "PureBasic can display all Unicode characters" I meant that the string is not manipulated beforehand.

Yes, often you want to manipulate a string before displaying it, e.g. you want to display only a part of the string, so your objection is valid. But then the programmer still has the option to create his own string manipulation functions that can correctly handle UTF-16 surrogate pairs. With these functions he can then also partially bring UTF-16 support into the RegEx engine, but he would fail with the RegEx character classes (e.g. \w) or with the case-insensitive mode. Therefore, I think it would be better if the RegEx engine already supports UTF-16 by itself, even if the PureBasic string manipulation functions have problems with it.

If that's an important assumption, it might be better to test it out first.

There are already tests confirming that UTF-16 surrogate pairs are treated as two valid UCS-2 characters by PureBasic string functions and that PB display the UTF-16 surrogate pair as a single Unicode character (tested on Windows, Linux, macOS):

• https://www.purebasic.fr/english/viewtopic.php?p=569391#p569391

from regex-engine.

There are already tests confirming that UTF-16 surrogate pairs are treated as two valid UCS-2 characters by PureBasic string functions and that PB display the UTF-16 surrogate pair as a single Unicode character (tested on Windows, Linux, macOS):

Thanks, that link was very helpful. I always wanted to carry out similar tests myself, to ascertain first hand how the UCS2 vs UTF-16 encoding actually works, but never got to doing it.

I guess the flip side of the coin is that you never known for sure when and if strings are being manipulated behind the scene in the final program, since many internal commands might be carrying out under the hood stuff that might compromise an UTF-16 string.

But then, if Unicode characters beyond UCS2 can be represented via escape sequence representations then it should be possible to correctly handle them via custom procedures, even in the RegEx engine, since PB would treat the escape as a plain text sequence of ASCII chars.

from regex-engine.