Convert between ordinary UTF-8 and CESU-8 (Wikipedia, Unicode Consortium) encodings.
CESU-8 encodes characters outside the Basic Multilingual Plane as two UTF-16 surrogate chacaters, which are then further re-encoded as invalid, 3-byte UTF-8 characters. This means that 4-byte UTF-8 sequences become 6-byte CESU-8 sequences.
Note that CESU-8 is only intended for internal use within tightly-coupled systems, and not for data interchange.
This encoding is sometimes needed when working with Java, Oracle or MySQL, and when trying to store emoji, hieroglyphs, or other characters on the Supplementary Multilingual Plane or the Supplementary Ideographic Plane.
We also support Java's Modified UTF-8 encoding, which additionally encodes \0
using a multi-byte UTF-8 sequence.
Installing:
cargo install cesu8str --features=build-binary
encode/decode between MUTF8/CESU8 and UTF8
$ cesu8str --help
Converts files or standard IO streams between standard UTF8 and CESU8, or the JVM's modified UTF-8.
Note that the default Windows' console does not support non-UTF8 sequences - attempting to type/print them will result in
an error.
This tool will immediately exit upon finding an invalid character sequence.
EXIT CODES:
0 - if completed normally
1 - if an IO error has occured
2 - if an encoding error has occured (invalid/incomplete character sequences/etc)
Usage: cesu8str [OPTIONS] [INPUT] [OUTPUT]
Arguments:
[INPUT] The input file. Defaults to stdin if '-' or not set
[OUTPUT] The output file. Defaults to stdout if '-' or not set
Options:
-j, --java Toggles the use of the JVM's modified UTF8. In effect, it encodes nul-bytes as 0xC0,0x80 while nul-bytes
are left alone in normal mode
-d, --decode Decodes CESU8 text into standard UTF8. By default, this tool encodes UTF8 to CESU8
-h, --help Print help information
-V, --version Print version information
More of a utility tool for verifying functionality during development.
$ count_codepoints --help
USAGE: count_codepoints [--json] [file] [KB_chunk_size=4]
Reads a file and emits the number of occurences differing lengths of UTF-8 codepoints on stdout.
If `file` is not supplied, stdin will be read.
Exit Codes:
0 Success
1 Command-line error
2 IO Error
$ count_codepoints ./test_files/random.txt
len4: 1001
len3: 1013
len2: 966
ascii: 1095
nul: 0
total (chars/bytes): 4075/10070
$ count_codepoints ./test_files/random.cesu8.txt
len4: 0
len3: 3015
len2: 966
ascii: 1095
nul: 0
total (chars/bytes): 5076/1207
Some of this code is adapted from Rust's src/libcore/str.rs file.
This code is covered by LICENSE-RUST.txt and copyright by The Rust Project
Developers and individual Rust contributors, as described in that file.
Most code in this project was forked, and adapted from, Eric Kidd's cesu8 crate. That adapted code is distributed under the same terms as the original project at time of fork.
The new code in this project is distributed under the same terms.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent