isutf8 is a program and a c library to check if a given file (or stdin) contains only valid utf-8 sequences.
$ make
$ isutf8 * -v
isutf8: line 1, char 97, byte 96: Expecting bytes in the following ranges: 00..7F C2..F4.
40 00 40 00 00 00 00 00 C0 01 00 00 00 00 00 00 | @.@.............
^^ | ^
is_utf8.o: line 1, char 66, byte 65: Expecting bytes in the following ranges: 00..7F C2..F4.
00 40 00 0E 00 0B 00 48 85 F6 48 C7 02 00 00 00 | [email protected].....
^^ | ^
libisutf8.so: line 1, char 153, byte 152: After a first byte of F0, expecting 2nd byte between 90 and BF.
68 12 20 00 00 00 00 00 F0 01 00 00 00 00 00 00 | h. .............
^^^^^ | ^^
main.o: line 1, char 76, byte 75: Expecting bytes in the following ranges: 00..7F C2..F4.
56 41 55 41 54 55 53 48 83 EC 18 48 8B 5C 24 58 | VAUATUSH...H.\$X
^^ | ^
isutf8 returns 0 if the file is correctly encoded:
$ isutf8 main.c
$ echo $?
0
Some files here only contain ASCII or correctly encoded UTF8:
$ isutf8 README.md
$ isutf8 test.sh
But an ELF is clearly not UTF8, a verbose error is printed:
$ isutf8 isutf8
isutf8: line 1, char 97, byte 96: Expecting bytes in the following ranges: 00..7F C2..F4.
-v adds some context:
$ isutf8 -v isutf8
isutf8: line 1, char 97, byte 96: Expecting bytes in the following ranges: 00..7F C2..F4.
40 00 40 00 00 00 00 00 C0 01 00 00 00 00 00 00 | @.@.............
^^ | ^
isutf8 reads on stdin if no file are given, also note that bash
helps a lot with the $'' syntax allowing you to write and test hexadecimal:
$ echo $'\xe9' | isutf8
(standard input): line 1, char 0, byte 0: After a first byte between E1 and EC, expecting two following bytes.
$ echo "Hellö world" | iconv -f utf8 -t latin1 | isutf8
(standard input): line 1, char 4, byte 4: Expecting bytes in the following ranges: 00..7F C2..F4.
As isutf8 can take multiple arguments it's easy classify
UTF8-compatible versus non UTF8-compatible files:
List non-UTF8 compatible files:
$ isutf8 --list *
isutf8
is_utf8.o
libisutf8.so
main.o
List UTF8-compatible files:
$ isutf8 --list --invert *
COPYRIGHT
is_utf8.c
is_utf8.h
main.c
Makefile
README.md
test.sh
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent