pythainlp / docker-thai-tokenizers Goto Github PK

Chrome has an API for tokenising text. This feature also includes Thai. Its evaluation result shows a promising performance both speed and quality. In brief, its speed is comparable to PyThaiNLP's newmm, but its quality is significantly better. This could be another good baseline.

Using Puppeteer, we can have this vendor through CLI. I've made the first version of the program: https://gist.github.com/heytitle/02b818b02114644152bd0317e62750a7.

Based on this discussion from PyThaiNLP/tokenization-benchmark#8.

It would be good if we also benchmark performance of each tokeniser on other aspects. These aspects include:

Speed

Characters per second (on standardized machine)
May be tested with different sizes of text (small and large), to notice the "boot time" of a tokenizer

Memory footprint

Memory used by the tokenizer (when tokenizing a certain amount of text), at the running time

Disk size

Total size of the tokenizer, including dictionary, models, and all non-standard dependencies (excluding runtime environment, like interpreter/VM)

It seems Docker CI isn't working properly. In fact, we probably have to build images manually via ./scripts/build.sh. This script will append the main function to the vendor file and we must keep also the copy commands like this

COPY vendor.py .
COPY entry .

cc: @wannaphong

I think that I will add kucut.

GitHub: https://github.com/Thanabhat/KUCut

https://github.com/KenjiroAI/SynThai

https://github.com/earthy123/Multi-Candidate-Word-Segmentation