docker-thai-tokenizers's Introduction

Thai Word Tokenizers

This repository is a collection of almost all Thai tokenisers that are publicly available. Having this collection allows us to try each algorithm as ease via Docker.

Technically, each project (called vendor) has its own Docker image with a entry script and auxiliary scripts. These scripts bring a unified interface, allowing us to run those algorithms in the same way.

Vendors

Vendor	Alias	Available Methods
PyThaiNLP	pythainlp	newmm, longest
DeepCut	deepcut	deepcut
CutKum	cutkum	cutkum
Sertis	sertis	sertis
Thai Language Toolkit	tltk	mm, ngram, colloc
Smart Word Analysis for Thai (SWATH)	swath	max, long
Chrome's v8Breakiterator	chrome	v8breakiterator

Please see Usages for more details.

Setup

Pull necessary Docker images. Please check Docker Hub for the avaliable images.
```
$ docker pull pythainlp/word-tokenizers:<vendor-alias>
```

Usages

Put text files that you want to tokenise into ./data.
Run the following command ...

$ ./scripts/tokenise.sh <vendor-alias>-<method> <**filename**>

Please check Vendors section for vendors and methods included here.

Example

Let's say you want to tokenise text in ./data/example.text using PyThaiNLP's newmm algorithm. You can use the following command:

$ cat ./data/example.text
อันนี้คือตัวอย่าง

$ ./scripts/tokenise.sh pythainlp:newmm example.text
# Please be aware that you don't need to have ./data in front of the filename.
# Command Output
Tokenising example.text using vendor=pythainlp and method=newmm
CMD: docker run -v /Users/heytitle/projects/tokenisers-for-thai/data:/data  thai-tokeniser:pythainlp newmm example.text
100%|██████████| 1/1 [00:00<00:00, 151.70it/s]
Tokenising /data/example.text with newmm
Tokenised text is written to /data/example_tokenised-pythainlp-newmm.text

$ cat ./data/example_tokenised-pythainlp-newmm.text
อันนี้|คือ|ตัวอย่าง

Development

Architecture

TBD.

Build a vendor's new Docker image

$ ./scripts/build <vendor>

Push a new Docker image to Docker Hub

$ ./scripts/push <vendor>

Acknowledgements

This repository was initially done by Pattarawat Chormai, whiling interning at Dr. Attapol Thamrongrattanarit's NLP Lab, Chulalongkorn University, Bangkok, Thailand.

docker-thai-tokenizers's People

Contributors

Stargazers

Watchers

docker-thai-tokenizers's Issues

Current CI build isn't working

It seems Docker CI isn't working properly. In fact, we probably have to build images manually via ./scripts/build.sh. This script will append the main function to the vendor file and we must keep also the copy commands like this

COPY vendor.py .
COPY entry .

cc: @wannaphong

New vendor: Multi-Candidate-Word-Segmentation (MCWS)

https://github.com/earthy123/Multi-Candidate-Word-Segmentation

New vendor: KUCut

I think that I will add kucut.

GitHub: https://github.com/Thanabhat/KUCut

Implement performance metrics

Based on this discussion from PyThaiNLP/tokenization-benchmark#8.

It would be good if we also benchmark performance of each tokeniser on other aspects. These aspects include:

Speed

Characters per second (on standardized machine)
May be tested with different sizes of text (small and large), to notice the "boot time" of a tokenizer

Memory footprint

Memory used by the tokenizer (when tokenizing a certain amount of text), at the running time

Disk size

Total size of the tokenizer, including dictionary, models, and all non-standard dependencies (excluding runtime environment, like interpreter/VM)

New vendor: SynThai

https://github.com/KenjiroAI/SynThai

Make a Docker image for Chrome's v8BreakIterator

Chrome has an API for tokenising text. This feature also includes Thai. Its evaluation result shows a promising performance both speed and quality. In brief, its speed is comparable to PyThaiNLP's newmm, but its quality is significantly better. This could be another good baseline.

Using Puppeteer, we can have this vendor through CLI. I've made the first version of the program: https://gist.github.com/heytitle/02b818b02114644152bd0317e62750a7.

Recommend Projects

pythainlp / docker-thai-tokenizers Goto Github PK