Why not use rayon? about tokenizers HOT 6 CLOSED

Sounds good! Thanks for the prompt responses on this, I find this work very impressive, and will spread the word.

from tokenizers.

We are actually using rayon to encode multiple sequences in parallel, see https://github.com/huggingface/tokenizers/blob/master/tokenizers/src/tokenizer/mod.rs#L357.

Otherwise, we don't make any assumption on the type of content that we are going to process. According to your choice of PreTokenizer and Model, there isn't necessarily a concept of word, and thus parallelization at this level isn't really possible, at least not trivially.

from tokenizers.

Thanks for the response! I am new to NLP, so I did not know that this concept is not generic. Also I had missed this call to par_iter(), so thanks for pointing that out.

On an unrelated note, could you please point me to a time/throughput comparison with WordPiece tokenizer in python that is used for the BERT model? It would help me push the case for switching to this tokenizer at my workplace.

from tokenizers.

Hey @smr97, you can have a look at this file: https://github.com/huggingface/tokenizers/blob/master/bindings/python/examples/example.py
It provides an example of processing between the Python tokenizer which is available in transformers and the one provided by this library. You can process a large enough file, like one of the wiki-text-raw to see the difference.
If you need to download a vocabulary for BERT, all the links are available here: https://github.com/huggingface/transformers/blob/master/src/transformers/tokenization_bert.py#L32

from tokenizers.

Thanks for the useful example, that is exactly what I wanted. I ran this with the wiki-text-raw-train file (541MB) and saw a pretty amazing speedup (MacBook 2017)!

Just FYI though, there seems to be a bug in the script, it throws this error as shown in the figure. I checked the rust source and it seems that the decode_batch function indeed has two positional arguments (so the error is un-called for). I have not worked with python bindings to Rust, so am not sure where the bug really is.

from tokenizers.

@smr97 It should be fixed on master now :)

from tokenizers.