README example tokenization seems off about sudachipy HOT 8 CLOSED

@polm I found good example 国家公務員 ! I'll update soon.

from sudachipy.

This all connected result is from full-dic.

from sudachipy.

Oh wow, I completely missed the comment explaining that - sorry for the false alarm!

Would it be better to use a string where the tokenization is the same with the full and core dict?

from sudachipy.

I appreciate any concern, thank you.
I agree almost. Actually one user (you) confused.

• mode A,B,C return different tokenized results
• all dic return same tokenized patterns with mode

is the best example.

But I think core dic example is enough if different mode returns different pattern because core dic is default dic.

from sudachipy.

Hi, for non-Japanese speakers, split modes are a bit confusing. Could you elaborate on what they correspond to? :)
Also, how can we get an output similar to mecab one from command line (mecab -O wakati input_file -o output_file returns a file with tokenized strings)?
I want to get rid of NER, etc from the output file obtained from sudachipy tokenize input_file -o output_file

Thank you!

from sudachipy.

@MastafaF

The split modes are invented and defined by us, and there isn't really anything that each mode corresponds to, but in our paper, "Sudachi: a Japanese Tokenizer for Business" (Takaoka et al., LREC2018), we state that

• A Unit (short): Compatible with UniDic
• B Unit (middle): Similar to the “words” in general sense
• C Unit (NE): Named entities

from sudachipy.

@MastafaF

Getting only the tokenized words without Part of Speech tags; Use SudachiPy as a Python library and call surface() method.

e.g.,

>>> from sudachipy import dictionary
>>> tokenizer_obj = dictionary.Dictionary().create()
>>> [m.surface() for m in tokenizer_obj.tokenize("これはテストの文です。")]
['これ', 'は', 'テスト', 'の', '文', 'です', '。']

from sudachipy.

@MastafaF

Also, your questions are not related to this issue #70; it would be nice if you could create a new issue for the questions next time. Thanks!

from sudachipy.