e-mail and URL tokenization about mecab-ipadic-neologd HOT 1 CLOSED

Thank you for the practical request.

In conclusion, the best way to solve the problem you pointed out is by pre- and post-processing, not by the morphological analysis process itself.
We think it's easier to control the splitting performance.

Since there are too many possibilities for notation of e-mail addresses and URLs, all patterns cannot be recorded in the dictionary in the developing phase.

If you want to resolve the problem by preprocessing, it is convenient to replace the email address or URL with an unknown word and restore it after parsing.

If the solution is post-processing, the morphological analysis results should be chained with CRF e.t.c.

These solutions are very common and are found in various textbooks of natural language processing.

If you don't need phonetic characters for Japanese processing, you might want to use Juman++ or Nagisa.

Thank you very much.

from mecab-ipadic-neologd.