[Paper] [Slides]

KWJA is a Japanese language analyzer based on pre-trained language models. KWJA performs many language analysis tasks, including:

• Typo correction
• Word segmentation
• Word normalization
• Morphological analysis
• Word feature tagging
• NER (Named Entity Recognition)
• Base phrase feature tagging
• Dependency parsing
• PAS analysis
• Bridging reference resolution
• Coreference resolution
• Discourse relation analysis

• Python: 3.8, 3.9, 3.10
• Dependencies: See pyproject.toml.

Install KWJA with pip:

$ pip install kwja

Perform language analysis with the kwja command (the result is in the KNP format):

# Analyze a text
$ kwja --text "KWJAは日本語の統合解析ツールです。汎用言語モデルを利用し、様々な言語解析を統一的な方法で解いています。"

# Analyze a text file and write the result to a file
$ kwja --file path/to/file.txt > path/to/analyzed.knp

# Analyze texts interactively
$ kwja
Please end your input with a new line and type "EOD"
KWJAは日本語の統合解析ツールです。汎用言語モデルを利用し、様々な言語解析を統一的な方法で解いています。
EOD

The output is in the KNP format, like the following:

# S-ID:202210010000-0-0 kwja:1.0.2
* 2D
+ 5D <rel type="=" target="ツール" sid="202210011918-0-0" id="5"/><体言><NE:ARTIFACT:KWJA>
KWJA ＫWＪＡ KWJA 名詞 6 固有名詞 3 * 0 * 0 <基本句-主辞>
は は は 助詞 9 副助詞 2 * 0 * 0 "代表表記:は/は" <代表表記:は/は>
* 2D
+ 2D <体言>
日本 にほん 日本 名詞 6 地名 4 * 0 * 0 "代表表記:日本/にほん 地名:国" <代表表記:日本/にほん><地名:国><基本句-主辞>
+ 4D <体言><係:ノ格>
語 ご 語 名詞 6 普通名詞 1 * 0 * 0 "代表表記:語/ご 漢字読み:音 カテゴリ:抽象物" <代表表記:語/ご><漢字読み:音><カテゴリ:抽象物><基本句-主辞>
の の の 助詞 9 接続助詞 3 * 0 * 0 "代表表記:の/の" <代表表記:の/の>
...

Here are some other options for kwja command:

--model-size: Model size to be used. Please specify 'tiny', 'base' (default) or 'large'.

--device: Device to be used. Please specify 'cpu' or 'gpu'.

--typo-batch-size: Batch size for typo module.

--seq2seq-batch-size: Batch size for seq2seq module.

--char-batch-size: Batch size for char module.

--word-batch-size: Batch size for word module.

--tasks: Tasks to be performed. Please specify 'typo', 'char', 'seq2seq', 'typo,char', 'typo,seq2seq', 'seq2seq,word', 'char,word', 'typo,char,word', 'typo,seq2seq,word', 'char,word,word_discourse', 'seq2seq,word,word_discourse', 'typo,char,word,word_discourse' or 'typo,char,word,word_discourse'.

• typo: Typo correction
• seq2seq: Word segmentation, Word normalization, Reading prediction, lemmatization, and Canonicalization.
• char: Word segmentation and Word normalization
• word: Morphological analysis, Named entity recognition, Word feature tagging, Dependency parsing, PAS analysis, Bridging reference resolution, and Coreference resolution
• word_discourse: Discourse relation analysis
  • If you need the results of discourse relation analysis, please specify this in addition to word.

You can read a KNP format file with rhoknp.

from rhoknp import Document
with open("analyzed.knp") as f:
    parsed_document = Document.from_knp(f.read())

For more details about KNP format, see Reference.

Make sure you have kwja command in your path:

$ which kwja
/path/to/kwja

Install rhoknp:

$ pip install rhoknp

Perform language analysis with the kwja instance:

from rhoknp import KWJA
kwja = KWJA()
analyzed_document = kwja.apply(
    "KWJAは日本語の統合解析ツールです。汎用言語モデルを利用し、様々な言語解析を統一的な方法で解いています。"
)

• The performance on each task except typo correction and discourse relation analysis is the mean over all the corpora (KC, KWDLC, Fuman, and WAC) and over three runs with different random seeds.
• We set the learning rate of RoBERTa_LARGE (word) to 2e-5 because we failed to fine-tune it with a higher learning rate. Other hyperparameters are the same described in configs, which are tuned for DeBERTa_BASE.

@InProceedings{植田2022,
  author    = {植田 暢大 and 大村 和正 and 児玉 貴志 and 清丸 寛一 and 村脇 有吾 and 河原 大輔 and 黒橋 禎夫},
  title     = {KWJA：汎用言語モデルに基づく日本語解析器},
  booktitle = {第253回自然言語処理研究会},
  year      = {2022},
  address   = {京都},
}

@InProceedings{児玉2023,
  author    = {児玉 貴志 and 植田 暢大 and 大村 和正 and 清丸 寛一 and 村脇 有吾 and 河原 大輔 and 黒橋 禎夫},
  title     = {テキスト生成モデルによる日本語形態素解析},
  booktitle = {言語処理学会 第29回年次大会},
  year      = {2023},
  address   = {沖縄},
}

• KNP format