Overview

The UniDic builder (https://github.com/ikawaha/kagome-dict/blob/main/uni/tool/main.go) produced an error when generating a binary file from the latest UniDic dictionary.

What I have tried

1. Downloaded the unidic-cwj-3.1.1-full.zip, unidic-cwj-2.2.0.zip, and unidic-mecab-2.1.2_src.zip from the following site:
   https://clrd.ninjal.ac.jp/unidic/back_number.html#unidic_bccwj
2. Tried to build a package from each of them like the following

❯ go run ./main.go -dict ../../../../../sandbox/tokenizer_dic_test/unidic-mecab-2.1.2_src -out ./dict/uni.dict_212
❯ go run ./main.go -dict ../../../../../sandbox/tokenizer_dic_test/unidic-cwj-2.2.0 -out ./dict/uni.dict_220
build failed: read error, "../../../../../sandbox/tokenizer_dic_test/unidic-cwj-2.2.0/lex_naist.csv", invalid format csv: ../../../../../sandbox/tokenizer_dic_test/unidic-cwj-2.2.0/lex_naist.csv, want col size 21, got 0, [/ 7062 8395 -1249 補助記号 一般 * * * * * / / * / * 記号 * * * * * * 補助 * * * * * * * 81022745596535296 294759]
exit status 1
❯ go run ./main.go -dict ../../../../../sandbox/tokenizer_dic_test/unidic-cwj-3.1.1-full -out ./dict/uni.dict_311
build failed: read error, "../../../../../sandbox/tokenizer_dic_test/unidic-cwj-3.1.1-full/lex_3_1.csv", invalid format csv: ../../../../../sandbox/tokenizer_dic_test/unidic-cwj-3.1.1-full/lex_3_1.csv, want col size 21, got 0, [/ 10643 4176 -1092 補助記号 一般 * * * * * / / * / * 記号 * * * * * * 補助 * * * * * * * 81022745596535296 294759]
exit status 1

2.1.2 can be successfully built, but newer versions are failed.
It seems that the format of lex.def has been changed since 2.2.0, which might be a reason for this issue. Please let me know If my operation needs to be corrected or if you need more information.

Reference

『現代日本語書き言葉均衡コーパス』形態論情報データベースの設計と実装 改訂版

Is the Go source code licensed by MIT?

see. 形態素解析の理論と実装 5.8.3

Hello, thank you for this nice package again!

I am currently trying to build bindings for this package (kagome with uniDict) for dart. This works great on Windows, MacOS and Android. However, I am having some problems on iOS.

I am compiling this library using CGO and then bundling it into a xcframework (here is the makefile). While this works, instantiating an instance of iPadic is extremely slow on my iPad Pro 2017. It takes smth between 5-10 minutes.
Specifically, I am creating a global Tokenizer and with an ipadDict instance code.

// global kagome tokenizer
var globalTokenizer tokenizer.Tokenizer

...

//export InitTokenizer
func InitTokenizer() {
	
	tokenizer, err := tokenizer.New(uni.Dict(), tokenizer.OmitBosEos())
	if err != nil {
		panic(err)
	}

	globalTokenizer = *tokenizer;
	globalTokenizerInitialized = true;
}

The weird thing is that on all other platforms it is really fast (< 2s). Do you have any idea what could be the problem?