mecab_controller outputs garbled characters for some kanji.
For the expression "粗末な家に住んでいる" in the examples, I got the following, debugger at basic_mecab_controller.py:110
:
(Pdb) outs
b'\xe7\xb2<ajt__component_separator>\xe7\xb2<ajt__component_separator>\xa5\xd2\xa5\xab\xa5\xac\xa5\xdf<ajt__component_separator>\xcc\xbe\xbb\xec<ajt__component_separator><ajt__node_separator>\x97<ajt__node_separator>\xe6\x9c\xab<ajt__node_separator>\xe3\x81\xaa<ajt__node_separator>\xe5\xae\xb6<ajt__node_separator>\xe3\x81\xab<ajt__node_separator>\xe4\xbd\x8f<ajt__node_separator>\xe3\x82\x93\xe3\x81\xa7\xe3\x81\x84\xe3\x82\x8b<ajt__node_separator><ajt__footer>'
(Pdb) outs.rstrip(b'\r\n').decode('utf-8', 'replace')
'�<ajt__component_separator>�<ajt__component_separator>�ҥ�����<ajt__component_separator>̾��<ajt__component_separator><ajt__node_separator>�<ajt__node_separator>末<ajt__node_separator>な<ajt__node_separator>家<ajt__node_separator>に<ajt__node_separator>住<ajt__node_separator>んでいる<ajt__node_separator><ajt__footer>'
echo '粗末な家に住んでいる' | /opt/homebrew/bin/mecab --dicdir=/opt/homebrew/lib/mecab/dic/ipadic --rcfile=/Users/MY_PATH_TO_MECAB_CONTROLLER/mecab_controller/support/mecabrc --userdic=/Users/MY_PATH_TO_MECAB_CONTROLLER/mecab_controller/support/user_dic.dic --input-buffer-size=819200
̾,,*,*,*,*,,ҥ,ҥ
,,*,*,*,*,*
末 ̾,̾,ȿ,*,*,*,*
な ̾,,*,*,*,*,*
家 ̾,,*,*,*,*,*
に ̾,,*,*,*,*,*
住 ̾,,*,*,*,*,*
んでいる ̾,,*,*,*,*,*
EOS
I'm not too familiar with mecab, is this how it is supposed to output if it can't parse a token?
If so, the section
gets split into components
at mecab_controller.py:57
, the value ends up ['�', '�', '�ҥ�����', '̾��', '']
. Should this be handled in this error block?