The unidic_lite dictionary is not installed about fugashi HOT 5 CLOSED

@polm could you please chime in

from fugashi.

I'm not sure I understand your problem entirely, but it sounds like you want to download the UniDic dictionary from code instead of the command line. You can do that like this:

from unidic.download import download_version
download_version()

The command line download command is just a wrapper for that function.

from fugashi.

I'm not sure I understand your problem entirely, but it sounds like you want to download the UniDic dictionary from code instead of the command line. You can do that like this:

from unidic.download import download_version
download_version()

The command line download command is just a wrapper for that function.

Thank you @polm!

But I am still getting the error "The unidic_lite dictionary is not installed".
My model is making a call to the class BertJapaneseTokenizer. (I would request you to take a look at the source code here. It will greatly facilitate my explanation: [https://huggingface.co/transformers/v4.11.3/_modules/transformers/models/bert_japanese/tokenization_bert_japanese.html]

You will see that on following the trail of function calls in reverse, it looks like that error can occur only when the variable mecab_dic is equal to = 'unidic_lite' (mecab_dic is an attribute of the class MecabTokenizer that you will also find in the link above).

My question is can I override this value of mecab_dic and change it to "unidic" instead of the current "unidic_lite"

That way I will not get the error.

from fugashi.

I looked at the code that you linked to but I'm not sure what to tell you - I didn't write that code and it's not part of this library, so I have no control over it. Your code (which you haven't shared) or the HuggingFace code is setting that value somewhere.

I'm also not sure why you think you can't use unidic-lite. Can you explain that or try it?

from fugashi.

Closing for lack of response / because this issue isn't relevant to fugashi directly.

For the record, if you want help with this you should show the code where you're using BertJapaneseTokenizer and explain what you're actually trying to do. It sounds like you have a usage question about the HuggingFace code, but with just the information you've given there's no way to tell what's going on.

from fugashi.