초보자가 'KiwiTokenizer' 한번 사용해볼려고 제시한 예제대로 해봤는데 다음과 같은 에러가 나네요. 그리고 KiwiTokenizer가 들어간 kiwi-farm에서 제시한 예제 코드나 그 외 다른 코드라던지 전부 'KiwiTokenizer' 때문에 에러가 발생하던데요?(혹시 잘못 알고 있다면 지적 바랍니다)

import kiwipiepy
from kiwipiepy import KiwiTokenizer

ImportError Traceback (most recent call last)
Cell In[3], line 2
1 import kiwipiepy
----> 2 from kiwipiepy import KiwiTokenizer

ImportError: cannot import name 'KiwiTokenizer' from 'kiwipiepy' (D:\projects\kiwi\kiwipiepy_init_.py)

from datasets import load_dataset
import os
from kiwipiepy import Kiwi
from kiwipiepy.utils import Stopwords

data_dir = './data'
dataset = load_dataset('nsmc')
os.makedirs(data_dir, exist_ok=True)
for split_key in dataset.keys():
doc_path = f"{data_dir}/{split_key}.txt"
with open(doc_path, 'w', encoding='utf-8') as f:
for doc in dataset[split_key]['document']:
f.write(doc + '\n')

kiwi = Kiwi()
stopwords = Stopwords()

ImportError Traceback (most recent call last)
Cell In[3], line 3
1 from datasets import load_dataset
2 import os
----> 3 from kiwipiepy import Kiwi
4 from kiwipiepy.utils import Stopwords
6 data_dir = './data'

File D:\projects\kiwi\kiwipiepy_init_.py:7
5 from kiwipiepy._version import version
6 from kiwipiepy._wrap import Kiwi, Sentence, TypoTransformer, TypoDefinition, HSDataset, MorphemeSet, PretokenizedToken
----> 7 import kiwipiepy.sw_tokenizer as sw_tokenizer
8 import kiwipiepy.utils as utils
9 from kiwipiepy.const import Match

File D:\projects\kiwi\kiwipiepy\sw_tokenizer.py:15
11 import warnings
13 import tqdm
---> 15 from _kiwipiepy import Sw_Tokenizer
17 from kiwipiepy import Kiwi, Token
19 @DataClass
20 class SwTokenizerConfig:

ImportError: cannot import name 'Sw_Tokenizer' from '_kiwipiepy' (D:\anaconda3\Lib\site-packages_kiwipiepy.cp311-win_amd64.pyd)

3. <= 이렇게 하는 코드는 원래 안되는 것인가요?
   from transformers import AutoModelForMaskedLM
   from kiwipiepy.sw_tokenizer import SwTokenizer
   import kiwipiepy.transformers_addon
   from kiwipiepy.sw_tokenizer import KiwiTokenizer

tokenizer = KiwiTokenizer.from_pretrained('kiwi-farm/roberta-base-32k')
#tokenizer = SwTokenizer.KiwiTokenizer("kiwi-farm/roberta-base-32k")
model = AutoModelForMaskedLM.from_pretrained('kiwi-farm/roberta-base-32k')
model.config.is_decoder = True # Set the model as a decoder

prompt = "Gim"
input_ids = tokenizer.encode(prompt, return_tensors="pt")

Generate text using top-k sampling

gen_ids = model.generate(
input_ids,
max_length=50,
do_sample=True, # Enable sampling
top_k=50 # Top-k sampling
)

generated_text = tokenizer.decode(gen_ids[0], skip_special_tokens=True)
print(generated_text)

ImportError Traceback (most recent call last)
Cell In[1], line 2
1 from transformers import AutoModelForMaskedLM
----> 2 from kiwipiepy.sw_tokenizer import SwTokenizer
3 import kiwipiepy.transformers_addon
4 from kiwipiepy.sw_tokenizer import KiwiTokenizer

File D:\projects\kiwi\kiwipiepy_init_.py:7
5 from kiwipiepy._version import version
6 from kiwipiepy._wrap import Kiwi, Sentence, TypoTransformer, TypoDefinition, HSDataset, MorphemeSet, PretokenizedToken
----> 7 import kiwipiepy.sw_tokenizer as sw_tokenizer
8 import kiwipiepy.utils as utils
9 from kiwipiepy.const import Match

File D:\projects\kiwi\kiwipiepy\sw_tokenizer.py:15
11 import warnings
13 import tqdm
---> 15 from _kiwipiepy import Sw_Tokenizer
17 from kiwipiepy import Kiwi, Token
19 @DataClass
20 class SwTokenizerConfig:

ImportError: cannot import name 'Sw_Tokenizer' from '_kiwipiepy' (D:\anaconda3\Lib\site-packages_kiwipiepy.cp311-win_amd64.pyd)

안녕하세요,
먼저, 좋은 형태소 분석기와, Tokenizer 등 오픈소스로 열어주셔서 감사합니다. 항상 잘 사용하고 있습니다.

관련 이슈가 있어서 말씀드립니다.

from transformers import (
AutoTokenizer,
AutoModelForMaskedLM,
)
import kiwipiepy.transformers_addon

kiwipiepy.transformers_addon를 import해야

KiwiTokenizer가 AutoTokenizer에 등록된다.

KiwiTokenizer는 PreTrainedTokenizer와 대부분의 기능이 호환되므로

기존의 transformers 코드를 그대로 사용할 수 있다.

tokenizer = AutoTokenizer.from_pretrained('kiwi-farm/roberta-base-32k')
model = AutoModelForMaskedLM.from_pretrained('kiwi-farm/roberta-base-32k')

KeyError: 'added_tokens'

에러가 발생합니다. OS 등 로컬환경 문제인지, 패키지가 업데이트 되면서 꼬인건지 판단이 어렵네요.

OS: Ubuntu
Python 3.8.10
kiwipiepy == 0.16.2
transformers == 4.36.2