kotan's Introduction

KoTAN: Korean Translation and Augmentation with fine-tuned NLLB

A KoTAN package can exercise korean data augmentation task and en->ko, ko->en translation task. In case of translation model, we are fine-tuning facebook NLLB model. About data augmentation task, we processe backtranslation task. In addition, we also provide speech-style conversion options.

Package install

torch=2.0.0 (cuda 12.0) and python>=3.8 are avaliable.
You can install the package with below command.

pip3 install kotan

Usage

You can use KoTAN with below command.
Import package.

>>> from kotan import KoTAN

Avaliable tasks

>>> KoTAN.available_tasks()

Avaliable languages

>>> KoTAN.available_lang()

Data augmentation options

>>> KoTAN.available_level()

origin: Before fine-tuning nllb model.
fine: After fine-tuning nllb model.

Speech-style conversion options

>>> KoTAN.available_style()

formal: 문어체
informal: 구어체
android: 안드로이드
azae: 아재
chat: 채팅
choding: 초등학생
emoticon: 이모티콘
enfp: enfp
gentle: 신사
halbae: 할아버지
halmae: 할머니
joongding: 중학생
king: 왕
naruto: 나루토
seonbi: 선비
sosim: 소심한
translator: 번역기

Translation

>>> from kotan import KoTAN
>>> mt = KoTAN(task="translation", tgt="en")
>>> inputs = ['나는 온 세상 사람들이 행복해지길 바라', '나는 선한 영향력을 펼치는 사람이 되고 싶어']
>>> mt.predict(inputs)

Data Augmentation

Origin nllb model (before fine-tuning)

>>> from kotan import KoTAN
>>> aug = KoTAN(task="augmentation", level="origin")
>>> inputs = ['나는 온 세상 사람들이 행복해지길 바라', '나는 선한 영향력을 펼치는 사람이 되고 싶어']
>>> aug.predict(inputs)

Fine-tuned nllb model with Aihub datasets.

>>> from kotan import KoTAN
>>> aug = KoTAN(task="augmentation", level="fine")
>>> inputs=['나는 온 세상 사람들이 행복해지길 바라', '나는 선한 영향력을 펼치는 사람이 되고 싶어']
>>> aug.predict(inputs)

Speech-style conversion

>>> from kotan import KoTAN
>>> style = KoTAN(task="style", style="king")
>>> inputs=['나는 온 세상 사람들이 행복해지길 바라', '나는 선한 영향력을 펼치는 사람이 되고 싶어']
>>> style.predict(inputs)

Demo

Huggingface KoTAN space

Citation

@misc{KoTAN,
  author       = {Juhwan Lee, Jisu Kim, TakSung Heo, and Minsu Jeong},
  title        = {KoTAN: Korean Translation and Augmentation with fine-tuned NLLB},
  howpublished = {\url{https://github.com/trailerAI/KoTAN}},
  year         = {2023},
}

Contributors

Jisu, Kim, Juhwan, Lee, TakSung Heo, and Minsu Jeong

License

KoTAN project follow Apache License 2.0 lisence

kotan's People

Contributors

Stargazers

Watchers

kotan's Issues

I couldn't import this library

ImportError Traceback (most recent call last)
Cell In[5], line 1
----> 1 from kotan import KoTAN
2 KoTAN.available_tasks()

File c:\Python310\lib\site-packages\kotan_init_.py:3
1 version = "1.1.3"
----> 3 from .job import KoTAN

File c:\Python310\lib\site-packages\kotan\job.py:3
1 import torch
----> 3 from .tasks import (
4 KoTANTranslationFactory,
5 KoTANAugmentationFactory,
6 KoTANStyleConversiontFactory
7 )
9 from .const import LANG_ALIASES, LEVEL, STYLE
12 # Task list

File c:\Python310\lib\site-packages\kotan\tasks_init_.py:1
----> 1 from .machine_translation import KoTANTranslationFactory
2 from .data_augmentation import KoTANAugmentationFactory
3 from .style_convert import KoTANStyleConversiontFactory
...
45 f"{requirement} is required for a normal functioning of this module, but found {pkg}=={got_ver}.{hint}"
46 )

ImportError: accelerate>=0.20.3 is required for a normal functioning of this module, but found accelerate==0.18.0.
Try: pip install transformers -U or pip install -e '.[dev]' if you're working with git main

I installed this with pip3 install kotan
Is it unavailable on python 3.10?

trailerai / kotan Goto Github PK

kotan's Introduction

KoTAN: Korean Translation and Augmentation with fine-tuned NLLB

Package install

Usage

Translation

Data Augmentation

Origin nllb model (before fine-tuning)

Fine-tuned nllb model with Aihub datasets.

Speech-style conversion

Demo

Citation

Contributors

License

kotan's People

Contributors

Stargazers

Watchers

Forkers

kotan's Issues

Recommend Projects

Recommend Topics

Recommend Org