Giter Club home page Giter Club logo

hindi2vec's Introduction

hindi2vec

State-of-the-Art Language Modeling and Text Classification in Hindi Language

Results

We achieved State of the Art Perplexity = 46.81 for Hindi compared to 40.68 for English (lower is better)

  • To the best of my knowledge on September 18, 2018

Update: nlp-for-hindi uses sentencepiece instead of the word based spacCy tokenizer which I use. On those tokens, the measured perplexity for that LM is ~35. I encourage you to check that work out as well.

Downloads

TODO

  • Language modeling based on wikipedia dump
  • Release Language Models: Hindi Language Model
  • Create Text classification Datasets: BBC Hindi
  • Benchmark text classification with FastText

Idea Dump

  • Change the custom head to be used for transliteration instead of classification, Hindi script (Devnagri) to English script (Roman)
  • MTL tasks for training and inference using custom heads
  • Text to Speech - using datasets from news recordings or Hindi subtitles of dubbed movies

FastAI Installation

This version of the notebook uses fastai lib's v0.7, used in their Part 2 v2 course in Summer 2018. The best way to install it via conda as mentioned here

Special thanks to Jeremy, Rachel and other contributors to fastai. This work is a reproduction of their work in English to Hindi. Thanks to @cstorm125 for thai2vec which inspired this work.

hindi2vec's People

Contributors

nirantk avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

hindi2vec's Issues

Add a requirements.txt ?

Hi, can you please add requirements.txt or setup instructions. I am stuck at figuring out the correct fastai version

Training weights

Hi,

Is the trained-weights available somewhere? It would be very beneficial for the community to head-start their models using yours.

not able to run wikiextrator.py

i am getting following error while running
!python wikiextractor/wikiextractor/WikiExtractor.py $filename -o data/wiki_extr --json --quiet

result

  File "wikiextractor/wikiextractor/WikiExtractor.py", line 62, in <module>
    from .extract import Extractor, ignoreTag
ModuleNotFoundError: No module named '__main__.extract'; '__main__' is not a package```

Not able to load encoder

getting error while loading encoder:
m3.load_encoder(f'adam1_enc')

RuntimeError                              Traceback (most recent call last)
~/anaconda2/envs/fastai/lib/python3.6/site-packages/torch/nn/modules/module.py in load_state_dict(self, state_dict, strict)
    513                 try:
--> 514                     own_state[name].copy_(param)
    515                 except Exception:

RuntimeError: invalid argument 2: sizes do not match at /opt/conda/conda-bld/pytorch_1518244421288/work/torch/lib/THC/generic/THCTensorCopy.c:51

During handling of the above exception, another exception occurred:

RuntimeError                              Traceback (most recent call last)
<ipython-input-154-c173cedbc36b> in <module>
----> 1 m3.load_encoder(f'/home/gamut/Downloads/fastai/courses/dl1/data/hindilm/models/adam1_enc')

~/Downloads/fastai/courses/dl1/fastai/nlp.py in load_encoder(self, name)
    164     def save_encoder(self, name): save_model(self.model[0], self.get_model_path(name))
    165 
--> 166     def load_encoder(self, name): load_model(self.model[0], self.get_model_path(name))
    167 
    168 

~/Downloads/fastai/courses/dl1/fastai/torch_imports.py in load_model(m, p)
     38             if n+'_raw' not in sd: sd[n+'_raw'] = sd[n]
     39             del sd[n]
---> 40     m.load_state_dict(sd)
     41 
     42 def load_pre(pre, f, fn):

~/anaconda2/envs/fastai/lib/python3.6/site-packages/torch/nn/modules/module.py in load_state_dict(self, state_dict, strict)
    517                                        'whose dimensions in the model are {} and '
    518                                        'whose dimensions in the checkpoint are {}.'
--> 519                                        .format(name, own_state[name].size(), param.size()))
    520             elif strict:
    521                 raise KeyError('unexpected key "{}" in state_dict'

RuntimeError: While copying the parameter named encoder.weight, whose dimensions in the model are torch.Size([67979, 300]) and whose dimensions in the checkpoint are torch.Size([25704, 300]).

I was able reproduce the notebook given by you but again getting the same error.
Not facing this issue when I load encoder from lesson 4 notebook which is for imdb data.
Can you look through it, otherwise people will not be able to use your pretrained LM

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.