kazimdzuber Goto Github PK

code-mixed-corpus

This repository contain multilingual code-mixed corpus created by own involves around 6k+ multilingual comments of different categories of Gujarati videos from YouTube in which Gujarati language comments written in native and roman script and mixed up with Hindi, English and other language. Corpus involves almost forty thousand tokens of Hindi, Gujarati and English languages. Since sentence contain various form of single language we define the rules to give label to each sentence, according to it we define guideline have three rules and instruct all human annotator to read each individual sentence and according to defined guideline give labels. If any new pattern available in corpus except mentioned label then treat it as other label. We are publishing this data publicly for further research on code-mixed script.

fetch-json-data-

In this project I fetched data from given Link as a task given by Volvere Solution IT Company

langauge-identification-in-code-mixed-script

In this project, we created a unique corpus, and the first time we used a highly imbalanced corpus to do language identification. we did language identification at sentence level in the code-mixed script by applied different machine learning algorithms such as Support vector classifier, Multinomial Naive Bayes, Random Forest, Decision Tree, and Logistic Regression. We manually tag each code-mixed sentence quantity around 6300 sentences involves three different languages (Gujarati, Hindi, and English). We achieved 92% accuracy which is recorded first-time high accuracy with highly imbalance code-mixed data. Also, have published a full-length survey paper on it.

language-identification-in-codemixed-and-code-switched-script-for-indian-language