ai4bharat / shoonya Goto Github PK

An open source platform to annotate and label data at scale

Shoonya is an open source platform to annotate and label data at scale, built with a vision to enhance digital presence of under-represented languages in India.

Shoonya offers support for multiple data types (Ex : parallel datasets, OCR, ASR, TTS etc) and labeling tasks (Ex : parallel datasets, OCR, ASR, TTS etc).

Shoonya, referring to zero, represents the start. It also represents universality in the sense that several cultures have a similar symbol for zero. We believe that the language resources that we collect in Shoonya will start a valuable open-source movement for Indian language technologies that will be available universally for all to adopt and improve upon.

The National Language Translation Mission (NLTM) has been announced in the budget by the Honorable Finance Minister in the backdrop of growing demand for accessing online services in local Indian languages. This will enable the wealth of governance-and-policy related knowledge on the Internet being made available in major Indian languages. The Ministry of Electronics and Information Technology (MeitY) has launched 'Bhashini' to help ensure that digital content is readily available to all citizens, in their preferred languages.

The goal of Bhashini is to develop an ecosystem of innovative practices for data collection, curation, develop technology for speech to speech translation and deliver solutions powered by open data, apps and services. Bhashini shall act as an orchestrator to bring contributions (like data, models etc.) received from government, industry, academia and society into an open “Hundi” or “Repository”. All contributions to Bhashini shall be validated and standardized using a Unified Language Contribution API (ULCA).

Reference : Bhashini Whitepaper

Also read Bhashini Data Report

Naturally, data collection/curation becomes the core of building state-of-the-art NLP ML models. This is where Shoonya comes into picture. Shoonya provides the platform for the Annotators/Translators to create such large datasets with highest quality.

• Support all possible data types and labeling tasks
• Build a reliable & scalable platform beneath Shoonya
• Keep the UI simple and intuitive

git clone --recurse-submodules https://github.com/AI4Bharat/Shoonya

git clone https://github.com/AI4Bharat/Shoonya-Backend.git

python3 -m venv 

source /bin/activate

pip install -r deploy/requirements-dev.txt

cp .env.example ./backend/.env

python backend/manage.py shell

# Import the utility function to generate a secret key.
>> from django.core.management.utils import get_random_secret_key

# Generate and print a new secret key.
>> get_random_secret_key()

docker-compose -f docker-compose-local.yml build

docker-compose -f docker-compose-local.yml up -d

# Check if there are any pending migrations.
docker-compose exec web python backend/manage.py makemigrations 

# Apply all pending migrations to the database.
docker-compose exec web python backend/manage.py migrate

docker-compose exec web python backend/manage.py createsuperuser

docker-compose exec web python backend/manage.py runserver

git clone https://github.com/AI4Bharat/Shoonya-Frontend.git

cd Shoonya-Frontend

npm i --force

npm start

Any information/help/discussion required, can be taken up using the following link : https://github.com/AI4Bharat/Shoonya/discussions

This project adheres to the Contributor Covenant code of conduct. By participating, you are expected to uphold this code. Please report unacceptable behavior to [email protected].