Giter Club home page Giter Club logo

ai-tools's Introduction

AI Toolchain

AI Toolchain is a collection of tools for quickly building and deploying machine learning models for various use cases. Currently, the toolchain includes a text translation model, and more models may be added in the future.

Prerequisites

Setup

AI-Tools Models at present can be packaged and deployed in two ways:

  • As a Single Image
  • Multiple Images

Single Image

In single image setup, we have a single quart app hosting all the models. It is bundled as a single Docker Image. All models can be accessed following this path convention http://domain_name/<use_case>/<provider>/<mode>

Services are usually provided in either remote mode or local mode.

For example, the text_translation service provided by google can be accessed on your local system at this URL: http://localhost:8000/text_translation/google/remote

Deploy Single Image Setup

docker build -t <image-name> .
docker run -p <host_port>:<container_port> <image-name>

Multiple Image

In Multi Image Setup, each model has its own quart application and docker image. All model APIs are served via NGINX by utilizing proxies to route incoming requests to the appropriate destinations.

There won't be any difference in accessing the models. The text_translation service provided by google can still be accessed on your local system at this endpoint: http://domain_name/text_translation/google/remote

Deploy Multi Image Setup

For mutliple image setup, the docker-compose file and nginx configuration files are generated dynamically based on the config.json file. The following commands might require sudo privileges.

Run this script to generate the configuration files.

./generate.sh

Setup NginX configuration

sudo cp <DOMAIN_NAME>.conf /etc/nginx/conf.d/
sudo nginx -t //Checks for configuration errors
sudo nginx -s reload
  • For Windows copy to C:\nginx\conf\ and reload the server.
  • For MacOS copy to /usr/local/etc/nginx/ and reload the server.

The .conf file will be present in your project root path after running the generate.sh script.

Build Image

sudo docker-compose -f ./docker-compose-generated.yaml build
sudo docker-compose -f docker-compose-generated.yaml up

Adding a new model locally

To add a new model you need to create sub-directory in src. The subdirectory will follow use_case/provider/mode format.

To deploy your application, the dockerfile must:

  • copy program files
  • install requirements
  • build and run the application

After you have developed and dockerized the application, to add a new model, you need to add a key pair inside models object in the config file with the following properties:

  • serviceName : Name of the service. This will also be the docker image's name.
  • modelBasePath: Dockerfile location of your model.
  • apiBasePath: Endpoint from where you wish to access the model. Should follow this convention: <use_case>/<provider>/<mode>.
  • containerPort: Port at which the application will be hosted inside the container.
  • environment: Another JSON object with key-value pairs. Should contain any relevant secrets or API_KEYS required to run the application.

For example, if you want to add a new Text Translation AI Model from OPENAI in remote mode, you need to do as follows:

{
      "serviceName": "text_translation_google",
      "modelBasePath": "src/text_translation/google/remote/.",
      "apiBasePath": "text_translation/google/remote",
      "containerPort": 8000,
      "environment": {
        "OPENAI_API_KEY": "${OPENAI_API_KEY}",
        "GOOGLE_CLOUD_SECRET": "${GCP_SECRET}"
      }
}

Run Ansible Script

Requirements

Steps

  1. Create a inventory.ini file with target machine configuration. Target Machine is where the ansible script will run.
    Here is a sample inventory.ini file:
[swarm_manager]
ee01:ac8:1561::2 ansible_connection=ssh ansible_user=github ansible_ssh_private_key_file=/home/github/.ssh/id_rsa

For More Information Refer https://www.digitalocean.com/community/tutorials/how-to-set-up-ansible-inventories and https://docs.ansible.com/ansible/latest/inventory_guide/intro_inventory.html You may modify it according to your needs.
2. Create a Hashicorp Vault to store your secrets. You need to store the following types of secrets:(In separate Secret Paths mentioned below. This is not optional),

  • Github Credentials. (Path = secret/github)

    • USERNAME (Github Username)
    • PAT (Github Personal Access Token. It should have read and write access to the contents of the repository)

  • Target Machine Credentials (Path = secret/common)

  • DDIR (Destination Directory, place where the files will be downloaded and generated)

  • REPONAME (Github Repository Name. This could be Samagra-Development/ai-tools or the name of your fork.)
    Please note: You don't have to add the entire Github URL, only the Repo Name. for ex: If you fork the ai-tools repo to your account. The repo URL would be something like <your_username>/ai-tools.
    This setup will only work if you have a restructure branch in your fork/repo with generate.sh and config.json in your root folder. Please be aware that modifying the file structure will break the code.

  • Environement Secrets for the Docker Images (Path = secret/config)

    • DOCKER_REGISTRY_URL (ghcr.io if you are pulling from Github Pacakges. Can be Docker Hub as well)
    • GITHUB_REPOSITORY (ai-tools unless told otherwise)
    • OPENAI_API_KEY, and other secrets pertaining to the docker images. This list might exapand as the project grows and supports multiple models. Please take a look at the sample.env for an exhaustive list of other env keys that are required by the models.

    Refer to the faq to see how to add secrets to hashicorp

  1. Set the Vault Credentials in the environment of the target machine so that it can access the remotely hosted Hashicorp Vault.
  • VAULT_ADDR: (Vault Address)
  • ANSIBLE_HASHI_VAULT_TOKEN: (Vault Root Login Token)
    You can add environment variables as follows:
export VAULT_ADDR=http://x.x.x.x:8200
export ANSIBLE_HASHI_VAULT_TOKEN=abc12345

For the Vault Credentials to persist add it to .bashrc

echo 'export VAULT_ADDR=http://x.x.x.x:8200' >> ~/.bashrc
echo 'export 'ANSIBLE_HASHI_VAULT_TOKEN=35r3zxcc' >> ~/.bashrc
source ~/.bashrc

Alternatively, you can pass the variables during run time as Command Line Arguments using --extra-vars field. Here is an example:

ansible-playbook -i inventory.ini swarm.yml --extra-vars "VAULT_ADDR='http://127.0.0.1:8200' VAULT_TOKEN=abc.123" --ask-become-pass

Additional Steps

  1. Change Hosts in Ansible Playbook to the Node which is acting as the swarm manager. (Make sure a Docker Swarm is running in that machine)

To run the Ansible Playbook

ansible-playbook swarm.yml -i inventory.ini

Contributing

Contributions to AI Toolchain are welcome! To contribute, please follow these guidelines:

  1. Fork the repository and create a new branch for your feature or bug fix.
  2. Write tests for your changes.
  3. Submit a pull request describing your changes and why they are needed.

Thank you for considering contributing to AI Toolchain!

ai-tools's People

Contributors

amruth-vamshi avatar chakshugautam avatar djgupta avatar gautamr-samagra avatar h4r5h1t-007 avatar ksgr5566 avatar manas23601 avatar nitesh639 avatar psn0w avatar sanjaysinghrajpoot avatar shubh-goyal-07 avatar singhalkarun avatar tushar5526 avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar

Forkers

gautamr-samagra ryanwalker277 djgupta sanjay-aurigait devayushdubey poonam1607 preetam1407 j-e-e-t nazi-pikachu wasup-yash itsrahulsehgal iamarjitgoyal parth1007 janeka1122 sanjaysinghrajpoot adityyax gupta-aryaman fykaa bruno-noir adityamisra01 psn0w apoorvgarg-creator rafey1104 shazy01 nitesh639 adv1k sarthakrw pyasma yb73 ajitg25 rishav-eulb harshinirhsv navneet-28 danishraza0912 itshmoh shrivastava95 anann99 cyber-machine goyalpramod ksgr5566 shradhayy posigangapavani uncanny69 asjad895 kaushikdey647 negrah-s unknownsentinel193 inkerton analystexplorer ysrastogi nishankjain1 jiya126 tejash11 sudeshna-s03 sathvikap paradise110302 animesh312 rohanhbtu aaryansahu utkarsh1210-tech killer-mohit sasank15 divyansh-tripathi7 abhinavarya7 dhruvsinghbora manas23601 ayush0x00 santhoshstark06 eeshnaugraiya shro-2002 wetleaf raksansuseendran smitjiwani arjunyaduvanshi02 pruntr sharma-satyam697 vishvjeet-thakur ajaman190 karun-development aakarshhhh prtkjakhar h4r5h1t-007 suyashchougale chatwithpdf shephinphilip amakrushai parth11791 bainaryglobe singhalkarun wassaf001 stophobia 35c4n0r xorsuyash prachipandey16 iamkrmayank soham-27 masterismail sanjay8602 tornado-hacker toshan07

ai-tools's Issues

Sagemaker Setup

Description

Evaluate Sagemaker as a potential solution for deploying AI-tools and provide documentation to guide users in deploying all models to Sagemaker.

Goals

  • Research and evaluate the feasibility of using Sagemaker for deploying AI-tools
  • Identify the steps and configurations required to deploy all models to Sagemaker
  • Document the deployment process in a comprehensive sagemaker.md guide

Expected Outcome

By evaluating Sagemaker for deploying AI-tools and documenting the deployment process, users will have clear instructions on how to deploy all models to Sagemaker. This enables the easy adoption and usage of AI-tools in a Sagemaker environment.

Acceptance Criteria

  • Decision tree created on feasibility of using Sagemaker for AI-tools deployment, detailing when Sagemaker is advantageous
  • Detailed steps and configurations are identified for deploying all models to Sagemaker
  • A comprehensive sagemaker.md guide is created, providing instructions for deploying AI-tools on Sagemaker

Project

AI-tools

Organization Name:

Samagra

Domain

Social Welfare

Tech Skills Needed:

AWS Sagemaker, AI-tools

Mentor(s)

@ChakshuGautam

Complexity

Hard

Category

Deployment

Sub Category

Infrastructure

[C4GT] Neural coreference for enabling conversational flow in bots

Project Details

AI Toolchain is a collection of tools for quickly building and deploying machine learning models for various use cases. Currently, the toolchain includes a text translation model, and more models may be added in the future. It abstracts the dirty details of how a model works similar to Huggingface and gives a clean API that you can orchestrate at a BFF level.

Features to be implemented

Neural coreference resolution (overview) is a natural language processing (NLP) task that involves identifying when two or more words or phrases in a text refer to the same entity or concept. This process is crucial for understanding the context and meaning of a text, as it helps in resolving ambiguities and connecting different parts of a text.

For example:
Non-coreferenced conversation:
User : 'Can you tell me where are the shops for paddy seeds?'
User : 'What is the price for them?'

Coreferenced conversation:
User : 'Can you tell me where are the shops for paddy seeds?'
User : 'What is the price for paddy seeds?'

How it works

Neural coreference resolution models usually involve various components, such as:

Feature Extraction:

Detecting the different parts of a sentence and their relationships with each other

Mention detection:

Identifying potential words or phrases (mentions) that may be involved in coreference relations (they all refer to the same entity)

Pairwise scoring:

Computing a score for each pair of mentions, representing the likelihood that they corefer.

Clustering:

Grouping mentions into clusters, where each cluster represents a single entity or concept.

Replacement:

For the last message in the conversation (or for all messages succeeding the first), replace the phrases in each cluster by the common word that gives the entire picture.

Deployment:

Deploy the above setup as a part of ai-tools package such that it can be dockerized and then accessed through an API setup.

Learning Path

Complexity

Hard

Skills Required

Python, NLP

Name of Mentors:

@GautamR-Samagra

Project size

8 Weeks

Product Set Up

See the setup here
The setup here is just an example or baseline solution of how it could be carried out. We are currently focusing this project on using the Spacy model to carry out coreference. Exploration of other techniques is encouraged only if the this one fails.

Acceptance Criteria

  • Unit Test Cases
  • OpenAPI Spec/Postman Collection
  • Dockerfile for this module

Milestone

  • Setting up spacy
  • Identifying type and format of data required to train
  • Creating synthetic data/collecting data for training
  • Training the model based on collected data
  • Carry out automated testing for models / integrate training based on our test cases
  • New trained model based on trained data

Reference

https://galhever.medium.com/a-review-to-co-reference-resolution-models-f44b4360a00

https://explosion.ai/blog/coref
explosion/spaCy#11585 (reply in thread)

C4GT

This issue is nominated for Code for GovTech (C4GT) 2023 edition.
C4GT is India's first annual coding program to create a community that can build and contribute to global Digital Public Goods. If you want to use Open Source GovTech to create impact, then this is the opportunity for you! More about C4GT here: https://codeforgovtech.in/

The scope of this ticket has now expanded to make it the 'enabling conversation' part of 'FAQ bot'.
The FAQ bot allows a user to be able to provide content input in the form on csvs, free text, pdfs, audio, video and the bot is able to add it to a 'Content DB'. The user is then able to interact with the bot via text/speech on related content and the bot is able to identify relevant content using RAG techniques and be able to be able to respond to the user in a conversational manner.

This ticket covers the enabling conversation +small model deployment/finetuning pipeline. It includes the following tasks in its scope:

Add prometheus server

Description

Add Prometheus server to the project and update the Docker Compose file to include the necessary configurations.

Goals

  • Install and configure Prometheus server for monitoring the project
  • Update the Docker Compose file to include the Prometheus service
  • Define appropriate configurations and targets for Prometheus scraping

Expected Outcome

By adding Prometheus server and updating the Docker Compose file, the project will have monitoring capabilities. Prometheus will be able to scrape metrics from the services defined in the Docker Compose file, providing valuable insights into the performance and health of the system.

Acceptance Criteria

  • Prometheus server is successfully installed and configured
  • Docker Compose file is updated to include the Prometheus service
  • Configurations and targets for Prometheus scraping are defined and working correctly

Project

AI-tools

Organization Name:

Samagra

Domain

Social Welfare

Tech Skills Needed:

Docker, Prometheus

Mentor(s)

@ChakshuGautam

Complexity

Medium

Category

DevOps

Sub Category

Monitoring

Upload CSV API - Sanjay

Python decorator over an API Call

Input

  • Streaming CSV file
  • Fetch a file from a remote location

Implementation Details

  • Keep it in memory
  • Store it in folder /tmp on the server => Load it in memory through a CSV Reader

Output

  • CSV Reader

Add a header to the response when it comes from a cached function

Description

Add a header to the response when it comes from a cached function to indicate its origin.

Goals

  • Modify the function logic to include a header in the response when it is served from a cached function
  • Define a custom header, such as ai-tools-cached, to indicate that the response is from a cached function
  • Ensure the header is included in the response for appropriate requests

Expected Outcome

By adding a header to the response from a cached function, it becomes possible to differentiate between responses served from cache and those generated in real-time. This information can be useful for debugging and monitoring purposes, allowing developers to understand the performance and behavior of the cached functions.

Acceptance Criteria

  • The function logic is updated to include a custom header, such as ai-tools-cached, in the response when it comes from a cached function
  • The header is included in the response for appropriate requests
  • The presence of the header accurately indicates whether the response is from a cached function or not

Project

AI-tools

Organization Name:

Samagra

Domain

Social Welfare

Tech Skills Needed:

Backend Development

Mentor(s)

@ChakshuGautam

Complexity

Medium

Category

Backend

Sub Category

API Development

Add OpenAPI Spec

  • POST /useCase/model
  • POST /health
  • POST /embeddings/model

Samvit

Update workflow to make package public while pushing to github package and also automatically connect it to repository

Description

Update the workflow to make a package public and automatically connect it to the repository in GitHub Packages.

Goals

  • Modify the workflow to configure the package to be public upon pushing it to GitHub Packages
  • Automate the connection of the package to the repository
  • Ensure the workflow runs smoothly and the package is accessible to users

Expected Outcome

The updated workflow will automatically make the package public and establish a connection to the repository in GitHub Packages. This ensures that the package is easily accessible and discoverable by users who can benefit from its functionalities.

Acceptance Criteria

  • The workflow is updated to include the necessary configuration for making the package public
  • The workflow automatically establishes a connection between the package and the repository in GitHub Packages
  • The package is accessible and visible to users in GitHub Packages

Project

AI-tools

Organization Name:

Samagra

Domain

Social Welfare

Tech Skills Needed:

GitHub Actions

Mentor(s)

@ChakshuGautam

Complexity

Medium

Category

DevOps

Sub Category

Maintenance

Docs: For new contributes and Revamp ReadMe docs for installation

It appears that the readme file of the project is currently outdated and lacks documentation for contributors. Assuming that the project team wants to improve the documentation, they may welcome contributions from interested parties who have the necessary expertise. It is also possible that they may have a plan in place to update the documentation themselves, but have not yet had the resources or time to do so. Regardless of the reason, it is essential that the documentation is kept up-to-date to ensure effective collaboration and understanding among the contributors. Interested parties may want to reach out to the project team to inquire about the status of the documentation and offer their assistance if possible.

Curl Requests for the following APIs

  • v1 - /translateX2Y/bhashini/remote - Input [(source, target, text)] => [text] @ChakshuGautam
  • v1 - /translateX2Y/bhashini/onprem (Indic to English, English to Indic) - Input [(source, target, text)] => [text] @djgupta
  • v1 - /llm/openai/remote - Input (description, examples, query) => text @djgupta
  • v1 - /detectLanguage/bhashini/remote - Input (text) => language @ChakshuGautam
  • v2 - /detectLanguage/bhashini/onprem - Input (text) => language @djgupta
  • v1 - /similarContent/BERT/onprem - Input (prompt, db, threshold) => [text] @GautamR-Samagra
  • v1 - /speech2text/bhashini/onprem - Input [(audio url, encoding)] => [text] @djgupta
  • v1 - /speech2text/bhashini/remote - Input [(audio url, encoding)] => [text] @ChakshuGautam
  • v1 - /embeddings/openai/remote @GautamR-Samagra @ChakshuGautam

Add CI using GitHub Workflows

The CI should

  • Installation
  • Run the integration test suite
  • Run the Unit tests suite

When

  • On a PR
  • Before merging

Outputs

  • Badges on coverage

Karun

Abstract out and add an async similarity helper function 

async def GPT3_farmer_prompt(data, context):
    # df1 = pd.read_csv('df1.csv')
    df1 = pd.read_csv('df_all_new_data.csv')
    content = df1['Content']
    actual_sentences = df1['sentence_constructed']
    encoded_string = df1['encoded_sentence']

    encoded = []
    for en_str in encoded_string:
        encoded.append(np.array(json.loads(en_str)))
    encoded = np.array(encoded)

    model = SentenceTransformer('bert-base-nli-mean-tokens')

    def get_prediction(sent):
        sim = cosine_similarity(
            model.encode([sent]),
            encoded
        )
        max_prob = np.argmax(sim)
        max_sim = np.max(sim)
        return actual_sentences[max_prob], content[max_prob], max_sim

We have to add this function as an async helper function (if possible) in the repo.

torch==2.0.0 is missing

When I run the ai-tools/src/text_translation/ai4bharat/deploy.sh file, facing this issue:
Screenshot from 2023-05-10 12-08-28

because of this line:

ai-tools/src/text_translation/ai4bharat/deploy.sh

Line 13 in 54f69fc

poetry add gunicorn flask_cors sacremoses pandas mock sacrebleu tensorboardX pyarrow indic-nlp-library mosestokenizer subword-nmt xformers triton flask_cors webvtt-py

Regarding this issue:
We should also need to add "torch==2.0.0" because xformers==latest depend on torch==2.0.0 and triton, which is already available but torch==2.0.0 is not.

Separate out components

Need to separate out componets such that GPU required and no required ones can be hosted separately and each component can be deployed in individual docker containers

Deployment Scripts

  • Horizontally scalable deployment using Ansible on a GPU based inventory

Karun

Add a health check API

Use this for any external dependent services.

  • A global /health endpoint that allows to check health of each model.
  • Update generate.sh to make health checks the part of generated docker-compose and nginx configuration file

Add Gitpod

Gitpod setup for faster hacking and contributions.

Create a fine grained access token for pushing docker images to github packages

Description

Create a fine-grained access token that allows pushing Docker images to GitHub Packages.

Goals

  • Generate a new access token with the necessary permissions for pushing Docker images to GitHub Packages
  • Configure the access token to have fine-grained permissions limited to the required repositories and packages
  • Document the access token and its usage for future reference

Expected Outcome

The access token will provide the necessary credentials to push Docker images to GitHub Packages. By creating a fine-grained access token, you can ensure that it only has the required permissions and restrict access to specific repositories and packages, enhancing security and access control.

Acceptance Criteria

  • A new access token is generated specifically for pushing Docker images to GitHub Packages
  • The access token is configured with fine-grained permissions limited to the necessary repositories and packages
  • The access token is documented, including its usage and any expiration or renewal instructions

Implementation Details

[List any technical details about the proposed implementation, including any specific technologies that will be used.]

Mockups / Wireframes

[N/A]

Project

AI-tools

Organization Name:

Samagra

Domain

Social Welfare

Tech Skills Needed:

GitHub, Docker

Mentor(s)

@ChakshuGautam

Complexity

Medium

Category

DevOps

Sub Category

Credentials Management

Refactor Remote Services

Remote service onboarding should not include code changes. They should be onboarded directly through repository.private.json which overrides the repository.json

Reducing the size of docker image

Current sizes

root@e25f48a1466e:/usr/local/lib/python3.8/site-packages# du -sh * | sort -hr
2.6G	nvidia
1.3G	torch
184M	triton
87M	torchvision
80M	plotly
76M	scipy
65M	cmake
49M	pandas
35M	sklearn
35M	scipy.libs
35M	numpy.libs
34M	sympy

Dependency Graph

aiohttp 3.8.4 Async http client/server framework (asyncio)
├── aiosignal >=1.1.2
│   └── frozenlist >=1.1.0 
├── async-timeout >=4.0.0a3,<5.0
├── attrs >=17.3.0
├── charset-normalizer >=2.0,<4.0
├── frozenlist >=1.1.1
├── multidict >=4.5,<7.0
└── yarl >=1.0,<2.0
    ├── idna >=2.0 
    └── multidict >=4.0 
async-cache 1.1.1 An asyncio Cache
asyncio 3.4.3 reference implementation of PEP 3156
en-coreference-web-trf 3.4.0a2 English transformer pipeline to demonstrate the experimental coreference components. Components: sentencizer, transfomer, coref, span_resolver, span_cleaner.
├── spacy >=3.3.0,<3.5.0
│   ├── catalogue >=2.0.6,<2.1.0 
│   ├── cymem >=2.0.2,<2.1.0 
│   ├── jinja2 * 
│   │   └── markupsafe >=2.0 
│   ├── langcodes >=3.2.0,<4.0.0 
│   ├── murmurhash >=0.28.0,<1.1.0 
│   ├── numpy >=1.15.0 
│   ├── packaging >=20.0 
│   ├── pathy >=0.3.5 
│   │   ├── smart-open >=5.2.1,<7.0.0 
│   │   └── typer >=0.3.0,<1.0.0 
│   │       └── click >=7.1.1,<9.0.0 
│   │           └── colorama * 
│   ├── preshed >=3.0.2,<3.1.0 
│   │   ├── cymem >=2.0.2,<2.1.0 (circular dependency aborted here)
│   │   └── murmurhash >=0.28.0,<1.1.0 (circular dependency aborted here)
│   ├── pydantic >=1.7.4,<1.8 || >1.8,<1.8.1 || >1.8.1,<1.11.0 
│   │   └── typing-extensions >=4.2.0 
│   ├── requests >=2.13.0,<3.0.0 
│   │   ├── certifi >=2017.4.17 
│   │   ├── charset-normalizer >=2,<4 
│   │   ├── idna >=2.5,<4 
│   │   └── urllib3 >=1.21.1,<1.27 
│   ├── setuptools * 
│   ├── smart-open >=5.2.1,<7.0.0 (circular dependency aborted here)
│   ├── spacy-legacy >=3.0.10,<3.1.0 
│   ├── spacy-loggers >=1.0.0,<2.0.0 
│   ├── srsly >=2.4.3,<3.0.0 
│   │   └── catalogue >=2.0.3,<2.1.0 (circular dependency aborted here)
│   ├── thinc >=8.1.0,<8.2.0 
│   │   ├── blis >=0.7.8,<0.8.0 
│   │   │   └── numpy >=1.15.0 (circular dependency aborted here)
│   │   ├── catalogue >=2.0.4,<2.1.0 (circular dependency aborted here)
│   │   ├── confection >=0.0.1,<1.0.0 
│   │   │   ├── pydantic >=1.7.4,<1.8 || >1.8,<1.8.1 || >1.8.1,<1.11.0 (circular dependency aborted here)
│   │   │   └── srsly >=2.4.0,<3.0.0 (circular dependency aborted here)
│   │   ├── cymem >=2.0.2,<2.1.0 (circular dependency aborted here)
│   │   ├── murmurhash >=1.0.2,<1.1.0 (circular dependency aborted here)
│   │   ├── numpy >=1.15.0 (circular dependency aborted here)
│   │   ├── packaging >=20.0 (circular dependency aborted here)
│   │   ├── preshed >=3.0.2,<3.1.0 (circular dependency aborted here)
│   │   ├── pydantic >=1.7.4,<1.8 || >1.8,<1.8.1 || >1.8.1,<1.11.0 (circular dependency aborted here)
│   │   ├── setuptools * (circular dependency aborted here)
│   │   ├── srsly >=2.4.0,<3.0.0 (circular dependency aborted here)
│   │   └── wasabi >=0.8.1,<1.2.0 
│   ├── tqdm >=4.38.0,<5.0.0 
│   │   └── colorama * (circular dependency aborted here)
│   ├── typer >=0.3.0,<0.8.0 (circular dependency aborted here)
│   └── wasabi >=0.9.1,<1.1.0 (circular dependency aborted here)
├── spacy-experimental >=0.6.1,<0.7.0
│   └── spacy >=3.3.0,<3.6.0 
│       ├── catalogue >=2.0.6,<2.1.0 
│       ├── cymem >=2.0.2,<2.1.0 
│       ├── jinja2 * 
│       │   └── markupsafe >=2.0 
│       ├── langcodes >=3.2.0,<4.0.0 
│       ├── murmurhash >=0.28.0,<1.1.0 
│       ├── numpy >=1.15.0 
│       ├── packaging >=20.0 
│       ├── pathy >=0.3.5 
│       │   ├── smart-open >=5.2.1,<7.0.0 
│       │   └── typer >=0.3.0,<1.0.0 
│       │       └── click >=7.1.1,<9.0.0 
│       │           └── colorama * 
│       ├── preshed >=3.0.2,<3.1.0 
│       │   ├── cymem >=2.0.2,<2.1.0 (circular dependency aborted here)
│       │   └── murmurhash >=0.28.0,<1.1.0 (circular dependency aborted here)
│       ├── pydantic >=1.7.4,<1.8 || >1.8,<1.8.1 || >1.8.1,<1.11.0 
│       │   └── typing-extensions >=4.2.0 
│       ├── requests >=2.13.0,<3.0.0 
│       │   ├── certifi >=2017.4.17 
│       │   ├── charset-normalizer >=2,<4 
│       │   ├── idna >=2.5,<4 
│       │   └── urllib3 >=1.21.1,<1.27 
│       ├── setuptools * 
│       ├── smart-open >=5.2.1,<7.0.0 (circular dependency aborted here)
│       ├── spacy-legacy >=3.0.10,<3.1.0 
│       ├── spacy-loggers >=1.0.0,<2.0.0 
│       ├── srsly >=2.4.3,<3.0.0 
│       │   └── catalogue >=2.0.3,<2.1.0 (circular dependency aborted here)
│       ├── thinc >=8.1.0,<8.2.0 
│       │   ├── blis >=0.7.8,<0.8.0 
│       │   │   └── numpy >=1.15.0 (circular dependency aborted here)
│       │   ├── catalogue >=2.0.4,<2.1.0 (circular dependency aborted here)
│       │   ├── confection >=0.0.1,<1.0.0 
│       │   │   ├── pydantic >=1.7.4,<1.8 || >1.8,<1.8.1 || >1.8.1,<1.11.0 (circular dependency aborted here)
│       │   │   └── srsly >=2.4.0,<3.0.0 (circular dependency aborted here)
│       │   ├── cymem >=2.0.2,<2.1.0 (circular dependency aborted here)
│       │   ├── murmurhash >=1.0.2,<1.1.0 (circular dependency aborted here)
│       │   ├── numpy >=1.15.0 (circular dependency aborted here)
│       │   ├── packaging >=20.0 (circular dependency aborted here)
│       │   ├── preshed >=3.0.2,<3.1.0 (circular dependency aborted here)
│       │   ├── pydantic >=1.7.4,<1.8 || >1.8,<1.8.1 || >1.8.1,<1.11.0 (circular dependency aborted here)
│       │   ├── setuptools * (circular dependency aborted here)
│       │   ├── srsly >=2.4.0,<3.0.0 (circular dependency aborted here)
│       │   └── wasabi >=0.8.1,<1.2.0 
│       ├── tqdm >=4.38.0,<5.0.0 
│       │   └── colorama * (circular dependency aborted here)
│       ├── typer >=0.3.0,<0.8.0 (circular dependency aborted here)
│       └── wasabi >=0.9.1,<1.1.0 (circular dependency aborted here)
└── spacy-transformers >=1.1.2,<1.2.0
    ├── spacy >=3.4.0,<4.0.0 
    │   ├── catalogue >=2.0.6,<2.1.0 
    │   ├── cymem >=2.0.2,<2.1.0 
    │   ├── jinja2 * 
    │   │   └── markupsafe >=2.0 
    │   ├── langcodes >=3.2.0,<4.0.0 
    │   ├── murmurhash >=0.28.0,<1.1.0 
    │   ├── numpy >=1.15.0 
    │   ├── packaging >=20.0 
    │   ├── pathy >=0.3.5 
    │   │   ├── smart-open >=5.2.1,<7.0.0 
    │   │   └── typer >=0.3.0,<1.0.0 
    │   │       └── click >=7.1.1,<9.0.0 
    │   │           └── colorama * 
    │   ├── preshed >=3.0.2,<3.1.0 
    │   │   ├── cymem >=2.0.2,<2.1.0 (circular dependency aborted here)
    │   │   └── murmurhash >=0.28.0,<1.1.0 (circular dependency aborted here)
    │   ├── pydantic >=1.7.4,<1.8 || >1.8,<1.8.1 || >1.8.1,<1.11.0 
    │   │   └── typing-extensions >=4.2.0 
    │   ├── requests >=2.13.0,<3.0.0 
    │   │   ├── certifi >=2017.4.17 
    │   │   ├── charset-normalizer >=2,<4 
    │   │   ├── idna >=2.5,<4 
    │   │   └── urllib3 >=1.21.1,<1.27 
    │   ├── setuptools * 
    │   ├── smart-open >=5.2.1,<7.0.0 (circular dependency aborted here)
    │   ├── spacy-legacy >=3.0.10,<3.1.0 
    │   ├── spacy-loggers >=1.0.0,<2.0.0 
    │   ├── srsly >=2.4.3,<3.0.0 
    │   │   └── catalogue >=2.0.3,<2.1.0 (circular dependency aborted here)
    │   ├── thinc >=8.1.0,<8.2.0 
    │   │   ├── blis >=0.7.8,<0.8.0 
    │   │   │   └── numpy >=1.15.0 (circular dependency aborted here)
    │   │   ├── catalogue >=2.0.4,<2.1.0 (circular dependency aborted here)
    │   │   ├── confection >=0.0.1,<1.0.0 
    │   │   │   ├── pydantic >=1.7.4,<1.8 || >1.8,<1.8.1 || >1.8.1,<1.11.0 (circular dependency aborted here)
    │   │   │   └── srsly >=2.4.0,<3.0.0 (circular dependency aborted here)
    │   │   ├── cymem >=2.0.2,<2.1.0 (circular dependency aborted here)
    │   │   ├── murmurhash >=1.0.2,<1.1.0 (circular dependency aborted here)
    │   │   ├── numpy >=1.15.0 (circular dependency aborted here)
    │   │   ├── packaging >=20.0 (circular dependency aborted here)
    │   │   ├── preshed >=3.0.2,<3.1.0 (circular dependency aborted here)
    │   │   ├── pydantic >=1.7.4,<1.8 || >1.8,<1.8.1 || >1.8.1,<1.11.0 (circular dependency aborted here)
    │   │   ├── setuptools * (circular dependency aborted here)
    │   │   ├── srsly >=2.4.0,<3.0.0 (circular dependency aborted here)
    │   │   └── wasabi >=0.8.1,<1.2.0 
    │   ├── tqdm >=4.38.0,<5.0.0 
    │   │   └── colorama * (circular dependency aborted here)
    │   ├── typer >=0.3.0,<0.8.0 (circular dependency aborted here)
    │   └── wasabi >=0.9.1,<1.1.0 (circular dependency aborted here)
    ├── spacy-alignments >=0.7.2,<1.0.0 
    ├── srsly >=2.4.0,<3.0.0 (circular dependency aborted here)
    ├── torch >=1.6.0 
    │   ├── filelock * 
    │   ├── jinja2 * (circular dependency aborted here)
    │   ├── networkx * 
    │   ├── nvidia-cublas-cu11 11.10.3.66 
    │   │   ├── setuptools * (circular dependency aborted here)
    │   │   └── wheel * 
    │   ├── nvidia-cuda-cupti-cu11 11.7.101 
    │   │   ├── setuptools * (circular dependency aborted here)
    │   │   └── wheel * (circular dependency aborted here)
    │   ├── nvidia-cuda-nvrtc-cu11 11.7.99 
    │   │   ├── setuptools * (circular dependency aborted here)
    │   │   └── wheel * (circular dependency aborted here)
    │   ├── nvidia-cuda-runtime-cu11 11.7.99 
    │   │   ├── setuptools * (circular dependency aborted here)
    │   │   └── wheel * (circular dependency aborted here)
    │   ├── nvidia-cudnn-cu11 8.5.0.96 
    │   │   ├── setuptools * (circular dependency aborted here)
    │   │   └── wheel * (circular dependency aborted here)
    │   ├── nvidia-cufft-cu11 10.9.0.58 
    │   ├── nvidia-curand-cu11 10.2.10.91 
    │   │   ├── setuptools * (circular dependency aborted here)
    │   │   └── wheel * (circular dependency aborted here)
    │   ├── nvidia-cusolver-cu11 11.4.0.1 
    │   │   ├── setuptools * (circular dependency aborted here)
    │   │   └── wheel * (circular dependency aborted here)
    │   ├── nvidia-cusparse-cu11 11.7.4.91 
    │   │   ├── setuptools * (circular dependency aborted here)
    │   │   └── wheel * (circular dependency aborted here)
    │   ├── nvidia-nccl-cu11 2.14.3 
    │   ├── nvidia-nvtx-cu11 11.7.91 
    │   │   ├── setuptools * (circular dependency aborted here)
    │   │   └── wheel * (circular dependency aborted here)
    │   ├── sympy * 
    │   │   └── mpmath >=0.19 
    │   ├── triton 2.0.0 
    │   │   ├── cmake * 
    │   │   ├── filelock * (circular dependency aborted here)
    │   │   ├── lit * 
    │   │   └── torch * (circular dependency aborted here)
    │   └── typing-extensions * (circular dependency aborted here)
    └── transformers >=3.4.0,<4.26.0 
        ├── filelock * (circular dependency aborted here)
        ├── huggingface-hub >=0.10.0,<1.0 
        │   ├── filelock * (circular dependency aborted here)
        │   ├── packaging >=20.9 (circular dependency aborted here)
        │   ├── pyyaml >=5.1 
        │   ├── requests * (circular dependency aborted here)
        │   ├── tqdm >=4.42.1 (circular dependency aborted here)
        │   └── typing-extensions >=3.7.4.3 (circular dependency aborted here)
        ├── numpy >=1.17 (circular dependency aborted here)
        ├── packaging >=20.0 (circular dependency aborted here)
        ├── pyyaml >=5.1 (circular dependency aborted here)
        ├── regex !=2019.12.17 
        ├── requests * (circular dependency aborted here)
        ├── tokenizers >=0.11.1,<0.11.3 || >0.11.3,<0.14 
        └── tqdm >=4.27 (circular dependency aborted here)
flake8 6.0.0 the modular source code checker: pep8 pyflakes and co
├── mccabe >=0.7.0,<0.8.0
├── pycodestyle >=2.10.0,<2.11.0
└── pyflakes >=3.0.0,<3.1.0
flask 2.2.3 A simple framework for building complex web applications.
├── click >=8.0
│   └── colorama * 
├── importlib-metadata >=3.6.0
│   └── zipp >=0.5 
├── itsdangerous >=2.0
├── jinja2 >=3.0
│   └── markupsafe >=2.0 
└── werkzeug >=2.2.2
    └── markupsafe >=2.1.1 
inference 0.1
matplotlib 3.7.1 Python plotting package
├── contourpy >=1.0.1
│   └── numpy >=1.16 
├── cycler >=0.10
├── fonttools >=4.22.0
├── importlib-resources >=3.2.0
│   └── zipp >=3.1.0 
├── kiwisolver >=1.0.1
├── numpy >=1.20
├── packaging >=20.0
├── pillow >=6.2.0
├── pyparsing >=2.3.1
└── python-dateutil >=2.7
    └── six >=1.5 
numpy 1.24.2 Fundamental package for array computing in Python
openai 0.27.4 Python client library for the OpenAI API
├── aiohttp *
│   ├── aiosignal >=1.1.2 
│   │   └── frozenlist >=1.1.0 
│   ├── async-timeout >=4.0.0a3,<5.0 
│   ├── attrs >=17.3.0 
│   ├── charset-normalizer >=2.0,<4.0 
│   ├── frozenlist >=1.1.1 (circular dependency aborted here)
│   ├── multidict >=4.5,<7.0 
│   └── yarl >=1.0,<2.0 
│       ├── idna >=2.0 
│       └── multidict >=4.0 (circular dependency aborted here)
├── requests >=2.20
│   ├── certifi >=2017.4.17 
│   ├── charset-normalizer >=2,<4 
│   ├── idna >=2.5,<4 
│   └── urllib3 >=1.21.1,<1.27 
└── tqdm *
    └── colorama * 
openai-async 0.0.3 A light-weight, asynchronous client for OpenAI API - text completion, image generation and embeddings.
├── httpx *
│   ├── certifi * 
│   ├── httpcore >=0.15.0,<0.18.0 
│   │   ├── anyio >=3.0,<5.0 
│   │   │   ├── idna >=2.8 
│   │   │   └── sniffio >=1.1 
│   │   ├── certifi * (circular dependency aborted here)
│   │   ├── h11 >=0.13,<0.15 
│   │   └── sniffio >=1.0.0,<2.0.0 (circular dependency aborted here)
│   ├── idna * (circular dependency aborted here)
│   └── sniffio * (circular dependency aborted here)
└── pytest *
    ├── colorama * 
    ├── exceptiongroup >=1.0.0rc8 
    ├── iniconfig * 
    ├── packaging * 
    ├── pluggy >=0.12,<2.0 
    └── tomli >=1.0.0 
pandas 2.0.0 Powerful data structures for data analysis, time series, and statistics
├── numpy >=1.20.3
├── numpy >=1.21.0
├── numpy >=1.23.2
├── python-dateutil >=2.8.2
│   └── six >=1.5 
├── pytz >=2020.1
└── tzdata >=2022.1
pipdeptree 2.0.0 Command line utility to show dependency tree of packages
└── pip >=6.0.0
plotly 5.14.1 An open-source, interactive data visualization library for Python
├── packaging *
└── tenacity >=6.2.0
python-dotenv 1.0.0 Read key-value pairs from a .env file and set them as environment variables
quart 0.18.4 A Python ASGI web microframework with the same API as Flask
├── aiofiles *
├── blinker <1.6
├── click >=8.0.0
│   └── colorama * 
├── hypercorn >=0.11.2
│   ├── h11 * 
│   ├── h2 >=3.1.0 
│   │   ├── hpack >=4.0,<5 
│   │   └── hyperframe >=6.0,<7 
│   ├── priority * 
│   ├── toml * 
│   └── wsproto >=0.14.0 
│       └── h11 >=0.9.0,<1 (circular dependency aborted here)
├── importlib-metadata *
│   └── zipp >=0.5 
├── itsdangerous *
├── jinja2 *
│   └── markupsafe >=2.0 
├── markupsafe *
└── werkzeug >=2.2.0
    └── markupsafe >=2.1.1 
scikit-learn 1.2.2 A set of python modules for machine learning and data mining
├── joblib >=1.1.1
├── numpy >=1.17.3
├── scipy >=1.3.2
│   └── numpy >=1.18.5,<1.26.0 
└── threadpoolctl >=2.0.0
sklearn 0.0.post4 deprecated sklearn package, use scikit-learn instead
spacy-experimental 0.6.2 Cutting-edge experimental spaCy components and features
└── spacy >=3.3.0,<3.6.0
    ├── catalogue >=2.0.6,<2.1.0 
    ├── cymem >=2.0.2,<2.1.0 
    ├── jinja2 * 
    │   └── markupsafe >=2.0 
    ├── langcodes >=3.2.0,<4.0.0 
    ├── murmurhash >=0.28.0,<1.1.0 
    ├── numpy >=1.15.0 
    ├── packaging >=20.0 
    ├── pathy >=0.3.5 
    │   ├── smart-open >=5.2.1,<7.0.0 
    │   └── typer >=0.3.0,<1.0.0 
    │       └── click >=7.1.1,<9.0.0 
    │           └── colorama * 
    ├── preshed >=3.0.2,<3.1.0 
    │   ├── cymem >=2.0.2,<2.1.0 (circular dependency aborted here)
    │   └── murmurhash >=0.28.0,<1.1.0 (circular dependency aborted here)
    ├── pydantic >=1.7.4,<1.8 || >1.8,<1.8.1 || >1.8.1,<1.11.0 
    │   └── typing-extensions >=4.2.0 
    ├── requests >=2.13.0,<3.0.0 
    │   ├── certifi >=2017.4.17 
    │   ├── charset-normalizer >=2,<4 
    │   ├── idna >=2.5,<4 
    │   └── urllib3 >=1.21.1,<1.27 
    ├── setuptools * 
    ├── smart-open >=5.2.1,<7.0.0 (circular dependency aborted here)
    ├── spacy-legacy >=3.0.10,<3.1.0 
    ├── spacy-loggers >=1.0.0,<2.0.0 
    ├── srsly >=2.4.3,<3.0.0 
    │   └── catalogue >=2.0.3,<2.1.0 (circular dependency aborted here)
    ├── thinc >=8.1.0,<8.2.0 
    │   ├── blis >=0.7.8,<0.8.0 
    │   │   └── numpy >=1.15.0 (circular dependency aborted here)
    │   ├── catalogue >=2.0.4,<2.1.0 (circular dependency aborted here)
    │   ├── confection >=0.0.1,<1.0.0 
    │   │   ├── pydantic >=1.7.4,<1.8 || >1.8,<1.8.1 || >1.8.1,<1.11.0 (circular dependency aborted here)
    │   │   └── srsly >=2.4.0,<3.0.0 (circular dependency aborted here)
    │   ├── cymem >=2.0.2,<2.1.0 (circular dependency aborted here)
    │   ├── murmurhash >=1.0.2,<1.1.0 (circular dependency aborted here)
    │   ├── numpy >=1.15.0 (circular dependency aborted here)
    │   ├── packaging >=20.0 (circular dependency aborted here)
    │   ├── preshed >=3.0.2,<3.1.0 (circular dependency aborted here)
    │   ├── pydantic >=1.7.4,<1.8 || >1.8,<1.8.1 || >1.8.1,<1.11.0 (circular dependency aborted here)
    │   ├── setuptools * (circular dependency aborted here)
    │   ├── srsly >=2.4.0,<3.0.0 (circular dependency aborted here)
    │   └── wasabi >=0.8.1,<1.2.0 
    ├── tqdm >=4.38.0,<5.0.0 
    │   └── colorama * (circular dependency aborted here)
    ├── typer >=0.3.0,<0.8.0 (circular dependency aborted here)
    └── wasabi >=0.9.1,<1.1.0 (circular dependency aborted here)
tiktoken 0.3.3 tiktoken is a fast BPE tokeniser for use with OpenAI's models
├── regex >=2022.1.18
└── requests >=2.26.0
    ├── certifi >=2017.4.17 
    ├── charset-normalizer >=2,<4 
    ├── idna >=2.5,<4 
    └── urllib3 >=1.21.1,<1.27

Error on deploy.sh file

I found some small issues in the deploy.sh file.

  1. missing shebang line
#!/bin/bash
  1. In line 3 given path is incorrect.

    ai-tools/deploy.sh

    Line 3 in e8c41e3

    cd text_translation/ai4bharat & sh deploy.sh & cd .. & cd ..

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.