Our tool is a web browser extension that is triggered once an image is clicked on, the image URL gets saved and then using Azure ComputerVision a text description of the image is generated. After this, we implemented Azure text-to-speech and used a couple of Azure functions to play the audio file generated from Azure through the extension so it can be heard by the user. The extension basically uses AI to describe what is happening in an image, then uses Azure services to generate an audio file that gets outputted by the extension.

Demo Video: Video Link

Powerpoint Presentation: PPTX Link

Business Requirement Document / Proposal PDF Link

Dev Spec: PDF Link

Demo of the CognitiveService-based Features: LINK

Architecture/Infrastructure diagram for creating an Edge Browser Extension that generates and plays audio-descriptions of images on a webpage upon user interaction with the image.

Note that to fit the structure of an Edge Extension, some additional infrastructure was deployed.

1. Azure Functions to turn every CognitiveService-based feature into an endpoint that can accept HTTP requests.
2. Deployment of an Azure Storage Blob Container to store the generated audio files.

Looking forward

1. Need to have remaining secrets stored in an Azure Keyvault
2. Deploy a custom App Service Domain to host all function apps created under a shared domain

Extension

ComputerVision Service Function App

OpenAI Function App

Text-to-Speech Function App

Extension/Frontend

HTML5, Edge Extensions Framework

Azure Cognitive Services

Azure OpenAI Services (Utilized GPT davinci model)

Cognitive Speech Services

Cognitive Service for Vision

Libraries/SDK

Semantic Kernel SDK

FFmpeg

Deployment

Azure Functions

Azure Storage (Blob storage container)

Handwritten Proposal of Audio-description generation proposal (for vidoes) by Sofia

Link

Proof-of-concept Files Created for video-based Audio-description Generation Link

Implemented scripts to generate POC of Audio-descriptions. Link