As a Event Org / User, I would add auto-labels and AI Summarisation about streameth-platform HOT 5 CLOSED

An update to this issue:

• The webhooks work, and is received from Livepeer.
• The OpenAI API works as well, though it does need improvements.
• Add three colums to MongoDB:
  • A link to the transcription text (which is the whole speech-to-text description) videoTranscription.
  • The summary made by the ai aiDescription
  • The labels made by the AI autoLabels

One of the challenges is to give green light to WhisperAI that the upload is illegible, and can be transcripted. One of my straightforward solution is to compare the name of the asset with the session. Only issue is that you are really reliable to the naming of the videos.

I still need to take a look at the standardized labels.

from streameth-platform.

Ideally the flow for this would be:

• user upload video to livepeer ( via our interface or via the clips page)
• we listen to livepeer webhook hooks ( https://docs.livepeer.org/developers/guides/setup-and-listen-to-webhooks) and see when an asset was created
• this triggers a process that runs this whole ai process and save everything to the database.

Important tech considerations:

• need to add a few new fields to the session model ( summary, tags)
• would be great if we can generate a set of standarized tags

@xannyxs

from streameth-platform.

This could work with Github Workflow, but we would be reliable to Github again.

I am also wondering in which package it should be put into...?

@pablovoorvaart

from streameth-platform.

So it think this process should happen in the new server. Its easy for the webhook to trigger an api request I think.

In regards to the package, it should prob live in av-tools as a standalone script that can be invoked by the server.

from streameth-platform.

Basic visualization of my plan

https://excalidraw.com/#json=Z6pLY_PY7LzhgXq4JJOVq,bCzfo53RNL5VgcOi3No56g

from streameth-platform.