jonesus / chatty-bear Goto Github PK

This could be used to trigger microphone recording and sending the current data to the server

https://www.npmjs.com/package/hark

Explanations for the underlying technologies
https://web.dev/media-recording-audio/

An example how to record audio from user, and how to send it to server
https://stackoverflow.com/questions/67229656/how-to-record-mic-from-client-then-send-to-server

As the task is for a emergent user interface, the interaction between the user and the service should be the main focus. We want to enable the user to have a conversation with an AI, so making the interaction as close to a real conversation is essential.

Technically, we can have multiple "tiers" of working interactions, with the latency between each participant's response improving further on

Approach 1:

• Record audio file from user on client
• Send audio file from client to our server
• Send audio file from our server to google
• Receive text response from google
• Send text response to ChatGPT
• Receive text response from ChatGPT
• Send text response to google
• Receive audio file from google
• Send audio file to client
• Play it back to user

We could technically simplify this to a possible optimum of
Approach 2:

• Initiate audio capture on client
• live-stream microphone feed to our server
• forward live-stream to google
• receive text from google
• Send text response to ChatGPT
• Receive text response from ChatGPT
• send text response to google
• receive audio file from google
• stream audio straight to user

Technically, this probably requires the use of websockets for data transfer instead of HTTP requests

Some inspiration:
https://github.com/vin-ni/Google-Cloud-Speech-Node-Socket-Playground
https://www.leeboonstra.dev/chatbots/building-your-own-voice-ai-4/

This seems like a good way to programmatically interact with ChatGPT

https://github.com/transitive-bullshit/chatgpt-api#docs

this could be a good example interface between the client and the server

https://stackoverflow.com/questions/67229656/how-to-record-mic-from-client-then-send-to-server

Example discussion:

Prompt: You should act as a psychologist, who is empathetic and tries to help me with anything I ask for. You should ask follow-up questions when appropriate, and try to have a productive discussion with me. Do not talk about being an AI. The discussion starts here: I am having problems with my motivation for studying

Response: Can you tell me more about what specifically is causing you to struggle with motivation for studying?

Prompt: I think the courses might be a bit hard for me currently

Response: If you're feeling overwhelmed, consider reaching out to your instructor or tutor for extra support. You can also break down course materials into smaller, manageable tasks or try studying with a group.