alperbahcekapili / roach Goto Github PK

At each step of user's stt result, a function matching should be run. For example if we have the following functions in the car:

• Name: Open music

  • Description: This function plays a track on the car
  • Possible Phrases: ["Turn on a music for me", "Play a song for me"]

• Name: Open the windows

• Description: Opens the car windows

• Possible Phrases: ["Open the left window for me", "Open the driver's window"]

and if the user says: "I am bored. Play a song", then open the music function should be matched. Then this function should be called at the background.

Having an exit function that terminates the dialog would be good.

Assistant should be informed about the match before generating an answer.

Technical Details:

There should be vectordatabase which stores all possible phrases and their functions' name, descriptions in the metadata fields.
A similarity search should be executed at each step.
Similarity treshold (st) can be set via experimentation.

We can store functions and their informations in a json file under backend folder.

In order to show the latest-matched functions, we can add indicators to assistant's messages to visualize which function is matched/no function is matched.

We need to do adjustments to visualize the state of the car as well

Chatting logic should be started upon sleepy driver detected.

After signal:

1. Assistant should be triggered to warn the driver
2. After warning driver's mic should be activated and response should be recorded
3. Recording should be transcribed and send back to the front-end
4. Front-end then should trigger assistant with driver's response
5. Iterate this loop until user want to exit(driver should say sth. like "Exit")

Car's state should be generated as if the values are being read by sensors in the car.
Example data:

{ "left-front-window":100 "right-front-window":100 "left-rear-window":95, "right-rear-window":95, "speaker-level":0, "temperature-inside":20 }

This data should be generated at each 0.1 seconds and matching-functions should invoke changes in the car interior. To do that, another backend endpoint can be created. At each timestep t, front-end can send a request to visualize the state of the car in the frontend.