Giter Club home page Giter Club logo

meetbot's Introduction

The idea for meetbot came to us from our internal company hackathon. We wanted to make meetings better for our peers and we built a bot that could join scheduled meetings, perform actions and provide the features that Google Meet sorely was missing. Meetbot performed well at first but with changes coming into Google Meet's UI. Our features started breaking as Meetbot is powered by puppeteer automating a browser instance to do its bidding. With maintainence burden ever increasing and Google Meet introducing official support for features. We decided to archive this project. I have tried to document the project as much as I could and it would be great to have a maintainer interested in keeping this alive. But, till then this is the end of the line for Meetbot.


TLDR: Meetbot has increasingly become hard to maintain with changing Google Meet UI and features breaking. Hence, the project has been archived. The project is open to new maintainers as well as individuals looking to build further on it. Reach out to @vipulgupta2048 for a briefing or any questions you have.

[Archived] Meetbot

Meetbot is a Google Meet bot that makes your meetings frictionless. We've all been there, we've all heard these questions being repeated over and over again on video calls:

  1. Can you hear me?
  2. Can you resend those links?
  3. Do you have the notes from the call?
  4. Was the meeting recorded?
  5. Does anyone have the links shared in the meeting?

Meetbot helps you out by:

  1. Validating audio input of attendees minus the awkward silence.
  2. Recording chat transcript.
  3. Recording voice caption transcript.
  4. Chat commands to resend transcripts for folks joining late.
  5. Auto-record meetings.
  6. Auto-join meetings using Google Calendar API.
  7. Saving it all to Google Docs for easy sharing!

And, many more features.

Try it out!

To deploy this project on balenaCloud, use the button below.

balena deploy button

Configuration

By default, meetbot will join meetings as an unauthenticated user and won't be able to perform some features. To enable all features, follow the authentication instructions.

Environment Variable Description Default value
GOOGLE_PASSWORD Password of the Google account meetbot uses for running authenticated features NA
GOOGLE_EMAIL Email address of the Google account meetbot uses for for running authenticated features NA
GOOGLE_TOTP_SECRET If the Google account has 2FA security, then the TOTP secret that is configured for 2FA goes here NA
HTTP_PORT (Optional) Port on which the meetbot server starts running. For balena devices, the server needs to run on port 80 80
MAX_BOTS (Optional) Maximum number of meetbots to run parallely on the server 5
GOOGLE_CALENDAR_NAME (Optional) The Google calendar ID that meetbot parses to auto-join events in that calendar NA
GREETING_MESSAGE (Optional) Greeting message which is posted when meetbot joins the Google Meet "Hello folks, it's your favorite bot, hubot!!"

Read more about variables in balenaCloud Dashboard.

Getting Started

After getting the bot server up and running, it will start listening for requests to provision new meetbots.

To get a meetbot to join your meeting, navigate to the Meetbot dashboard using the public device URL or the port you configured for the server to run on.

Click the "Join" button and enter the Google Meet URL in the modal that opens. Click the "Join meeting" button and wait for a few seconds for the bot to join the meeting. When the bot joins, it will post the configured greeting message on the Google Meet chat. This means, the meetbot is ready to go!

Running the bot locally

After cloning the repository, install the dependencies:

npm ci
HTTP_PORT=8080 npm start

The bot will now be running but functionality is limited until the bot is authenticated. To get a bot to join a Google Meet, head to the Meetbot dashboard available on http://localhost:8080 and click the Join button. The project is configured using dotenv. To configure the bot, add the variables to a .env file in the root of the project with the variables listed above.

Development

Meetbot is based on a plugin event driven architecture. Most features work independently either creating events or using events to perform a specific function. To add new features to meetbot, start by adding it in src/meetbot/features. Check out other features for help in understanding how things work. After finishing your changes, run the bot locally to check if your feature loads and test your changes.

To develop the UI, run npm run dev-ui. This will spin up the development server. You should then be able to access the dashboard at http://localhost:3000/. Note that the data you will see on the tables is only mocked and no actual requests are sent to the API. This makes it easier to develop the UI independently.

To create new chat commands for meetbot, start by creating a file in src/meetbot/features with the name chat-command-<feature-name>.ts. Use the chat event and parse the data in order to perform a specific action when a specific chat message is sent. Don't forget to add a comment in tsdoc format to indicate what the chat command does. These files are parsed by the help chat command and are used to populate the help message automatically. The help message can be triggered using the using the /help chat command.

Authentication

Authentication for meetbot is 3 phased and crucial to run the following features:

  1. Record meetings
  2. Create Google Docs to save transcripts
  3. Join Google Meet automatically without the "Allow User" prompt
  4. (Optional) Check the Google calendar to join meetings automatically

In unauthenticated mode, the bot will only be able to join meetings as an unverified user and performs limited functionality. To verify and join a Google meet as a user, the meetbot will need login credentials to a Google account (GOOGLE_EMAIL and GOOGLE_PASSWORD). If the Google account has 2FA security enabled, then the TOTP secret that is configured for 2FA would also be required (GOOGLE_TOTP_SECRET). This will help the bot join the Google meet as a user.

Since, recording meetings is not a free feature available outside of the Gsuite organization. The meetbot user being created preferrably should be a member of the Gsuite organization. This will also allow meetbot to join meetings automatically without the "Allow User" prompt.

Finally, for integrations with Google docs and calendar you must download the credentials file containing data for oauth2 flow. This is used to authenticate requests to the Google API. See the following steps to create the credentials needed or watch this video.

Troubleshooting: https://stackoverflow.com/questions/58460476/where-to-find-credentials-json-for-google-api-client

After following the steps, run the command below and follow the instructions to generate a token.json file. This authentication process is one time only and after this the token.json file will be used for any further authentication process.

ts-node src/google/create-token.ts

Deployment

To activate all the features of meetbot, authenticate the meetbot and create the token.json file before deployment.

To deploy to your device with the help of balenaCloud, use balena CLI's balena push command as stated below. For more information check out the balena docs.

balena push <Name of fleet>

Credits

Meetbot is free software, and may be redistributed under the terms specified in the license.

meetbot's People

Contributors

20k-ultra avatar somombo avatar tmigone avatar vipulgupta2048 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

meetbot's Issues

[Bug] Meetbot queue should only count active meetings

Since we are storing all bots in the BOTS map object, we reach the max limit of bots pretty fast which leads to meetbot not able to join any other meetings.

14.03.22 21:35:58 (+0530)  main  queuing: starting the recording...
14.03.22 21:35:58 (+0530)  main  Current bot queue size: 5

Find a new home for meetbot

Consider running the bot on a Raspberry Pi 4. Alternatively, ask ops for a spare NUC in one of the offices.

Feature request: Search for people in meets

Meetbot(s) could pass back the information to the collective intelligence (Our server) over which group of people are currently in which meet? Maybe we keep these to scheduled meetings only as I don't like the idea of snooping into ad-hoc meetings.

The information can then be listed in the dashboard and probably can be searched too.

participants always 0

I saw the bot with latest code always said participants were zero. I think the selector is wrong.

Validate meet URL

The bot will say it will go to a page even if it's not a meet url. We could make it smarted to know it can't join a meet if it isn't for the meet.google.com domain.

UI improvements checklist

  • Add button to leave the meet
  • Add link to chat transcripts (The links would be represented as - Voice // Chat )
  • Rename column "Transcripts URL" to "Transcripts"
  • Remove button to join the call, since there is already a link to the meet below the ID (Make that link clickable)
  • #26
  • #25
  • Clean up code / refactor
  • #24

Meetbot won't join meetings again

At the moment, Google Meet doesn't have an option to join meetings again. For recurring meets having the same meet link every week. This will be a problem.

Screenshot from 2022-03-14 21-40-13

Handle Google API service unavailable error

On one meetbot run, the docs transcript was not created and the log is below. Maybe Somo and Andrei can look into handling this for next time.

Running caption feature..
07.12.21 19:49:27 (+0530)  main  Attached Chat Saver
07.12.21 19:49:27 (+0530)  main  Introducing hubot in the meet - feature
07.12.21 19:49:27 (+0530)  main  deactivating Jellyfish integration because of missing JELLYFISH_AUTH_TOKEN env
07.12.21 19:49:27 (+0530)  main  Running raw caption feature..
07.12.21 19:49:27 (+0530)  main  Running record feature..
07.12.21 19:49:27 (+0530)  main  Running storage feature..
07.12.21 19:49:27 (+0530)  main  Attached Transcript Streamer
07.12.21 19:49:27 (+0530)  main  Current bot queue size: 1
07.12.21 19:49:28 (+0530)  main  typing out email
07.12.21 19:49:30 (+0530)  main  typing out password
07.12.21 19:49:31 (+0530)  main  doing 2FA login
07.12.21 19:49:42 (+0530)  main  going to Meet after signing in
07.12.21 19:49:46 (+0530)  main  turn off cam using Ctrl+E
07.12.21 19:49:49 (+0530)  main  turn off mic using Ctrl+D
07.12.21 19:49:52 (+0530)  main  Changing layout to Spotlight mode
07.12.21 19:49:57 (+0530)  main  turn on captions
07.12.21 19:49:57 (+0530)  main  Chatlog joined a meeting https://meet.google.com/ipq-rsdm-vgm
07.12.21 19:49:58 (+0530)  main  Joined the meeting:  https://meet.google.com/ipq-rsdm-vgm
07.12.21 19:49:58 (+0530)  main  queuing: starting the recording...
07.12.21 19:49:58 (+0530)  main  (node:33) UnhandledPromiseRejectionWarning: Error: The service is currently unavailable.
07.12.21 19:49:58 (+0530)  main      at Gaxios._request (/usr/src/node_modules/gaxios/build/src/gaxios.js:129:23)
07.12.21 19:49:58 (+0530)  main      at runMicrotasks (<anonymous>)
07.12.21 19:49:58 (+0530)  main      at processTicksAndRejections (internal/process/task_queues.js:95:5)
07.12.21 19:49:58 (+0530)  main      at async OAuth2Client.requestAsync (/usr/src/node_modules/google-auth-library/build/src/auth/oauth2client.js:368:18)
07.12.21 19:49:58 (+0530)  main  (Use `node --trace-warnings ...` to show where the warning was created)
07.12.21 19:49:58 (+0530)  main  (node:33) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
07.12.21 19:49:58 (+0530)  main  (node:33) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
07.12.21 19:49:59 (+0530)  main  PAGE LOG: Autofocus processing was blocked because a document already has a focused element.
07.12.21 19:49:59 (+0530)  main  Meeting ipq-rsdm-vgm (2021-12-07T14:19:57.962Z) Chat is available at 
07.12.21 19:49:59 (+0530)  main  https://docs.google.com/document/d/1hJSzybzqc7s5VZUZa7EtxkrkVlZls7DG6yJpiHJRjhE
07.12.21 19:49:59 (+0530)  main  setting chat transcript url tohttps://docs.google.com/document/d/1hJSzybzqc7s5VZUZa7EtxkrkVlZls7DG6yJpiHJRjhE
07.12.21 19:50:00 (+0530)  main  open people list to activate feature
07.12.21 19:50:01 (+0530)  main  PAGE LOG: [stenographer] loaded - Debug: false
07.12.21 19:50:01 (+0530)  main  PAGE LOG: [stenographer] init called
07.12.21 19:50:01 (+0530)  main  captions are on
07.12.21 19:50:02 (+0530)  main  streaming captions
07.12.21 19:50:03 (+0530)  main  open chat section and send a message to all
07.12.21 19:50:05 (+0530)  main  close chat section again
07.12.21 19:50:07 (+0530)  main  starting the recording...
07.12.21 19:50:15 (+0530)  main  open chat section and send a message to all
07.12.21 19:50:19 (+0530)  main  close chat section again

Change /reping command to /links

reping is not clear what is being pinged again. Since you are trying to get links a command titled /links makes the most sense.

Modify the command to also send all links it has created (voice and chat transcripts)

Meetbot can't login

Meetbot failing to "sign in to Google" and doesn't work due to this precautionary check by Google shown when signing in.

exception

Workarounds,

  • We can handle this by clicking the button and signing in when the screen shows "Verify it's you"
  • OR, we can terminate the running browser session and let meetbot sign in into a new browser instance preferably without cookies or an authentication notice. That what I will do by restarting the container,

The error:

07.03.22 20:28:37 (+0530)  main  we're already logged in
07.03.22 20:28:37 (+0530)  main  going to Meet after signing in
07.03.22 20:29:15 (+0530)  main  Unrecoverable bot error occured: waiting for XPath `//*[contains(text(),'call_end')]|//*[contains(text(),'Join now')]` failed: timeout 30000ms exceeded
07.03.22 20:29:15 (+0530)  main  Leaving the meeting: https://meet.google.com/yhc-fmfq-hxx
07.03.22 20:29:52 (+0530)  main  typing out email
07.03.22 20:30:22 (+0530)  main  Unrecoverable bot error occured: waiting for selector `input[type="email"]` failed: timeout 30000ms exceeded
07.03.22 20:30:22 (+0530)  main  Leaving the meeting: https://meet.google.com/yhc-fmfq-hxx
07.03.22 20:30:42 (+0530)  main  typing out email
07.03.22 20:31:12 (+0530)  main  Unrecoverable bot error occured: waiting for selector `input[type="email"]` failed: timeout 30000ms exceeded

Impact:
Meetbot doesn't work until this is fixed or workarounds are deployed

DEBUG mode to click screenshots

Triggering the debug mode or a certain log level if decide to work on #13 will click pictures of each step to debug steps easily on meetbot run

handle secrets properly

My PR #37 is using .env for secrets in production which isn't the best way to go about them but it's better that they are all in one place and at least documented.

Stenographer returns the same names and images for caption events

Stenographer has broken the meet transcription feature. While debugging, I found that even while having multiple users on the meet then the transcription was being generated using the same name and image of the person. The data received from the problematic caption event is mentioned below.

{
  image: 'https://lh3.googleusercontent.com/a-/AOh14GigH5zYWQA4OaIiQhNQhsqzrrOjBUvvbWDdrTuv=s40-p-k-no-mo',
  person: 'Vipul Gupta',
  text: 'Hello, checking the transcript,  checking the transcript. ',
  startedAt: '2022-05-10T16:01:15.043Z',
  endedAt: '2022-05-10T16:01:28.044Z',
  id: 'dea879a21a26'
}
{
  image: 'https://lh3.googleusercontent.com/a-/AOh14GigH5zYWQA4OaIiQhNQhsqzrrOjBUvvbWDdrTuv=s40-p-k-no-mo',
  person: 'Vipul Gupta',
  text: 'He transcript from another user. ',
  startedAt: '2022-05-10T16:01:15.043Z',
  endedAt: '2022-05-10T16:02:28.043Z',
  id: 'dea879a21a26'
}
{
  image: 'https://lh3.googleusercontent.com/a-/AOh14GigH5zYWQA4OaIiQhNQhsqzrrOjBUvvbWDdrTuv=s40-p-k-no-mo',
  person: 'Vipul Gupta',
  text: 'Checking the transcript now.  Why is the photo?  Same text same. ',
  startedAt: '2022-05-10T16:01:15.043Z',
  endedAt: '2022-05-10T16:03:07.043Z',
  id: 'dea879a21a26'
}

Storing calendar events in a file to track status

At times, you can't find the current status of what hubot's next calendar event will be. This can be rectified locally at least by writing the calendarEvents to a file which will make debugging easier.

Consider adding retries for page navigation

We hit navigation timeout quite a lot, and it is great to wrap these in a retry. I was thinking to use this https://www.npmjs.com/package/async-retry

28.12.21 19:56:37 (+0530)  main  ERROR in meet! Exiting... https://meet.google.com/oxd-hqer-qno TimeoutError: Navigation timeout of 30000 ms exceeded
28.12.21 19:56:37 (+0530)  main      at /usr/src/node_modules/puppeteer/lib/cjs/puppeteer/common/LifecycleWatcher.js:106:111
28.12.21 19:56:37 (+0530)  main      at async FrameManager.waitForFrameNavigation (/usr/src/node_modules/puppeteer/lib/cjs/puppeteer/common/FrameManager.js:128:23)
28.12.21 19:56:37 (+0530)  main      at async Frame.waitForNavigation (/usr/src/node_modules/puppeteer/lib/cjs/puppeteer/common/FrameManager.js:441:16)
28.12.21 19:56:37 (+0530)  main      at async Page.waitForNavigation (/usr/src/node_modules/puppeteer/lib/cjs/puppeteer/common/Page.js:1218:16)

Abstract out Google services authentication

I don't really like how complex the Google authentication is. This function checks for credentials but the GoogleCalendar implementation also performs some authentication (creating as token). It's become hard to follow how authentication is done so if a token.json file is what all these google APIs use then I think 1 single place should create it and the rest of the code doesn't have to always check if it exists and create if not...not worth doing in this PR but wanted to mention my thoughts about this.

Originally posted by @20k-ultra in #56 (comment)

add balena-ci

We talked to Stathis about this, and the quick and easy way to add CI to this repo is to move it to an org that has CI access. Playground doesn't and CI access can't be activated for specific repos, it's an org-level thing.

So maybe we do a product call and this goes into people-os probably?

Emit left whenever meetbot leaves a meeting

Meetbot only emits a left event when the participants reaches 0. There are however other ways meetbot could leave a meet and it should also emit a 'left' event:

  • meetbot gets kicked from the meet
  • the rest API tells the bot to leave

Fix Meetbot queue to track active meetbots currently

From #56, we have made meetbot's meet history persistent. Now, https://bit.ly/getmeetbot shows all meetings that happened in order for it to surface transcripts better. This caused a regression where as the meet queue has become persistent. The meetbot logic to track active meetbots wasn't patch to actually track active meetbots. Rather, it's tracking both active and dead meetbots together.

This leads to an error where we hit our set limit of MAX_BOTS (Atm set to 10) and then no meetbots can be deployed. We need to fix how we calculate active number of meetbots in order for the MAX_BOTS limit to function.

Workarounds:

  1. To prevent the bot queue from filling up we can restart the meetbot container couple of days to reset it
  2. Take MAX_BOTS limit to 100 or something

[Feature Request] Create /reping command

Context is lost for folks that join after hubot has sent the transcript links. With the /reping command, people can get access to both the past chat as well as the transcripts. Maybe this makes the chat a little more noisy with all the links but it will kill the dialogue:

"Oh hey Ryan, we sent the links before you joined. Let me send them (all) again."

Voice transcript broken due to Caption Settings cog

Screenshot from 2022-02-07 12-55-27

Google Meet has implemented this settings tab in the closed captions that is breaking the Google Doc feature with the following error. We probably need the stenographer to ignore this settings tab and keep sending out caption events properly.

(node:68795) UnhandledPromiseRejectionWarning: Error: Invalid requests[0].insertInlineImage: The URL should not be empty.
    at Gaxios._request (/home/vipulgupta2048/work/voicebot/node_modules/gaxios/build/src/gaxios.js:129:23)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at async OAuth2Client.requestAsync (/home/vipulgupta2048/work/voicebot/node_modules/google-auth-library/build/src/auth/oauth2client.js:368:18)

When this error occurs, here's what we see in the Google Docs https://docs.google.com/document/d/1AA01PnHlW4QjzwtOClqZSsJdiA0Vo05A9ytOkWIxJVQ/edit?pli=1

Allow browsing historical meet data

Currently once hubot leaves a meet the meeting listing is removed from the UI.
Would be nice to have a toggle or a "past meetings" section where you can go and grab old transcripts for example if you forgot to copy the link.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.