seanfhear / iotea Goto Github PK

No support in Mongoose :(

Wrap some of the API functions that we'll need in a separate class to avoid cluttering init.js

First steps of #11
Use PlantCV to perform plant detection and segmentation. Their docs provide a sample VIS workflow which does this. The segmented the plant will be useful for trying to measure the height of the plant.

As a first step of issue #12 we want to set up espressif's tooling for building and flashing firmware to the ESP32. The reasoning for this is due to the ESP-WHO sample project provided by espressif that runs on the ESP-EYE and can be modified to suit our needs. This may be able to be written using Mongoose OS however the ESP-WHO project uses camera firmware provided in the ESP-IDF sdk and so this would need to be rewritten.

Upsides

• Less dev time
• Easy to modify example project
• Camera reading and buffering handled by sdk

Downsides

• Different tooling and build cycle than other device
• Much more work to do OTA updates if needed

...Please don't use up all my money :'(

Background

Write the initial firmware for the ESP-EYE. May be able to use Mongoose OS with it but espressif have an example program which we could use as a start. The example program performs face detection and voice recognition on the device itself, while also running a HTTP server to stream the video. I also found another example project that uploads the photos to Azure to process the images. This may be a good place to look for "inspiration".

Tasks

• Create HTTP server to stream video (to see if camera placement is right)
• Upload photos to some online storage (will need to integrate with the AWS work. I believe @CSIGildea mentioned seeing talk of picture uploads over MQTT in his brief research)

Recommended approach

Use espressif's tooling and the examples in ESP-WHO as a starting point. This uses code from the ESP-IDF sdk for reading from the camera which would need to be rewritten to work with Mongoose OS so not recommended (this leaves us with issues if we wanna update the camera OTA).
Either use the example programs and flash those using the espressif tools or try to recreate the same code using Mongoose OS. Mongoose OS has some documentation for HTTP servers on the device. There is also a table of the camera connections to the esp here.

Look at possible options, likely Telegram and Slack.

There are some memory issues needed to be improved upon and fixed. It can crash due to the task watchdog being triggered. Sometimes photos are too large to send. Some improvements can be made.

Related to #12

Stretch Goal: If a plant was meant to actuate the water pump however on next soil moisture reading no increase in soil moisture has been detected, trigger a lambda function, which would notify plant stakeholder on telegram.

Add ability to change device identifiers from config, to a user configured plant identifier value

e.g. "optiplex_790_MTea" over existing device id.

Background

We want to be able to detect if the plant needs to be turned. The idea being if the plant is leaning towards the window we may want to rotate the plant to ensure it grows straight to avoid it toppling over in the future. The plan is to take a picture of the plant every day, or couple of days (#12), calculate statistics such as height (#25) and to detect if the plant needs to be turned. The motor will be used to turn the plant if needed.

Tasks

• Check if plant is leaning from segmented objects from #12.
• Determine a valid threshold for plant leaning
• Send message to actuate on result

Recommended approach

We could try to fit a line of best fit to the detected objects and determine if the curve of that line is too severe or too much to one side. We could also try to detect the amount of leaves (green) to the left and right of the center of the pot and determine if there are too many leaves on one side. Either way the camera should ideally be parallel to the window to determine if the plant is leaning towards it. The image below describes what I mean.

Want to have a bot created that is able to send some readings to a Telegram chat.

Getting MAC Address for use in MQTT Topics.

For use in manual identification of plant dedicated ESP32 boards from MAC addresses.

Could make this more advanced/automated but likely not worth effort compared to new features.

Set up an Action which will use mos to build firmware when the repo is updated

Look at options for automatically sending firmware updates over the air when the repo is updated,

Background

Mongoose OS has no library for controlling stepper motors so I'll have to write my own.

Stepper motors typically have a number of stators that when activated in a particular order, turn the motor a step.

The stepper motor we have has a control board with an IC that allows for a 5V line to be given and a control pin for each stator that can be connected to input pins on the ESP32 to control it.

Tasks

• Write a library that controls the stepper motor.

Background

This is a continuation from #10. Here the idea is to use the segmented plant objects captured to find the top and bottom of the plant to estimate it's height. We may need photos of plants with a white background as the segmentation picks out wood that is in the background of photos and considers it part of the plant. The photos also may need to be from closer to the plant as it loses some of the details of the thin stems and doesn't recognize them as part of the plant.

Tasks

• Add known scale to photos
• Detect and segment scale to compare to size of plant
• Improves segmentation added in #10
  • Change white balance
  • Change thresholds

Have a new command /water

Without any argument it responds with the water threshold for the plant associated with that bot.

Passing a number between 0-100 updates that plant's water threshold to the passed value.

Enabling device management control via mdash.

Stored via AWS IoT Core Rules, likely to DynamoDB table.

Needed Meeting Discussion: Periods of upload and how much we want to store

Currently when you @ a bot in the telegram group it doesn't respond appropriately.

The message is forwarded to only that bot (we know since Yoshi sent Michael the invalid user message when he tried and none of the other bots replied) so the problem is likely that the message the bot gets is the command and the tag for the bot instead of just the command.

Need to look at message payload and see if there is a different value that can be used, or some way of stripping the tag out.

Research methods for processing images of plants.
Investigate the appropriate measures for our use case.

Add another bot for one of the other plants.

We want to be able to have more than one bot account in a Telegram group to demonstrate that we can request the status of multiple plants with one command while also being able to talk to individual bots.

Will want to replicate the lambda function for a new bot which is tied to another plant.

Modify the water and turn actuation functions to store the time they were executed in the database. The status function for the bot can then return the last time these actions were triggered.

Need to:

• Setup cron lambda to trigger every X mins (X currently decided to be 30)
• Cron lambda needs to read all plant IDs in dynamo DB and trigger vitals request per plant id.
• Upon vitals message an AWS IoT rule needs to trigger a soil moisture threshold lambda function
• Moisture threshold lambda function needs to trigger water pump actuation when needed for the given plant (prototype actuation already built).

Create an AWS IoT Rule and DynamoDB to cache the last vitals per plant, with plantID being used as a primary key.

Usage:

• Prevents need to publish and subscribe to MQTT to get specific plant vitals.

Trade-off:

• Vitals aren't real time but within last time automation is ran (expected to be <30 mins).

😕

Right now the bot tells a user they're not verified even if they didn't send a command.

Change so that the unverified message is only sent on commands.

Works seamlessly with mos locally however issue is to figure out best way to generate and distribution certificates for each of our separate boards whilst we are in different locations.

Potentially this could just be a readme or a script to use each person's own AWS credentials to generate certificates.

First step in AWS IoT core integration

Build lambda function, to be triggered by computer vision component, to actuate the plant motor for better sunlight distribution.

Actuation should publish to relevant topic with the number of steps the motor needs to turn.

Add unit indicators to the bot's status output to give more clarity to the readings.

Moisture -> %
Humidity -> ?
Temp -> Celsius?

Actuate based on soil moisture values to trigger water pump per plant, returning number of seconds the water pump should run for.