briggysmalls / detectordag Goto Github PK

Currently the API sets Access-Control-Allow-Origin to *.

Realistically we can restrict it to the frontend hosting domain.

Currently the frontend shows "Updated" and a date.

This should be converted in an easier to understand format. If the device hasn't been seen for a while, the power status should be thrown into doubt. Something should indicate visually that the power status is "out of date".

The API should remove as many 'fatal' calls as possible, instead returning an error response

Currently there is no way to create an account nicely. Not even manually.

#12 relies on an account having a validated email account, and this should probably be solved here first.

Obviously there is no good having just anyone register, because they'll need a device for the site to be any use.

There could be an invite-only account signup system:

1. Restricted route exposes a 'register' form that asks for an email and password
2. Account created and verify email sent

More prefereably:

1. Restricted route exposes a 'register' form that asks for email only
2. Account created with empty password and verify email sent
3. Verify link redirects to create password page

The second option relies on #30 being fixed

Currently a generic AWS SES verification email is sent when verifying (#19).

This will be confusing to users, so a custom email template should be used.

There should be a frontend that will ultimately provide a few synchronous features:

• Check status of all edges associated with an account (e.g. power/no-power, battery level, last-seen)
• Update contact emails

There will need to be some sort of authentication, which then probably gives rise to an password recovery/reset via email service 🙄 🔫

Everyone loves the little sharing descriptions that pop up in whatsapp ;)

I did try this briefly and was unhappy with the complex syntax when trying to write a subquery.

However perhaps in this simple application there isn't really a need for such complex queries.

The query I have got at the moment is as follows:

SELECT email
FROM emails
INNER JOIN accounts
ON accounts.id = emails.account_id
WHERE accounts.id = (
    SELECT accounts.id
    FROM accounts
    INNER JOIN devices
    ON accounts.id = devices.account_id
    WHERE devices.id = UUID_TO_BIN(?, true)
)

But really, I probably want to break this into two parts, because I should check that a device is associated with an account (the subquery) before moving on to fetching the emails for it? Perhaps? It might be useful to catch in the future devices trying to operate when they don't have an account.

gorilla/mux has, unsurprisingly, an ability to add middleware.

The boilerplace of adding:

• Content-Type header
• Authenticating

Should be put into middleware so that it can be applied across requests.

There is a beta AWS IoT service for fleet provisioning.

There are two options:

1. Embed a single provisioning cert into all devices which it uses to register and obtain full certs with.
2. Allow an authorised user (i.e. a detectordag customer) to register a device at setup.

I had thought I'd do the 1st, however it is true that #46 remains open. If #5 was actually implemented for solving creating an account, then this device registration could occur at the same time.

It's quite complex, but also the most automated method for solving all the simultaneous concerns

AWS errors seem to have more information in them that I'm currently printing.

A user should be able to re-register a device. This would happen when 'resetting' it.

AWS will succeed if the device already exists, and that is fine.

However, as it stands, any existing certificates attached to the device are not deleted. This will mean a build of up obsolete certificates. These should be cleaned up when re-registering a device.

This need:

• authentication (check password in accounts and issue JWT?)
• get devices (this may be called asynchronously in a refresh action)
• get/set contact emails
• get/set device name

A later iteration may expose an actual request to the device to send an update/ping to see if it’s alive.

In order for the dashboard (#10) to have information to display, the consumer will need to store the received status into the database.

I'm interested in things like balenaOS or AWS IoT for managing fleets of IoT devices

Currently the unit tests on the device end of the system are pretty light.

If implementing #11 or other features it would probably make sense to introduce unit tests for the EdgeApp class, with the CloudClient injected as a dependency.

The pi is supposed to use a dongle. It should be checked if Balena will access WiFi by USB.

Currently there is no API route for invalidating a JWT token. So when the frontend logs out it simply removes the token, but it would remain valid if kept.

Instead there should be an API route that updates the contents of the token to indicate it is invalid.

The edge device can check the battery level.

This might be useful for estimating at what point the device might be lost after power is lost.

AWS IoT seems to be ready for handling authentication of devices and other stuff.

Using rabbitMQ would be more 'rolling my own'.

It could be good experience to understand what it is that AWS IoT offers.

Currently the emails only display the device ID which is useless for humans.

Instead we should fetch the name from the database to display in the emails.

Radical idea.

In thinking about device provisioning I got into a bit of a twist. How to create a device table row and an AWS IoT 'thing' and make sure they both end up with the same 'name'. How to keep them synchronised.

In fact, I'm probably going about this all wrong. The 'thing' is essentially a specalised database row. AWS is already managing my things. I shouldn't have to keep track of them in a database in parallel!

Proposed approach

Use attributes on a 'thing' to keep track of what account a device is assigned to. Attributes are specifically designed so that things may be searched for a given attribute value: exactly what the API needs to do when finding devices for a given account.

It should be trivial for the consumer to get the account ID from a device vs. from the database.

It looks like AWS IoT thing types are for a common type of thing, e.g. lightbulb.

You can set attributes on a type that are present on all things of that type. A thing can only have one type.

groups are more loose. A thing can be in multiple groups. I think there is less call for 'groups' currently in my application. Unless every user account is a group 🤔 .

Regardless, currently all things are added to the detectordag group which is useless and should be stopped.

CORS protocol would say that the frontend ought to send the Origin header to say who it is.

However because I control the server, this isn't a massive deal.

The system should make it possible for a user to reset their password

The method for configuring a pi (associating with an account, connecting to WiFi) could be to boot as an access point.

The user could then connect to the device via computer/phone on a static IP and submit account information.

If a pi needs to be connected to another network (e.g. slave? or connecting to house WiFi), then it could be submitted here too.

I ❤️ structured logs!

The API is quite dilligent at responding with the right HTTP status code:

• 400 for bad request
• 401 when the auth token has not be supplied correctly
• 403 when the token is expired/invalid access
• 500 when the server crapped itself

However currently any failures whatsover redirect to login, which is quite rubbish.

The return code should be checked. Probably only 403 of the above is relevant. The rest are client or server errors.

This feature takes messages to 'send an email' and will... send an email.

This should probably be its own, stubbable program

It turns out if emails are already verified they still received the verification email.

So we need to check prior to sending the request to verify.

The API now exposes the GET and PATCH methods for an account, allowing a user to update their contact emails.

This should be utilised by the frontend.

We will wish to have username/name displayed as a 'logged in user' indicator.

/accounts/{accountId} get request should be updated to return this.

Derivative of #3

The edge device should periodically update the shadow with the current status.

This is mainly just an 'I'm alive'. However it would be useful for battery updates when #11 is introduced.

The API currently displays the AWS-reported 'current' time, rather than the time that the power status was updated.

This is probably a bit annoying to do neatly: every shadow property has its own metadata. However for the time being we can just use the power status (this is the only device shadow property thus far).

The frontend is currently served over plain old http.

A lot of development was done without any testing.

I started writing more tests after a while, but there are some notable gaps.

DynamoDB, being NoSql, does not sport auto increment keys. This would make integer fields for account ID awkward to create more of.

Better to use UUIDs. Furthermore then there are less security risks if we expose these as resource IDs in the web API

Ideally a user should be able to name their device. If they have multiple devices on their account then they'll want to differentiate them when the statuses start changing.

The PATCH on an account should return the new account resource status - the equivalent result as a GET.

Locally the verification emails seem to be sent fine.

However when deployed the API doesn't seem to send them correctly.

Could this be to do with the rate limit of 1/s? Or something else?

Whilst we could use dynamodb to manually record abs maintain device state, this seems to reimplement AWS IoT device shadow features.

This souls be investigated as it is probably more robust. But is it added cost?

Bcrypt is slow by design.

But this makes the authentication lambda slow. It currently takes approximately 4s to complete the auth request. Whereas fetching all devices is just 80ms.

It might be better to use a different hashing algorithm. It's not like this is the most sensistive application.

It might be useful for users to see which emails have been validated, so they know which emails to expect to receive resposes on.

There might be two sensors on a site, where only one needs the WiFi dongle, the other skills just connect to it.

Determining which is which could be done by simply checking the presence of an ‘Ethernet’ port.

Where it gets complex is security. How does the slave know the network login of the master? We have two entities where we can’t really trust either, and a trusted 3rd party (the server).

I’m thinking Something a bit like the asymmetric aspect of TLS:

• The devices will al have certs of server baked in.
• The server will be able to verify the public keys of all devices (recorded at ‘manufacture’)
• slave asks for WiFi details over some other channel (Bluetooth?)

Bootstrap allows us to highlight the active tab. This should be done for better navigation.

Before SES can send an email to an address, the address first needs to be verified.

This can be done using the SDK.

The update account API call needs to not just save new emails into dynamodb, but also send instruction to verify these emails using SES.

It would be nice to make a device name configurable.

Derives from #3

Once #40 is implemented, some other service will need to check for devices that haven't been seen for a while and send emails.

This sounds inefficient (a scan) and so I wonder if there are better ways of doing this?

It is currently a bit of a faff deploying the 4 elements:

• frontend
• edge
• consumer
• api

This should be done via CD with circleci

To gain confidence that we haven't missed anything, the sensor should probably send 'I'm alive' messages. I think this is our only way of knowing that the internet hasn't gone down. I think regardless of whether we are using a SIM or the internet, this is probably pretty important.

The consumer should then store a 'last seen' timestamp against each device.

The db-cron job should periodically check for missing devices and send notification emails if not already (requires some 'has_emailed?' state to be stored in the db)