oxfordhcc / aretha Goto Github PK

back end is still pending on initial impacts query.

restarting all the daemons doesn't help.

Currently, geos with lat/lon of 0,0 get rendered on the map just off the coast of Africa.

Currently "Mostly: 192.168.0.1"
Should be "Mostly github.com"

have api / capture ignore multicast addresses

The script grabs the PID but this isn't correct, as we call env to run it, which then calls python

Can probably fix by explicitly calling python3

Would be better to list them separately to make identifying them easier

new crash in chi-master

At bre the geobar didn't show the OnePlus phone's data for some reason

And save us all from this madness

These are the same device, but when the front end updates the mac address one appears briefly for some reason. The impacts are split between these "devices" and merge back into the proper named one.

API returns no variation over time:

i think that loop.py adds unnecessary complexity and CPU burden and memory pressure which could be eliminated simply by integrating some of the functionality into capture.py and the rest into api.py.

When we deploy this, we need a mechanism to see which instances are running normally/which ones are offline or have developed problems. This will probably be something in loop that pings us every (hour?) with stats from the database so we can check the numbers are going up.

The device dictionary only links manufacturers to devices, which is a problem if you're (say) a big apple fanboy and have 10 apple devices. This dictionary needs to be keyed by mac and manufacturer, not just manufacturer.

have capture.py automatically determine local address, will prevent futzing around

colours cycle randomly (albeit consistently), it's a bit disorientating and it would be nice if we could figure out a way to make this not happen :)

to help users identify autogenerated names for new devices etc.

When running angular and flask bound to external interfaces, loading the front end fails (as tested on the media fire tablet).

This happens because calls to the API time out (/api/refine and /stream)

I think the long term solution here is to move to a more robust server like gunicorn, rather than use the internal flask development server.

Using the websci list

Particularly angular et al.

IoT Refine (front-end) is inexplicably calling out to our server infrastructure even though it's not using any of the data that comes back ?!

I'm so confused

Not so much an issue when it's default, but urgh

Would be nice to be able to click on devices in the legend in the middle of the screen and remove these from all the charts.

Could aid in users seeing what the effects could be of removing particular devices from their homes

So that they're not all "unknown". Would probably want some threshold circa 75-90% after which the burst gets categorised as that destination

(not in /scripts)

Further to #50 we should look into having a mechanism to restart parts of IoT Refine that stall for whatever reason (basically a more robust version of what we used at BRE).

I've hit the 1million packet mark in the packets table and now i'm getting postgres consistently hitting 44-60% of my CPU - we should keep a watch on this and maybe figure out ways to stress postgres out less

also possibly help it with more RAM / buffers / whatnot?

I think that 1894cc5 removes the need for categorisationMain.py

The only other reference is in

https://github.com/OxfordHCC/IoT-refine/blob/c20db92063d711003d18f1706228e8e2d4a40267/db/testCategorisation.py#L4

which can just import the other file directly.

But, we do need to sort out access to files - might make sense to colocate the test and the capture scripts in the same directory, but I'll let you reorganise those first.

Our capture script discards the content of the packet and so later when trying to categorise bursts of traffic we will be unable to inspect and process the HTTP packets that the Hue bridge sends, and so won't be able to display to the user when lights were turned on or off.

Many of the node modules specified in ui/package-lock.json have vulnerabilities that can be resolved by using newer versions

i.e. only read from the config file if no argument was provided

They need to subscribe to updates

There should probably be a cut off point after which data is not displayed in the graphs.

Ideally we would have a selector for 1 day/ 1 week/ 1 month

This ties in with one of the main purposes of the tool - plugging something new in and seeing what changes (which is close to real time?) which isn't going to work if it displays all the data ever collected.

GeoBar doesn't display mB and I'm not sure what data it is accessing at all, it works but should look into this to make sure they're consistent

It also doesn't seem to be consistent with the sizes of bars in refineBar. The data seems to be correct now, but the bars are still the wrong size

There are hardcoded paths to /home/mcnutty in the following files:

service/daemon.sh
service/iotrefine.service

these need to be replaced by something parameterised for the installation

At present the api never closes connections to /stream, causing problems on shutdown

Fix: on SIGTERM or SIGINT event_stream should break out of the eternal loop

Split capture and categorise into separate scripts (i.e. they don't call each other)
Stop burtsification reading two files per packet

Maybe we want to have devices we don't recognise be deactivated by default in the refine interface (e.g. laptops). Quite often these devices have high traffic volumes compared to other IoT devices, and end up swamping the graph with GBs of cat gifs.

I think it would be a very small amount of work to add v6 support. It was a source of confusion for me yesterday as most of my traffic to/from google is v6 now.

• npm install -g @angular/cli
• pacman -S wireshark-cli (or platform equivalent)
• pip instal flask flask_restful
• is ipwhois still needed? If so then it should be on the list, if not then remove it from the code
• README.md still refers to the db reset script by its old name

Have other predictions use the company name. Do this by opening the company file before starting the categorisation loop and passing it in, as well as pickling the dictionary of ips/companies.

iptables doesn't allow mac filtering on the OUTPUT chain, so we're using FORWARD at the moment. We need to make sure that this fires properly on packets that come from connected devices.

(see a147283)

As in, it should forget impacts etc. after 5 mins (or whatever the period is). At least, I don't think it does.

Might not be easy given the current architecture

Will check tomorrow when I'm not on my phone :P

Packet timestamps are generated in the local time zone (e.g. BST), but are recorded in the database as UTC. This causes problems later on when we try to select packets relative to the current time, which will be measured in the local time zone.

Alternative: generate all timestamps in UTC.

As packets get binned into companies, they don't get moved in the front end.

Probably in the API somewhere?

Study participants need a way of reviewing what data IoT Refine has captured, as well as a way of selectively removing data they don't want to submit at the end of an experiment