Don't bother fetching a data file if the requested timespan is already covered by the cache DB. This should mean that, for example, the current month is only fetched when very recent data is needed.
The current report is mostly just a guess based on comments around the internet and in the manual. It doesn't exactly match the behaviour of the console. Once the consoles behaviour is know the report needs to be corrected. Determining the real algorithm used requires monitoring data from the console across many storm start/end conditions.
It may also be worth moving much of the reports work to JavaScript - these queries are pretty hard to write in SQL in such a way that they perform acceptably.
Start Rule
Two clicks (0.4mm) are required to start a storm. The exact allowed time between the two clicks is currently unknown but up to 1h45 and less than 2h55m.
End Rule
Based on observed behaviour this is likely: on the hour if the previous 24 hours have less than 0.4mm of rain then any current storm is ended.
Reports support internationalisation but the manual doesn't currently discuss how this works. The manual needs updating.
How it works:
/ReportName -- "international" version - it will show up regardless of a users locale.
/ReportName.en_NZ -- localised version. Any files in here will be loaded instead of the international
files if the users local matches en_NZ.
/Report2.en_US -- This report only exists in the en_US local. Users in other locales won't see it at all.
So search path for the above is:
:/reports/ReportName.en_NZ
:/reports/Report2.en_US
:/reports/ReportName
Note how localised directories always appear before international directories.
The sum and running total aggregate functions only apply to rainfall and evapotranspiration - all other columns get an average IIRC. So selecting a running total for temperature actually just gets you an average which is confusing.
Best to just hide these options when they're not useful.
When, eg, temperature and rainfall are selected and the user chooses a running total or sum it might also be nice to let the user know what will be done with the temperature column
Currently only console loggers 100% compatible with the official ones are supported. The WeatherLink Live provides support for a much wider range of sensor configurations plus it works over a LAN making deployment a bit more flexible too. So having support for this would certainly be nice, especially when #43 is implemented and we can actually ingest, store and present data for arbitrary sensor configurations.
The Web API requires internet access. This makes it unsuitable for locations with limited or no internet connectivity.
No on-device storage of archive data means that the software needs 100% uptime if gaps in archived data are to be avoided.
With the console loggers avoiding gaps in data is trivial - ensure the software runs at least once a week to download archive records stored on the console data logger. In over six years of operation the Sandy Bay station has only failed to achieve this once despite unreliable mains power - someone turned off the wrong circuit breaker and due to the distance it was a little over a week before someone could turn it back on. A few hours of data were lost.
To achieve the same level of reliability with WeatherLink Live I'd probably need:
Two separate computers running data logging software to allow for one to be down at any given time for maintenance/faults.
A UPS large enough to power at least one computer for the maximum likely power outage. To match the console data loggers reliability here the UPS would need to be large enough to run one computer for a full week.
This is impractical so at this stage I've got little use for the device myself - without a pile of extra sensors the console logger is simply more reliable.
Implementing this enhancement will be difficult without access to a device for testing. The easiest option is probably to build a WLL emulator, validate that with other software and then use the emulator to build the data logger. This would still require someone with the real hardware to test it so its unlikely to happen unless someone comes along wanting to run zxweather with one of these devices.
This usually seems to be caused by a corrupted download likely caused by the video being downloaded multiple times in parallel (the web data source could be smarter here). Fix for now is to delete the cached file and redownload it when this happens.
Problem was likely introduced in cf87802 - weatherpush works fine for a while but eventually stops transmitting live data and has to be restarted. Systems currently running this code need a cron job to restart the service daily to ensure it keeps working.
Debian 11 is out and it does not support Python 2.7. This means that running zxweather on current debian or ubuntu is going to be quite hard so its really time to make everything compatible with Python3.
So far the admin tool has been ported fully and work has started on porting the Davis Logger. Progress so far indicates it should be possible to complete the port without dropping Python 2.7 compatibility (possibly required as Sandy Bay only has quite old versions of python3 available).
Ideally components should be made compatible with both 2.7 and 3.10 where possible. Once a minimal python3 port has been completed further work on adding type annotations and unit tests should be carried out in a branch.
admin-tool
Basic Port
Tidyup: Type annotations, unit tests (where appropriate)
If the desktop app is set to imperial units the reports probably should be too. The reports that currently have options for units could probably either loose that option or have it changed to (default/metric/imperial)
Out of the box its completely unusable on mixed-DPI and possibly HiDPI setups. Setting the environment variable QT_AUTO_SCREEN_SCALE_FACTOR=1 and running the app with -platform windows:dpiawareness=0 makes it at least work though right-clicking on the systray icon crashes the app.
The column pickers around the place (live plots, charts, view data, export) were significantly overhauled sometime after the software was last tested with a FineOffset weather station.
Need to check that no davis-specific options have crept in when running against non-davis hardware (eg, soil moisture, rain rate, UV, etc)
All pages need to include a header or link element referring to the default UI version of the page. Probably a header as less likely to interfere with older browsers.
This would allow, for example, the station owner to pre-set the normals for the NOAA Year report, or the coordinates for the sunrise/sunset time, etc.
These defaults would only be used for the particular station URL.
Loading order would be:
Report defaults
Saved report criteria (global)
Station URL report defaults supplied by WebUI
Station URL saved report criteria (values modified while this particular station URL is in effect)
This means the user isn't forced to use the provided defaults - they can override them and have their changes persisted.
Might be worth considering if this can be done for DB-connected clients too. The Web UI actually has some of these reports too so maybe the defaults could be saved in the DB and the desktop client could fetch them from there when its running directly from the DB. The Admin Tool would need adjusting to allow these values to be configured.
Currently they're double click and its there is no UI hint that this is even possible. It might be better to have the bars single-click with a hand cursor when the mouse is over the bars.
New live plot: Tick all the temperature variables. Strip charts appear. Go into options and switch to single axis rect. Only one temperature graph is plotted - the rest go missing!
Transmitting images, etc, that have exceeded their retry count is fairly painful right now - have to use psql to reset the error count and then either wait for the next new piece of data to be inserted or manually send a notification.
Neutral to the datas origin (could be live data, samples from the database, rows from a report query)
Supports both live and static data
Supports multiple Axis rects each with a potentially different data set
Supports multiple types of plot (not just line charts - should histograms, error bars, polar plots, etc)
Can be embedded in documents
Supports edit interactions that can optionally be individually disabled
Can be serialized to disk optionally with or without data.
All data sources should be serializable. Something like the SamplePlotDataSource would have two options:
Serialize as a SamplePlotDataSource (this really just serialises the timespan, aggregate settings and I guess the column selection)
Serialize as a StaticPlotDataSource (this writes out the raw data so no option to add columns or change the timespan after the fact)
When a plot is deserialized it would hand the serialized data source stuff over to a PlotDataSourceFactory which would then instantiate the appropriate class and tell it to load from json or whatever
Serialized plot should cover all customisations the user has done.
Ideally we have some way of supporting plot templates that allow the user to specify some parameters (eg a timespan or set of timespans)
All of this will enable a number of new features such as:
Charts embedded in reports
Mix of live and static graphs in one plot
Auto-updating 5-minute plots so you can have, for example, the last few hours (5m resolution) and last hour (2.5s) resolution in one plot
Visible symptom of this is gaps in graphs - the only time these should ever happen is if there is data missing on the server (which is almost never the case). Either a data file is not being downloaded or the data file isn't being correctly stored in the cache DB.
This one is going to be difficult to fix and is going to require some major refactoring. Possibly the easiest solution is to refactor out all the data source code into DLLs and run reports as a subprocess.
Something like the following refactoring would probably work for the current data source code:
Refactor settings code (settings.cpp/h, settingsdialog.cpp/h, settings.ui) out into settings.dll
(UI can probably stay in the main app if there are significant resource dependencies)
(if UI moves perhaps move config_wizard/* too?)
link click event handling needs to be abstracted away.
Create dsfactory.dll
This contains DataSourceProxy plus the new DataSourceFactory that just hands back whatever AbstractDataSource
the configuration file says we should be using. This would probably almost always be a DataSourceProxy instance
that is hooked up to listen for settings changes so we can reconnect all datasource instances everywhere on
settings change.
Create report.exe
When run as a subprocess with /pipe parameter:
Receive report name and parameters via pipe
Receives report search path via pipe (incase the search path is being overridden by command line parameters)
Report progress back to the main app via pipe
Report link click events back to the main app via pipe
When run directly
Run report dialog is displayed. Perhaps include a settings button in the bottom left corner.
If we have a settings button the settings UI should hide the two charting tabs plus the minimise action
and late live data group boxes on the general tab
Link click events communicate with existing instance of desktop.exe somehow to open the URL.
If desktop.exe isn't already running for that station it is started somehow.
Can be run with command line parameters
can specify report name which skips the report selection step
can specify report timespan which skips the timespan selection step if valid
can specify all custom criteria which skips criteria step
can speciy output filename or directory which skips display step
have an example batch file or two which runs NOAA reports for all months/years.
Perhaps have a CLI interactive mode that prompts for custom criteria, etc.
The current data layer could at best be described as a mess. Its entirely synchronous and runs on the main thread which causes problems for reports (#27) and results in weird side-effects like the WebDataSource feeling faster than the DatabaseDataSource even though its actually slower (it just doesn't lock up the UI due to being async internally).
This work needs to be done before The Great Chart Refactoring (#42) can happen. This work is independent of #27 - even after the data sources are refactored/rewritten it may still be preferable to run reports in a subprocess and there are other drivers for refactoring the data source code into libraries than report subprocesses.
There are design notes for how the new data sources should be structured on onedrive but the work will be a bit disruptive so shouldn't be tackled for v1.0 - that's been delayed for long enough already.
Would be neat to be able to build the desktop for web assembly and run the whole thing in a browser. Probably not super useful except as a demo of what its like without having to install it.
There are a number of Qt bugs that need fixing or working around to make this possible:
Remove any instances of QDialog::exec() that aren't really necessary
For any remaining QDialog::exec() instances, add a build option to replace them with signals and slots or lambdas. This wouldn't be used on regular desktop builds.
A few other changes would likely be needed to the app itself too:
Limit cache DB size so as to not use up all of the computers RAM
Limit image cache size or disable this cache entirely if we can rely on the browser caching stuff instead.
Add a WebSocketLiveDataSource as TCP connections aren't allowed in wasm without server help
Connection options in the settings window should act as though they're supplied on the command line (or actually supply the connection settings via command line if that is a thing we can do)
Postgres data source won't be available at all
Web data source can only talk to the hosting domain (or a server that is doing CORS)
Live data source must be websocket
station code should be fixed to whatever station the app was loaded for
Maybe provide a pre-populated cache DB with recent data to speed things up a bit
Note that in places where QDialog::exec can be replaced with singals and slots it would reduce in worse usability - a lot of the time exec is used because we don't want multiple copies of the dialog open at once (eg, settings).
Known QDialog::exec() usages:
Known locations: Signal/Slot replacement OK?
MainWindow::showSettings() Yes
MainWindow::showAbout() Yes
MainWindow::showChartWindow() Yes
MainWindow::showExportDialog() Yes
MainWindow::viewData() Yes
ChartWindow::changeDataSetTimeSpan() Yes
ChartWindow::setSelectedKeyAxisTickFormat() Yes
ConfigWizard SelcetStationPage::stationDetailsClick() Yes
LivePlotWindow::showAddGraphDialog() Yes
LivePlotWindow::showOptions() Yes
LivePlot::changeGraphStyle() Yes
Report getSaveDirectory() No - function returns a value depending on the result of the users response
In a regular plot it would be nice to be able to just double-click on the X axis to change the timespan rather than having to go through the context menu.
The Web UI runs quite slowly on a Rasbperry Pi 2B. It shouldn't. It should run fine on a Raspberry Pi 1.
A lot of this probably comes down to database optimizations. The records query in particular is quite expensive and no doubt there is a pile of other things that could be improved.
This was (partially) built years ago and put in the master branch when it shouldn't be. Now it really needs to be finished off. The intended use-case for this thing is being able to support the desktop app running remotely without needing the full web UI - the app just generates the required resources on disk where they can be served up by a regular web server such as Apache.
Port to recent python3 - its new, its never been deployed, no reason to support python2.7
Handle samples arriving for a previous day or month
Send a reset to gnuplot at the end of each chart to try and clear file locks
Only build outputs for the current month if the current day is >1 (on the 1st this function will just duplicate the work we're already doing to build the data files for today)
Privacy option for sysconfig.json coordinates
Test other chart output formats work properly
Don't regenerate last month if the latest_ts in the json file is for some date in the current month
Add a way of directing regeneration of specific months on startup
Option to trash files only used for generating charts once charts are generated
Multiple output set options (desktop support, web UI Basic support (charts), additionals?)
Don't leave gnuplot running forever - stop it once the batch has finished (need some way of knowing this)
More gnuplot instances
Implement charting interval setting
Implement live data update interval setting (once realtime.txt and live.json are in place)
Currently loss of network connectivity results in an error message box appearing (with a sound effect if the speakers aren't muted) every 2 seconds or so from the LiveDataSource. The WebDataSource doesn't really detect when network connectivity comes and goes which can result in toolbar options being greyed out until you force the data source to re-connect somehow.
The QNetworkConfiguration class might be of some help here. It was deprecated in some 5.x release and removed in 6.0 but 6.1 provides QNetworkInformation which should do the job there.
Probably the showAddGraphDialog function needs to set the dialogs parent to be the LivePlotWindows parent when the LivePlotWindow itself isn't visible.
Note the LivePlotWindow isn't given a parent currently so mainwindow.cpp will need adjusting. Problem is setting a parent results in it always being on top of the parent. So really we need to pass in an alternate parent for the add graph dialog.
Double click on Year in rainfall visualisation to get a plot for the current year. Optionally rename the current line
Hit add data set and choose Rainfall, entire previous year, aggregate running total, hourly
Resulting graph has no data visible. Bottom axis is showing 2019, top axis is showing garbage (1995, 2020, 1945, 1970, 1995...)
The top (garbage) axis appears to be associated with the 2021 data set while the bottom 2019 axis seems to be associated with the 2020 dataset. This is the wrong way around - 2021 was in the plot first so should be on the bottom.
Need to check removing and adding datasets is working properly and cleaning up axes as it should.
The live data source receives new samples as they arrive on the server. If these are also stored in the cache DB it should reduce the frequency with which the data file covering the current month needs to be downloaded.
This may require extending the subscribe command in the Weather Server - the only fields it appears to provide from DavisSample are UV Index and Solar radiation.
Not all weather stations will run forever. You might move house and setup a new station for that location. Or maybe you change to far more accurate hardware and don't want to mix your old data with the new stuff. One example is http://weather.zx.net.nz/s/rua/ - this station was replaced with a Vantage Vue on a new station code for accuracy reasons (rua2). The vue was later turned off due to moving house (and also hardware failure). New location, new weather station - (hlz). rua and rua2 will likely never have new data added - they're static. archived.
Currently zxweather has no special abilities when it comes to archived stations. The Web UI just shows a "no recent data" message on the home page and the desktop client just shows whatever the final current conditions update was.
Minimal support for archived stations would be:
Option in admin tool to mark a station as archived. Probably supply a message with this too (eg, why the station was archived)
Web UI changes
Web UI shows some sort of archived station message rather than "no data". Maybe allow it to load an HTML file (like it does for the about station page) instead.
List archived stations under an "archived" subheading in the station selector.
Desktop UI archived station mode:
Current conditions panels hidden (current conditions, forecast, rainfall, status)
Live charts hidden
Shows a message in the main window saying the station is archived/no longer active along with the archived station message. Maybe some statistics too (date span covered, number of rows of data, etc)
Don't setup the live data source
Fix the date range around the place (chart, export, view data, etc)
An option or tool to load the cache DB with the entire available data set in one go so no further requests to the server are required.
Consider setting the minimum Qt version to 5.7 - this is what Debian oldstable ships with in July 2020. This allows a bunch of old Qt5 stuff to be dropped too (like the old scripting engine) and gives a newer minimum SQLite version.
Without them you can scroll all axes by scrolling with no axis selected. The lock option only seems relevant if you want to zoom all axes which the separate lockX/Y menu items don't allow.
The functionality it allows (select one Y axis and all Y axes are selected) is useful. So removing the axis locks entirely isn't really a solution. Instead we should do one of:
Rename the option. The current name "Lock Y Axis" is confusing and doesn't really suggest what it actually does or why you'd use it. At first I had no idea what it did and turning it on and off didn't actually do anything. I had to look at some old documentation to realize you needed multiple axes of the type for it to have any effect.
Drop the option and replace it with double click a Y axis to select all Y axis
Drop the option and have a button somewhere to "Select all Y axis" or "Select all X Axis"
Currently any station supporting more than the basic common sensors provided by the likes of the FineOffset WH1080 require extra database tables and special support added through out desktop app and web UI. This makes supporting extra sensors quite a lot of work.
A way is needed to support arbitrary extra sensors for any station without code changes. This would allow things like one or more Davis AirLink stations to be added to any random existing station. It would also enable the wider selection of sensor configurations allowed by the WeatherLink Live.
Some work towards this has already been done on the configuration side. For Davis stations, extra Leaf/Soil station and extra Temp/Humidity stations are configured via the database with support for renaming and turning on and off individual sensors. Whats missing now is a generic way for adding extra sensors via configuration and a way of storing data for those sensors.
Further work on this really shouldn't be tackled until #41 and probably #42 are dealt with - otherwise its just making that work harder.
The weather-push protocol really needs some version information transmitted during connection startup to allow for future protocol changes without breaking existing clients in non-obvious ways. The server should reject unsupported clients in a useful way (client reports incompatible server version to the user) where its unable to maintain backwards compatibility.
Main driver for this is systems at sandy-bay being stuck on python 2.7 (or some equally useless version of 3.x) for the foreseeable future. When weather-push is eventually moved to Python 3 the systems at sandy-bay will be unable to upgrade but they'll still need to be able to submit data.
Client transmits protocol version on connect
Server rejects incompatible clients and specifies minimum client version
Images received via live data broadcast don't have their metadata reloaded. They forever remain as <partial metadata> or similar.
This bug appeared with the data source optimisation. These partial metadata entries need to be considered as not-cached by the WebCacheDB and their presence should result in the date being re-downloaded.
Also need to check the images window behaves properly when images come in via the LiveDataSource
The main window has a number of minor UI layout issues including:
Current conditions labels don't have enough vertical space (the g in "light breeze" is being cut off)
The image panel isn't resized properly on show if the application is started with hit hidden due to there being no active image sources
Switching from a Davis to an inactive FineOffset station results in the UI not resizing properly leaving lots of empty space too. The resize doesn't happen until first current conditions update which never happens - need to find a way of doing this sooner.
Really these are side-effects of the fixed layout the main window currently uses. The code for it is fragile and inflexible. The proper solution is probably to remove the fixed layout entirely as its likely a major source of problems for #12.
React
Vue.js
Typescript
TensorFlow
Django
Laravel
D3
Facebook
Microsoft
Google
Alibaba
D3
Tencent