Request: JSON output about dumphfdl HOT 19 CLOSED

Thanks all for the feedback. If there's interest then fine, I'll implement it, no problem.

from dumphfdl.

JSON support is now available for testing in the devel branch.

from dumphfdl.

There has been a very large uptake of interest in all aspects aircraft data of late due to many factors.

For the longest time, the 'only' method of working with that data from different sources has been working with flat strings.
Late last year Jaero added JSON output and this started the swing over to this format.
dumpVDL2 using JSON helped move the applications to that method.
PC-HFDL only sent flat strings, so that made accessing that data difficult. (That and it was abandon ware and Windows only).

My website has been built around flat data and it has been beyond frustrating for the past 7 years working with substrings and counting characters to get the data (ICAO, lat/lon, groundstation pings etc) from each message.
By migrating my site feeds from Jaero strings to JSON we are able to quickly highlight interesting data.
The same effect is now happening with dumpVDL2 with apps like ACARShub and airframes.io. By using JSON in those ACARS messages we can quickly link data with other data and 'join the dots' much quicker.

The goal is to link HFDL with VDL2 with Satcom (with ADSB). If all the apps were to use JSON it will dramatically reduce development time to bring all data sources into each app and extract the data in a consistent library.
It will also help create a standardized API for this data.

Lastly, I personally have helped over two dozen people get up and running with Node-RED using either dumpvdl2 and Jaero and I always recommend using JSON for beginners with software like Node-RED.
While personally I have tried using Regex, the substring command in Javascript is more beginner friendly, but JSON beats both methods and will always recommended it to anyone starting out.

from dumphfdl.

I'm the author of ACARS Hub. I would also enjoy seeing JSON output added in to dumphfdl and would add that support in to ACARS Hub so users could use that as an input source.

from dumphfdl.

from dumphfdl.

one of the biggest hurdles for me to use dumphfdl as a newbie (after getting a suitable SDR and loop antenna) has been finding what HFDL frequencies were actually in use, besides having to spend hours manually trying them all in groups to find out.

@jaymot123 while I understand the usefulness of central log gathering and processing in general, this is not strictly necessary to achieve the goal that you're aiming for. Regardless of where you are in the world, you should be able to receive traffic from at least one (and most probably from several) ground stations all day (and night) long. Each station broadcasts the current state of the whole HFDL network, so you should be able to collect the current frequency set in a couple of minutes just by yourself, without reaching for data from others. Just try frequencies listed in the system table for your nearby stations, so that you can receive uplinks from them.

The way I approach this is I basically monitor HFDL bands that make sense for the current time of day. In these bands I monitor all HFDL frequencies that may potentially be active over the course of the whole day, rather than frequencies which are currently active. This simplifies things, because the frequency list for each band is mostly fixed and static. The first idea might be to throw in all frequencies listed in the system table, but this is an overkill, since there are currently 105 listed frequencies, out of which only 59 are in active use at this time of the year. I have crafted a little Perl script that parses the log file and builds an array of frequencies that have been found advertised in squitters. I run it with:

cat hfdl_20220210.log | hfdl_generate_bands_sh.pl -e 'Kras|Bahr'

It analyzes squitter messages that look like this:

[2022-02-11 00:08:26 CET] [11348.0 kHz] [-0.2 Hz] [-81.8/-83.0 dBFS] [1.2 dB] [300 bps] [S]
Uplink SPDU:
 Src GS: Canarias, Spain
 Squitter: ver: 0 rls: 0 iso: 0
  Change note: None
  TDMA Frame: index: 2603 offset: 4
  Minimum priority: 0
  System table version: 51
  Ground station status:
  ID: Canarias, Spain
   UTC sync: 1
   Frequencies in use: 13303.0, 11348.0
  ID: Auckland, New Zealand
   UTC sync: 1
   Frequencies in use: 17916.0, 10084.0
  ID: Hat Yai, Thailand
   UTC sync: 1
   Frequencies in use: 13270.0, 6535.0

and the result looks like this:

declare -A bands
bands=(
        [2]="2998"
        [3]="3900"
        [4]="4681"
        [5]="5502 5508 5514 5544 5583 5589 5720"
        [6]="6529 6535 6559 6565 6589 6619 6652 6661"
        [8]="8834 8894 8912 8927 8939 8942 8977"
        [10]="10027 10060 10063 10081 10084 10093"
        [11]="11184 11306 11312 11318 11348 11387"
        [13]="13270 13276 13303 13315 13321 13324 13342 13351"
        [15]="15025"
        [17]="17916 17919 17928 17934 17958 17985"
        [21]="21928 21931 21934 21937 21949 21955 21997"
)

-e 'Kras|Bahr' command line option causes frequencies advertised for Bahrain and Krasnoyarsk to be excluded from the result, since these stations don't work.

The output is essentially a bash script that is used by another script that I use to start dumphfdl for a chosen band.

The next part of the game is to create a proper band switching schedule and put it into crontab. You may pick bands according to time of day only, however this is usually suboptimal. In order for the band to be interesting, there must be at least one (preferably more) channels that is active and you are actually able to receive it. For example, I'm in Europe, so I don't optimize my schedules for frequencies used by Hawaii or San Francisco, since I barely have any reception from there. I pay attention mostly to Shannon, Iceland, Canaries and also a bit to Thailand and New York. So I switch bands according to time of day and to the schedule of these stations, which I can estimate using this script. You can run it just with:

cat hfdl.log | hfdl_gs_freq_changes.pl

to get a list of channel down/up events extracted from squitters found in the log. It looks like this:

[2022-02-10 23:06:24 CET] [Reykjavik, Iceland] channel down: 3900.0 (reported by Reykjavik, Iceland on 8977.0 kHz - channels now in use: 11184.0 8977.0)
[2022-02-10 23:07:28 CET] [Reykjavik, Iceland] channel up:   5720.0 (reported by Reykjavik, Iceland on 8977.0 kHz - channels now in use: 11184.0 8977.0 5720.0)
[2022-02-10 23:17:50 CET] [Canarias, Spain] channel down: 17928.0 (reported by Shannon, Ireland on 8942.0 kHz - channels now in use: 13303.0 11348.0)
[2022-02-10 23:17:50 CET] [Canarias, Spain] channel up:   11348.0 (reported by Shannon, Ireland on 8942.0 kHz - channels now in use: 13303.0 11348.0)
[2022-02-10 23:42:49 CET] [Agana, Guam] channel down: 6652.0 (reported by Canarias, Spain on 11348.0 kHz - channels now in use: 21928.0 11306.0 8927.0)
[2022-02-10 23:42:49 CET] [Agana, Guam] channel up:   21928.0 (reported by Canarias, Spain on 11348.0 kHz - channels now in use: 21928.0 11306.0 8927.0)
[2022-02-10 23:44:25 CET] [Auckland, New Zealand] channel down: 13351.0 (reported by Canarias, Spain on 11348.0 kHz - channels now in use: 17916.0 10084.0)
[2022-02-10 23:44:25 CET] [Auckland, New Zealand] channel up:   17916.0 (reported by Canarias, Spain on 11348.0 kHz - channels now in use: 17916.0 10084.0)
[2022-02-10 23:48:09 CET] [Riverhead, New York] channel down: 13276.0 (reported by Canarias, Spain on 11348.0 kHz - channels now in use: 11387.0 8912.0)
[2022-02-10 23:48:09 CET] [Riverhead, New York] channel up:   8912.0 (reported by Canarias, Spain on 11348.0 kHz - channels now in use: 11387.0 8912.0)

This looks quite chaotic and it's hard to draw any conclusions from this, however you can filter the output to a particular frequency with -f option or to a particular station(s) with -s option. So for example:

$ cat hfdl_20220208.log hfdl_20220209.log hfdl_20220210.log | hfdl_gs_freq_changes.pl -f 6661
[2022-02-08 01:16:46 CET] [Riverhead, New York] channel up:   6661.0 (reported by Shannon, Ireland on 2998.0 kHz - channels now in use: 8912.0 6661.0)
[2022-02-08 14:17:02 CET] [Riverhead, New York] channel down: 6661.0 (reported by Shannon, Ireland on 8942.0 kHz - channels now in use: 13276.0 8912.0)
[2022-02-09 02:21:46 CET] [Riverhead, New York] channel up:   6661.0 (reported by Canarias, Spain on 11348.0 kHz - channels now in use: 8912.0 6661.0)
[2022-02-09 14:15:29 CET] [Riverhead, New York] channel down: 6661.0 (reported by Reykjavik, Iceland on 15025.0 kHz - channels now in use: 13276.0 8912.0)
[2022-02-10 02:16:58 CET] [Riverhead, New York] channel up:   6661.0 (reported by Canarias, Spain on 11348.0 kHz - channels now in use: 8912.0 6661.0)
[2022-02-10 14:15:49 CET] [Riverhead, New York] channel down: 6661.0 (reported by Shannon, Ireland on 10081.0 kHz - channels now in use: 13276.0 8912.0)

Can you see a pattern now?

If you find out that you have a good reception on that channel, it might be worth it to switch to the 6 MHz band whenever this channel is turned on, so that you can make the most of it.

From now on it's just a matter of collecting data, analyzing them and tweaking the resulting schedule periodically. Because yes, station schedules change every couple of days. I also recommend collecting reception statistics via statsd and plotting them somewhere (eg. in Grafana), to get a view on when you receivers are busy and when they just sit decoding nothing (which indicates that the schedule might require tweaking). You can't manage what you don't measure.

Could all this be automated? Perhaps. I'm currently doing all the above (and more) to get an understanding of which schedule optimization tricks work and which don't. I haven't yet made up my mind on how such an automation should work so that it could achieve significantly better results than a statically configured time-based schedule tweaked by hand every couple of days. Suggestions welcome.

In case you are interested in continuing this topic, I suggest creating an entry in the Discussions section, since this does not seem related to the JSON feature itself.

from dumphfdl.

Hey @szpajder -- activating the dumphfdl feed support at https://app.airframes.io. One thing. dumphfdl's JSON support doesn't seem to have support for --station-id. Could you add that to match dumpvdl2?

Thanks for everything so far!

from dumphfdl.

from dumphfdl.

Once upon a time there was a request for a JSON output in the VDL2 decoder dumpvdl2 which could have allegedly enabled many new possibilities, great apps and integrations. I have implemented it (which was a considerable amount of work), however judging from what's going on on mailing lists, forums, etc., people still prefer to do regexp sorcery to parse text output instead. So I'm yet to see at least one of these great apps that utilize dumpvdl2's JSON output. Are you aware of any? OTOH, over the years of my professional and hobby work I got pretty used to people requesting various features but never using them after they are delivered.

I am familiar, because I spent time implementing and submitted JSON support to dumpvdl2 and you rejected it 😝

We do use the JSON output for airframes.io, and we are in the process of doing some very useful things with dumphfdl... specifically, we are hoping to do three things:

1. receive squitter msgs from ground stations in realtime across the world to then centrally process and make available via an API the status of all ground stations and which channels are active -- with an overall purpose of being able to control dumphfdl to listen on the most active channels on the active bands for a given location.

2. publish a realtime web status of HFDL activity around the world (along with API access to this).

3. receive all other msgs to support the overall airframes.io goal of aggregating, parsing, and making available all ACARS related data as possible.

We will definitely use it. Over on the ADSBX discord on #project-acars, we have a large group of folks who are actively using ACARS/VDL2/AERO/etc and feeding Airframes. There are at least 6 of us who are now using dumphfdl actively and having to parse the text output (semi-successfully of course, since the data is not easily parsable).

If you need me to implement it, I will do it again, but I think last time you had planned a different approach (and my implementation was different), so just let me know ahead of time 😄

Cheers!

from dumphfdl.

Are you aware of any?

Airframes https://app.airframes.io
ACARS Hub https://github.com/fredclausen/docker-acarshub (which is currently powering a few public instances, which we can see by searching google for "ACARS Hub")

from dumphfdl.

@szpajder Thank you so much! Let me know if I can help in any way.

from dumphfdl.

Hi @szpajder just checking in. Anything I can do to help?

from dumphfdl.

"receive squitter msgs from ground stations in realtime across the world to then centrally process and make available via an API the status of all ground stations and which channels are active -- with an overall purpose of being able to control dumphfdl to listen on the most active channels on the active bands for a given location."

Not intending to try to rush @szpajder but I just wanted to add that from an end-user's perspective, this would be a wonderful feature as one of the biggest hurdles for me to use dumphfdl as a newbie (after getting a suitable SDR and loop antenna) has been finding what HFDL frequencies were actually in use, besides having to spend hours manually trying them all in groups to find out. Days were spent trying to find the frequencies that had actual messages.

Another user in the ADSB Exchange's Discord server (in the #acars channel) has even come up with a bash script that configures dumphfdl then runs it for a fixed period of time scanning individual groups of frequencies, saving the resulting messages to a log, counting the number of messages in each frequency group, then running dumphfdl with the busiest group and exiting (leaving dumphfdl running) for the purpose of letting a user run the script twice or thrice daily as a cron job. Several of us have been testing this shell script with mixed and often underwhelming results. If this could be done a better way by pulling the realtime information from a site via an API as the others are working on implementing it would make it a lot easier for new dumphfdl users to get going and start receiving messages and tracking planes globally in VRS.

from dumphfdl.

@szpajder Oh, cool! Thanks for this information. More things to play with and have fun! :)

from dumphfdl.

So far, tests have been good. Thank you for adding!

from dumphfdl.

Released in v1.3.0.

Closing.

from dumphfdl.

So why it works for me?

$ ./dumphfdl --iq-file ../hfdl_8900000_250000_2.cs16 --sample-rate 250000 --sample-format CS16 --station-id mystation --output decoded:json:file:path=- --centerfreq 8900 8977 | head -1
dumphfdl 1.3.0-70f5a17 (libacars 2.1.4)
{"hfdl":{"app":{"name":"dumphfdl","ver":"1.3.0-70f5a17"},"station":"mystation","t":{"sec":1653127396,"usec":418895},"freq":8977000,"bit_rate":300,"sig_level":-69.617294,"noise_level":-73.071358,"freq_skew":5.700855,"slot":"S","spdu":{"err":false,"src":{"type":"Ground station","id":3},"spdu_version":0,"rls":false,"iso":false,"change_note":"None","frame_index":982,"frame_offset":0,"min_priority":0,"systable_version":51,"gs_status":[{"gs":{"type":"Ground station","id":3},"utc_sync":true,"freqs":[{"id":2},{"id":3},{"id":5}]},{"gs":{"type":"Ground station","id":4},"utc_sync":true,"freqs":[{"id":4},{"id":5}]},{"gs":{"type":"Ground station","id":5},"utc_sync":true,"freqs":[{"id":4},{"id":5}]}]}}}
Waiting for all threads to finish
Shutting down decoder thread
output_file(-): shutting down

from dumphfdl.

Thanks @szpajder, I do see that it is working as expected! False alarm.

from dumphfdl.

You should expect to see this supported at https://app.airframes.io today or tomorrow.

from dumphfdl.