lazywalker / digiskimmer Goto Github PK

Hi All,
I suspect I have something wrong in my schedules in settings.py. Maybe someone can help me figure out what to do to fix it.

I get these warnings over and over again:
2022-05-05 21:33:53,300 WARNING 788622 [kl7na-12|12+|12!] kiwisdr.local:8073 too busy now. Reconnecting after 15 seconds 2022-05-05 21:33:53,343 WARNING 788622 [kl7na-20|20+|20!] kiwisdr.local:8073 too busy now. Reconnecting after 15 seconds 2022-05-05 21:33:53,376 WARNING 788622 [kl7na-17|17+|17!] kiwisdr.local:8073 too busy now. Reconnecting after 15 seconds 2022-05-05 21:33:53,398 WARNING 788622 [kl7na-30|30+|30!] kiwisdr.local:8073 too busy now. Reconnecting after 15 seconds 2022-05-05 21:33:53,427 WARNING 788622 [kl7na-40|40+|40!] kiwisdr.local:8073 too busy now. Reconnecting after 15 seconds 2022-05-05 21:33:53,469 WARNING 788622 [kl7na-15|15+|15!] kiwisdr.local:8073 too busy now. Reconnecting after 15 seconds 2022-05-05 21:33:53,511 WARNING 788622 [kl7na-10|10+|10!|80|80+|80!] kiwisdr.local:8073 too busy now. Reconnecting after 15 seconds /home/frohro/Projects/DigiSkimmer/./fetch.py:200: DeprecationWarning: getName() is deprecated, get the name attribute instead
I do get plenty of spots, but the warnings bother me, because it appears I messed up the settings.py somehow.
From settings.py:

`SCHEDULES = {
'*': {'kl7na': ['12|12+|12!', '15|15+|15!', '17|17+|17!', '20|20+|20!', '30|30+|30!', '40|40+|40!', '10|10+|10!|80|80+|80!']},

# simple used:
# '21:00-08:00': {'czsdr': [20, 30, 40, 60, 80, 160]},
# '08:00-14:30': {'czsdr': [10, 12, 15, 17, 20, 30]},
# '14:30-21:00': {'czsdr': [10, 15, 17, 20, 30, 40]},

# band hops example:
# # four slots all FT8, the last one is rotate between 60-80-160 when localtime is 21:00-08:00
# '21:00-08:00': {'szsdr': [20, 30, '40~', '60|80|160']},    
# # when localtime is 08:00-12:00
# '08:00-12:00': {
#     # station no.1, slot1 is rotate between 20(FT8)-20(FT4), slot3 is 40(FT8)-40(FT4)
#     'szsdr': ['20|20+', 30, '40~|40+', '60|80|160'],

#     # station no.2, slot1 is rotate between 10-12-15-17-20-30-40 at mode FT8, then 20-30-40 at mode FT4
#     'czsdr': ['10|12|15|17|20|30|40|20+|30+|40+'],
# }

`
Any suggestions?

Thanks,
rob

Hi @lazywalker,
I wonder if we could have JS8? It is now the third most used mode on PSKreporter.
:-)
73,
Rob
KL7NA

Hello and thank you very much for a great tool - I've wanted to try my hands with FT8/FT4 for a long time, and since yesterday running your DigiSkimmer for less than 24 hours already had more than 1500 reports from 77 countries on pskreporter!!

The setting up was quite easy, I was just following, word by word, the README instructions. But I kept getting errors for a while, that finally I've traced to the 'czsdr' labels in the settings.py.template. Changing them all to 'szsdr' fixed everything.

Not sure if this is only my system (Debian 11 freshly installed on a Hyper-V virtual machine running off a Windows 10 PC, connecting to the original KiWiSDR with v.1.461 software on the same local network), or these are typos in the settings.py.template and README files. If the latter, correcting the files may help other enjoy more your great software.

Thank you and kind regards - Mac Ha

I think compression should be disabled to get the best audio quality (not sure if it is lossless or lossy compression?)

'compression': False,
in base.py

I get this message, and no spots hen I look under kl7na, though I see them on my terminal window
2021-08-14 18:48:08,827 WARNING 1923449 [psk.uploader-kl7na] uploading 18 spots

I uploaded my settings.py with a .log extension so github would take it. Here it is: settings.py.log

As an enhancement I would vote for the following:

• Define a minimum of free slots per RX in the config
• Before starting a new worker, check KiwiSDRs /status and pull "users" and "max_users"
• If max_users - users < minimum_free_slots, put DS to sleep for a while

This way, we could let DigiSkimmer run on an idle receiver but keep a slot free if the RX is getting really busy during "rush hour"

I am not advanced enough to find the right spots to implement and do a PR. As an example I did it like that in the run method of KiwiWorker:

`# check if enough slots free for digiskr
parsed_list = {}
min_free_ports = 2
url = "http://"+str(self._options.server_host)+":"+str(self._options.server_port)+"/status"
print("Asking " + url + " for free ports")
response = urlopen(url)
string = response.read().decode('utf-8')
list_string = string.split('\n')

        for item in list_string:
            if item:
                key, value = item.split("=")
                parsed_list[key] = value

        if int(parsed_list['users_max']) < int(parsed_list['users']) + min_free_ports:
            print("not enough free slots")
            self.connect_count -= 1
            if self._options.connect_timeout > 0:
                self._event.wait(timeout = self._options.connect_timeout)

`

Please excuse for the beginner style python. I didn't find the right spot to check users on every freq change, but maybe you get the idea and I am sure this could be useful for others as well.

FT8 (default) ~
FT4 +
WSPR !

What is the code for FST4W ?

from the file config.py >>> MODES = {'~': 'FT8', '#': 'JT65', '@': 'JT9', '+': 'FT4', '!': 'WSPR', '`': 'FST4W'}

Typo in Example : # station no.3, slot1-4 working 10m/20m/30m/40m with WSPR mode at the same time,
# slot5 rotate from 20m to 40m with FT mode.
'cdsdr': ['10!', '20!', '30!', '40!', '20+|30+|40m'],

Meaning of the 40m ?? 40+ maybe

Hi,
I've been running Digiskimmer for months with no problems. Tonight I went to look at spots, and am getting an error. The number of the error doesn't seem to be relevant, as it changes when I stop and restart fetch.py. The messages are 2048 requested and 0 written after the error number. Any tips on what might be going on?
Thanks,
Rob
KL7NA

Seeing the following errors on a fairly constant basis:

2022-07-24 09:00:00,130 ERROR 1764 [KIWIC-20+] decoding queue overflow; dropping one file 2022-07-24 09:00:00,130 ERROR 1764 [FLYDOG-60~] decoding queue overflow; dropping one file 2022-07-24 09:00:00,141 ERROR 1764 [RASP-160~] decoding queue overflow; dropping one file 2022-07-24 09:00:00,141 ERROR 1764 [RASP-160!] decoding queue overflow; dropping one file 2022-07-24 09:00:00,142 ERROR 1764 [KIWIC-15!] decoding queue overflow; dropping one file 2022-07-24 09:00:00,143 ERROR 1764 [RASP-630!] decoding queue overflow; dropping one file 2022-07-24 09:00:00,144 ERROR 1764 [KIWIA-80~] decoding queue overflow; dropping one file 2022-07-24 09:00:00,145 ERROR 1764 [KIWIC-10~] decoding queue overflow; dropping one file 2022-07-24 09:00:00,146 ERROR 1764 [KIWIB-30+] decoding queue overflow; dropping one file 2022-07-24 09:00:00,147 ERROR 1764 [KIWIA-80!] decoding queue overflow; dropping one file 2022-07-24 09:00:00,147 ERROR 1764 [FLYDOG-40~] decoding queue overflow; dropping one file 2022-07-24 09:00:00,148 ERROR 1764 [KIWIC-20~] decoding queue overflow; dropping one file 2022-07-24 09:00:00,148 ERROR 1764 [KIWIB-12!] decoding queue overflow; dropping one file

What I'd like to understand is how the queue works and how it decides there is an overflow. I suspect the logic is based on if it can decode everything within the cycle {i.e. 15s for FT8). In other words is the CPU going to fail to decode in time.

Is this easily tweakable? I would be willing to bet that it wouldn't run away and build up an excessive queue as it's not unsual to have a a very busy cycle(s) followed by less busy cycle(s), thus the queue would catch up within, say, two minutes or less.

There might be a problem with the Raspberry Pi instructions.

I built a brand new DigiSkimmer install yesterday on an RPi4 and when I got to this step:

sudo echo "tmpfs /tmp/digiskr tmpfs defaults,noatime,nosuid,size=300m 0 0" >> /etc/fstab

It didn't want to take the write to fstab. Just gave me a permission denied error even though the sudo command was used, and the Pi user is definitely a sudoer..

I'm not well versed enough in Linux to know what is probably the correct way of writing that line but, I did find that just manually adding the line to fstab seems to have worked and the mount point shows up when you do a df -h.

Possible option to log all conversation messages to file

From Yuri RZ3DVP:

"When I use it with my Kiwi (8 channels RX mode) with a configured scheduler it occupies a new channel before leave old and using my full RX channels, but I want to use 6 for DigiSkimmer and 2 for users listening."

From the front page document : It is also interesting to see how long it takes to spot 100 different DXCC/countries. (A well placed station with a decent antenna can do this within a week of monitoring, but the best systems can do it within a single day).

Got 111 countries on FT8 after 7 hours from first setup.

This software WORKS for FT8 !

Hello,
as KiwiSDR is a spinoff of openwebrx, I tried DigiSkimmer with openwebrx using my HackRF One as receiver, but DigiSkimmer could not connect. The URLs are different.
It would be very good if DigiSkimmer could also support the API of openwebrx. Yes, HackRF is only for one channel with only 8bit resolution, but that's one channel more that no channel ;-). Also, HackRF One can receive on 6m and above, while KiwiSDR is limited to 10m and below.
Thanks in advance
73 Christian DB9CR

2022-10-04 19:00:00,338 WARNING 1 [QW-1] decoder return code: -11
2022-10-04 19:00:00,419 WARNING 1 [QW-0] decoder return code: -11
2022-10-04 19:00:00,488 WARNING 1 [QW-2] decoder return code: -11
2022-10-04 19:00:00,591 WARNING 1 [QW-1] decoder return code: -11
2022-10-04 19:02:00,351 WARNING 1 [QW-1] decoder return code: -11
2022-10-04 19:02:00,391 WARNING 1 [QW-0] decoder return code: -11
2022-10-04 19:02:00,459 WARNING 1 [QW-2] decoder return code: -11
2022-10-04 19:02:00,661 WARNING 1 [QW-1] decoder return code: -11

What could be the reason for this behavior ? FT8 decoded fine.

This could be fixed by reporting to Reverse Beacon Network (RBN) in addition to pskreporter.info

Hi,
Is it possible to use two (or more) receivers simultaneously?
I have defined two receivers, kiwi and kiwi2.
The schedule is

SCHEDULES = {
     '*': {'kiwi': [80, 60, 40, 30, 20, 17, 15, 12]},
     '*': {'kiwi2': ['80!']}
     }

Now it is only running WSPR on kiwi2, FT8 on kiwi is not used.

I'm not sure what happened, but rebooting fixed it. It happened twice so far. I have been using it only about a week, so it took three or four days to happen. Last time I was in a video meeting, so I could not debug the problem further. I will try again to see what happens.

Hi,

It seems that spots with single digit dBm value are not recognized by digiskimmer and thus not uploaded.

For example I get:
2023-08-17 09:10:37,028 INFO 209383 [QW-0] [kiwi] WSPR T090800 DB-21.0 DT0.2 F14.097133 PA3BTI JO22 3 : -
It prints "-" at the end instead of "PA3BTI 3".
Happens also with 0 dBm and 7 dBm spots.

The problem could be, that on the output there is an additional blank space between the locator and single digit power level.

I think it needs to be
wspr_splitter_pattern = re.compile("[<]?([A-Z0-9/]*)[>]?\s([A-R]{2}[0-9]{2}[\w]{0,2})\s+([0-9]+)")
instead of
wspr_splitter_pattern = re.compile("[<]?([A-Z0-9/]*)[>]?\s([A-R]{2}[0-9]{2}[\w]{0,2})\s([0-9]+)")

Tried it and it works (But I don't know if there could be side effects)

Hello! I am testing digiskimmer with KiwiSDR.
After a few hours, it is not uploading to pskreporter. When I Ctrl-C to stop, it uploads many thousands at the end, as shown below.

Regards, Kijun/HL3AMO

2021-01-14 22:59:46,655 INFO 1 [QW-1] [hikijun1] FT8 T225915 DB-20.0 DT0.7 F7.074604 K7XB JH1LAH -11 : -
2021-01-14 22:59:46,656 INFO 1 [QW-1] [hikijun1] FT8 T225915 DB-10.0 DT0.5 F7.075500 W9CSX YC9VM -14 : -
2021-01-14 22:59:46,657 INFO 1 [main] KeyboardInterrupt: threads successfully closed
2021-01-14 22:59:50,282 WARNING 1 [psk.uploader-hikijun1] uploading 36185 spots

With the current code it doesn't detect when someone is calling a hashed callsign:
FT8 T233545 DB7.0 DT0.8 F7.075886 <...> F1TXI IN96 : -
If the regex
".+[A-Z0-9/]+\s([A-Z0-9/]+?)\s([A-R]{2}[0-9]{2})$")
Is modified to
".+[A-Z0-9/<>]+\s([A-Z0-9/]+?)\s([A-R]{2}[0-9]{2})$")

It does also catch those.
I'm not a regexexpert, not sure if this could have side effects, I didn't catch any.

I upgraded to debian bookworm and now i get depreciation warnings when running digiskimmer:

fetch.py:30: DeprecationWarning: currentThread() is deprecated, use current_thread() instead
threading.currentThread().setName("main")

fetch.py:30: DeprecationWarning: setName() is deprecated, set the name attribute instead
threading.currentThread().setName("main")

etch.py:200: DeprecationWarning: getName() is deprecated, get the name attribute instead
if r.getName() == "%s-%s" % (st, band):

This should fix it:

30c30
< threading.currentThread().setName("main")
---
> threading.current_thread().name = "main"
136c136
<     [t.join() for t in threading.enumerate() if t is not threading.currentThread()]
---
>     [t.join() for t in threading.enumerate() if t is not threading.current_thread()]
200c200
<                             if r.getName() == "%s-%s" % (st, band):
---
>                             if r.name == "%s-%s" % (st, band):

Hi all,

I've recently installed DigiSkimmer and enjoying contributing to the spot maps.

Are other digi modes able to be added? I was thinking of some broadcast radio signatures, RTTY, SSTV, etc. Not all would be uploaded to PSK Reporter.

Thanks again.

As I am not a docker user I do need a lot more specific and detailed instructions after downloading the docker file.

Q1. Where do I find the [settings.py.template] ? I need to learn to read : /var/lib/docker/volumes/digiskr/_data/settings.py
Q2. When I have edited the file with my configuration how do I start the app ? Probably the same as the initial run : docker run -eTZ=cat /etc/timezone -v digiskr:/opt/digiskr --tmpfs=/tmp/digiskr lazywalker/digiskr

Needed to use sudo su to get access to the docker directory

And the end is:

2022-10-04 17:11:32,312 WARNING 1 [psk.uploader-Kiwi_0] uploading 23 spots

Thank you for this.

Erwin

Hi,

Thanks for the great software. It runs well with python3.8.
Debian testing seems to have removed python3.8 support and DigiSkimmer now returns an error:

2020-12-07 23:33:02,216 ERROR 282401 [kiwi-80] KW Error
Traceback (most recent call last):
  File "./lib/kiwi/worker.py", line 38, in run
    self._recorder.open()
  File "./lib/kiwi/client.py", line 510, in open
    self._set_auth('kiwi', self._options.password, self._options.tlimit_password)
  File "./lib/kiwi/client.py", line 173, in _set_auth
    self._send_message('SET auth t=%s p=%s' % (client_type, password))
  File "./lib/kiwi/client.py", line 163, in _send_message
    self._stream.send_message(msg)
  File "./lib/mod_pywebsocket/_stream_hybi.py", line 523, in send_message
    self._write(self._writer.build(message, end, binary))
  File "./lib/mod_pywebsocket/_stream_hybi.py", line 343, in build
    return create_text_frame(
  File "./lib/mod_pywebsocket/_stream_hybi.py", line 165, in create_text_frame
    return create_binary_frame(encoded_message, opcode, fin, mask,
  File "./lib/mod_pywebsocket/_stream_hybi.py", line 157, in create_binary_frame
    return _filter_and_format_frame_object(frame, mask, frame_filters)
  File "./lib/mod_pywebsocket/_stream_hybi.py", line 149, in _filter_and_format_frame_object
    return _build_frame(header, frame.payload, mask)
  File "./lib/mod_pywebsocket/_stream_hybi.py", line 139, in _build_frame
    return header + masking_nonce + masker.mask(body)
  File "./lib/mod_pywebsocket/util.py", line 201, in _mask_using_array
    result.fromstring(bytes(s))
AttributeError: 'array.array' object has no attribute 'fromstring'
2020-12-07 23:33:02,217 ERROR 282401 [kiwi-80] websocket close: "'array.array' object has no attribute 'fromstring'"

73