Image transfers are working great to replicate allsky data from a remote site (local c

Merged <a class="issue-link js-issue-link" data-error-text="Failed to load title" data

syncapi timelapse video transfers failing but images OK even with 900s timeouts ,about aaronwmorris/indi-allsky

Comments (17)

aaronwmorris commented on June 8, 2024 1

I might know what is going on. The syncapi transfers are validated based on the HMAC signature of metadata of each file. The metadata also includes a timestamp, so even if the HMAC signature were somehow recovered, it would only be valid during a 15 minute window: the current 5 minute block, -5 minutes, and +5 minutes to account for time skew between the local and remote indi-allsky systems.

The HMAC signature is based on the START TIME of the transfer and if the transfer takes longer than 5-10 minutes, you could easily start to get API key validation failures (depending on the time skew). 100% of transfers longer than 10 minutes would probably fail.

Give me a day or two to release an update. I have an existing code change that I am testing and I need to complete that first.

from indi-allsky.

aaronwmorris commented on June 8, 2024 1

Interesting, I will read about the requests toolbelt. The more I read about this, I do not think this will solve the immediate issue. It will make the uploads more memory efficient on the client, but I do not think it is a problem on the server.

from indi-allsky.

aaronwmorris commented on June 8, 2024 1

Well, I feel kind of silly now. This ended up just being a simple typo. I setup the variables to test the HMAC signature if transfers took longer than 5 minutes (even before all of these changes), but they were never actually used.

        time_floor = math.floor(time.time() / self.time_skew)

        # the time on the remote system needs to be plus/minus the time_floor period
        time_floor_list = [
            time_floor,
            time_floor - 1,
            time_floor + 1,
            time_floor - 2,  # large file uploads my take a long time
            time_floor - 3,
            time_floor - 4,
        ]

        for t in time_floor_list:
-            hmac_message = str(time_floor).encode() + data
+            hmac_message = str(t).encode() + data

I suppose it is nice to have the streaming optimizations in place, but this would have worked previously without that. Hopefully, I did not introduce any other bugs fixing this.

Merged #1069

from indi-allsky.

aaronwmorris commented on June 8, 2024

Is the remote site a standard install using Apache and gunicorn? Are there any interesting errors in the web logs at /var/log/indi-allsky/webapp-indi-allsky.log?

Do you have sufficient space available in /tmp on the remote server? When the server downloads the file, it temporarily stores the file in /tmp before copying the file to the final location.

from indi-allsky.

keno242 commented on June 8, 2024

Is the remote site a standard install using Apache and gunicorn? Are there any interesting errors in the web logs at /var/log/indi-allsky/webapp-indi-allsky.log?

Do you have sufficient space available in /tmp on the remote server? When the server downloads the file, it temporarily stores the file in /tmp before copying the file to the final location

Yes, the remote site is a dedicated install of indi-allsky running on a nas container with plenty of space here is the support dump

sudo ./support_info.sh 
#################################
### indi-allsky support info  ###
#################################

Distribution: Ubuntu
Release: 22.04
Arch: x86_64
Bits: 64

CPUs: 6
Memory: 16287720 kB

Uptime
 09:23:24 up 9 days, 14:50,  2 users,  load average: 3.54, 2.78, 1.79

Filesystems
Filesystem              1K-blocks        Used  Available Use% Mounted on
/dev/mapper/cachedev1 32759190536 27816864380 4941785484  85% /
none                          492           4        488   1% /dev
tmpfs                         100           0        100   0% /dev/lxd
tmpfs                         100           0        100   0% /dev/.lxd-mounts
tmpfs                     8143860         172    8143688   1% /dev/shm
tmpfs                     3257544         180    3257364   1% /run
tmpfs                        5120           0       5120   0% /run/lock
tmpfs                        4096           0       4096   0% /sys/fs/cgroup
tmpfs                     1628772           0    1628772   0% /run/user/1001

system python: Python 3.10.12

indiserver: /usr/bin/indiserver


User info
uid=0(root) gid=0(root) groups=0(root)

Process info
khall      200 15.9  0.1 107420 25352 ?        Ss   Dec13 166:53 /usr/bin/indiserver -p 7624 indi_simulator_telescope indi_webcam_ccd
khall      201  0.0  0.0  40088  1688 ?        S    Dec13   0:13 indi_simulator_telescope
khall      237  0.0  0.0  36080  2820 ?        SNs  Dec13   0:03 /home/khall/indi-allsky/virtualenv/indi-allsky/bin/python /home/khall/indi-allsky/virtualenv/indi-allsky/bin/gunicorn --config /etc/indi-allsky/gunicorn.conf.py indi_allsky.wsgi
khall      239  0.1  0.3 831060 58076 ?        SNl  Dec13   1:18 /home/khall/indi-allsky/virtualenv/indi-allsky/bin/python /home/khall/indi-allsky/virtualenv/indi-allsky/bin/gunicorn --config /etc/indi-allsky/gunicorn.conf.py indi_allsky.wsgi
khall      255  0.0  0.2 764156 43340 ?        SNsl Dec13   0:07 /home/khall/indi-allsky/virtualenv/indi-allsky/bin/python3 allsky.py --log syslog run
khall      263 10.2  0.5 928256 89636 ?        Sl   Dec13 107:01 /home/khall/indi-allsky/virtualenv/indi-allsky/bin/python3 allsky.py --log syslog run
khall      264  0.3  0.7 914924 118368 ?       SNl  Dec13   3:37 /home/khall/indi-allsky/virtualenv/indi-allsky/bin/python3 allsky.py --log syslog run
khall     7248  4.6 44.4 12799968 7232144 ?    SNl  01:19  22:17 /home/khall/indi-allsky/virtualenv/indi-allsky/bin/python3 allsky.py --log syslog run
khall    12443  0.0  0.1 1049336 24040 ?       SLl  07:29   0:01 indi_webcam_ccd

USB info
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 004: ID 046d:c05a Logitech, Inc. M90/M100 Optical Mouse
Bus 001 Device 003: ID 0665:5161 Cypress Semiconductor USB to Serial
Bus 001 Device 002: ID 1005:b155 Apacer Technology, Inc. Disk Module
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Module info
Module                  Size  Used by
xt_ipvs                16384  0
ip_vs_rr               16384  0
ip_vs_ftp              16384  0
ip_vs                 139264  6 ip_vs_rr,xt_ipvs,ip_vs_ftp
xt_nat                 16384  0
xt_addrtype            16384  6
kvm_intel             225280  0
kvm                   516096  1 kvm_intel
vhost_net              24576  0
vhost                  40960  1 vhost_net
vhost_iotlb            16384  1 vhost
tun                    49152  5 vhost_net
tap                    24576  1 vhost_net
thunderbolt_icm        49152  0
rfcomm                 69632  0
usdm_drv               94208  0
intel_qat             286720  1 usdm_drv
uio                    20480  1 intel_qat
iscsi_tcp              20480  0
libiscsi_tcp           28672  1 iscsi_tcp
libiscsi               53248  2 libiscsi_tcp,iscsi_tcp
scsi_transport_iscsi    90112  4 libiscsi_tcp,iscsi_tcp,libiscsi
fbdisk                 36864  0
xt_conntrack           16384  5
cfg80211              397312  0
br_netfilter           24576  0
bridge                172032  1 br_netfilter
stp                    16384  1 bridge
bonding               163840  0
dummy                  16384  0
xt_connmark            16384  2
xt_TCPMSS              16384  0
xt_LOG                 16384  0
xt_set                 16384  8
ip_set_hash_netiface    45056  1
ip_set_hash_net        45056  7
ip_set                 40960  3 ip_set_hash_netiface,xt_set,ip_set_hash_net
xt_MASQUERADE          16384  7
xt_REDIRECT            16384  0
iptable_nat            16384  1
nf_nat                 36864  5 xt_nat,iptable_nat,xt_MASQUERADE,xt_REDIRECT,ip_vs_ftp
xt_policy              16384  0
xt_mark                16384  12
8021q                  28672  0
uvcvideo              106496  0
videobuf2_v4l2         24576  1 uvcvideo
videobuf2_vmalloc      16384  1 uvcvideo
videobuf2_memops       16384  1 videobuf2_vmalloc
videobuf2_common       45056  2 videobuf2_v4l2,uvcvideo
snd_usb_caiaq          49152  0
snd_usb_audio         262144  0
snd_usbmidi_lib        28672  1 snd_usb_audio
snd_seq_midi           16384  0
snd_rawmidi            32768  3 snd_seq_midi,snd_usbmidi_lib,snd_usb_caiaq
fnotify                61440  4
nfsd                 1204224  68 fnotify
udf                   114688  0
isofs                  45056  0
iTCO_wdt               16384  1
vfio_pci               61440  0
irqbypass              16384  2 vfio_pci,kvm
vfio_virqfd            16384  1 vfio_pci
vfio                   28672  1 vfio_pci
dm_cache_smq           36864  1
dm_cache              139264  3 dm_cache_smq
ipv6                  475136  656 bridge,br_netfilter
exfat                  77824  0
ufsd                  794624  0
jnl                    32768  1 ufsd
cdc_acm                32768  0
pl2303                 24576  0
usbserial              40960  1 pl2303
qm2_i2c                16384  0
i2c_imc                20480  0
intel_ips              24576  0
drbd                  413696  2
lru_cache              16384  1 drbd
flashcache            167936  0
dm_tier_hro_algo       24576  1
dm_thin_pool          229376  11 dm_tier_hro_algo
dm_bio_prison          24576  2 dm_thin_pool,dm_cache
dm_persistent_data     81920  2 dm_thin_pool,dm_cache
hal_netlink            16384  0
atlantic              266240  0
r8152                 221184  0
usbnet                 36864  0
mii                    16384  1 usbnet
igb                   225280  0
e1000e                245760  0
mv14xx                651264  0
mpt3sas               368640  0
scsi_transport_sas     40960  1 mpt3sas
raid_class             16384  1 mpt3sas
qla2xxx               856064  0
scsi_transport_fc      57344  1 qla2xxx
k10temp                16384  0
coretemp               16384  0
uas                    28672  0
usb_storage            69632  2 uas
xhci_pci               16384  0
xhci_hcd              184320  1 xhci_pci
usblp                  24576  0
uhci_hcd               45056  0
ehci_pci               16384  0
ehci_hcd               81920  1 ehci_pci

indi version: 2.0.4

indi packages
ii  libindi-data                         2.0.4+202309301936~ubuntu22.04.1          all          Instrument-Neutral Device Interface library -- shared data
ii  libindi-dev                          2.0.4+202309301936~ubuntu22.04.1          amd64        Instrument-Neutral Device Interface library -- development files
ii  libindi1:amd64                       2.0.4+202309301936~ubuntu22.04.1          amd64        Instrument-Neutral Device Interface library -- shared library

libcamera: not detected

libcamera packages

libcamera cameras
libcamera-hello not available

python packages
ii  libpython3-dev:amd64                 3.10.6-1~22.04                            amd64        header files and a static library for Python (default)
ii  libpython3-stdlib:amd64              3.10.6-1~22.04                            amd64        interactive high-level object-oriented language (default python3 version)
ii  libpython3.10:amd64                  3.10.12-1~22.04.3                         amd64        Shared Python runtime library (version 3.10)
ii  libpython3.10-dev:amd64              3.10.12-1~22.04.3                         amd64        Header files and a static library for Python (v3.10)
ii  libpython3.10-minimal:amd64          3.10.12-1~22.04.3                         amd64        Minimal subset of the Python language (version 3.10)
ii  libpython3.10-stdlib:amd64           3.10.12-1~22.04.3                         amd64        Interactive high-level object-oriented language (standard library, version 3.10)
ii  libpython3.11:amd64                  3.11.0~rc1-1~22.04                        amd64        Shared Python runtime library (version 3.11)
ii  libpython3.11-dev:amd64              3.11.0~rc1-1~22.04                        amd64        Header files and a static library for Python (v3.11)
ii  libpython3.11-minimal:amd64          3.11.0~rc1-1~22.04                        amd64        Minimal subset of the Python language (version 3.11)
ii  libpython3.11-stdlib:amd64           3.11.0~rc1-1~22.04                        amd64        Interactive high-level object-oriented language (standard library, version 3.11)
ii  python-apt-common                    2.4.0ubuntu2                              all          Python interface to libapt-pkg (locales)
ii  python3                              3.10.6-1~22.04                            amd64        interactive high-level object-oriented language (default python3 version)
ii  python3-apt                          2.4.0ubuntu2                              amd64        Python 3 interface to libapt-pkg
ii  python3-blinker                      1.4+dfsg1-0.4                             all          fast, simple object-to-object and broadcast signaling library
ii  python3-cffi-backend:amd64           1.15.0-1build2                            amd64        Foreign Function Interface for Python 3 calling C code - runtime
ii  python3-cryptography                 3.4.8-1ubuntu2.1                          amd64        Python library exposing cryptographic recipes and primitives (Python 3)
ii  python3-dbus                         1.2.18-3build1                            amd64        simple interprocess messaging system (Python 3 interface)
ii  python3-dev                          3.10.6-1~22.04                            amd64        header files and a static library for Python (default)
ii  python3-distlib                      0.3.4-1                                   all          low-level components of python distutils2/packaging
ii  python3-distro                       1.7.0-1                                   all          Linux OS platform information API
ii  python3-distro-info                  1.1ubuntu0.1                              all          information about distributions' releases (Python 3 module)
ii  python3-distutils                    3.10.8-1~22.04                            all          distutils package for Python 3.x
ii  python3-filelock                     3.6.0-1                                   all          platform independent file locking module
ii  python3-gi                           3.42.1-0ubuntu1                           amd64        Python 3 bindings for gobject-introspection libraries
ii  python3-httplib2                     0.20.2-2                                  all          comprehensive HTTP client library written for Python3
ii  python3-importlib-metadata           4.6.4-1                                   all          library to access the metadata for a Python package - Python 3.x
ii  python3-jeepney                      0.7.1-3                                   all          pure Python D-Bus interface
ii  python3-jwt                          2.3.0-1ubuntu0.2                          all          Python 3 implementation of JSON Web Token
ii  python3-keyring                      23.5.0-1                                  all          store and access your passwords safely
ii  python3-launchpadlib                 1.10.16-1                                 all          Launchpad web services client library (Python 3)
ii  python3-lazr.restfulclient           0.14.4-1                                  all          client for lazr.restful-based web services (Python 3)
ii  python3-lazr.uri                     1.0.6-2                                   all          library for parsing, manipulating, and generating URIs
ii  python3-lib2to3                      3.10.8-1~22.04                            all          Interactive high-level object-oriented language (lib2to3)
ii  python3-minimal                      3.10.6-1~22.04                            amd64        minimal subset of the Python language (default python3 version)
ii  python3-more-itertools               8.10.0-2                                  all          library with routines for operating on iterables, beyond itertools (Python 3)
ii  python3-netifaces:amd64              0.11.0-1build2                            amd64        portable network interface information - Python 3.x
ii  python3-oauthlib                     3.2.0-1ubuntu0.1                          all          generic, spec-compliant implementation of OAuth for Python3
ii  python3-pip                          22.0.2+dfsg-1ubuntu0.4                    all          Python package installer
ii  python3-pip-whl                      22.0.2+dfsg-1ubuntu0.4                    all          Python package installer (pip wheel)
ii  python3-pkg-resources                59.6.0-1.2ubuntu0.22.04.1                 all          Package Discovery and Resource Access using pkg_resources
ii  python3-platformdirs                 2.5.1-1                                   all          determining appropriate platform-specific directories (Python 3)
ii  python3-pyparsing                    2.4.7-1                                   all          alternative to creating and executing simple grammars - Python 3.x
ii  python3-secretstorage                3.3.1-1                                   all          Python module for storing secrets - Python 3.x version
ii  python3-setuptools                   59.6.0-1.2ubuntu0.22.04.1                 all          Python3 Distutils Enhancements
ii  python3-setuptools-whl               59.6.0-1.2ubuntu0.22.04.1                 all          Python Distutils Enhancements (wheel package)
ii  python3-six                          1.16.0-3ubuntu1                           all          Python 2 and 3 compatibility library (Python 3 interface)
ii  python3-software-properties          0.99.22.8                                 all          manage the repositories that you install software from
ii  python3-tk:amd64                     3.10.8-1~22.04                            amd64        Tkinter - Writing Tk applications with Python 3.x
ii  python3-venv                         3.10.6-1~22.04                            amd64        venv module for python3 (default python3 version)
ii  python3-virtualenv                   20.13.0+ds-2                              all          Python virtual environment creator
ii  python3-wadllib                      1.3.6-1                                   all          Python 3 library for navigating WADL files
ii  python3-wheel                        0.37.1-2ubuntu0.22.04.1                   all          built-package format for Python
ii  python3-wheel-whl                    0.37.1-2ubuntu0.22.04.1                   all          built-package format for Python (wheel wheel)
ii  python3-yaml                         5.4.1-1ubuntu1                            amd64        YAML parser and emitter for Python3
ii  python3-zipp                         1.0.0-3                                   all          pathlib-compatible Zipfile object wrapper - Python 3.x
ii  python3.10                           3.10.12-1~22.04.3                         amd64        Interactive high-level object-oriented language (version 3.10)
ii  python3.10-dev                       3.10.12-1~22.04.3                         amd64        Header files and a static library for Python (v3.10)
ii  python3.10-minimal                   3.10.12-1~22.04.3                         amd64        Minimal subset of the Python language (version 3.10)
ii  python3.10-venv                      3.10.12-1~22.04.3                         amd64        Interactive high-level object-oriented language (pyvenv binary, version 3.10)
ii  python3.11                           3.11.0~rc1-1~22.04                        amd64        Interactive high-level object-oriented language (version 3.11)
ii  python3.11-dev                       3.11.0~rc1-1~22.04                        amd64        Header files and a static library for Python (v3.11)
ii  python3.11-minimal                   3.11.0~rc1-1~22.04                        amd64        Minimal subset of the Python language (version 3.11)
ii  python3.11-venv                      3.11.0~rc1-1~22.04                        amd64        Interactive high-level object-oriented language (pyvenv binary, version 3.11)

Detected indi-allsky virtualenv
virtualenv python: Python 3.11.0rc1
virtualenv PATH: /home/khall/indi-allsky/virtualenv/indi-allsky/bin:/usr/local/bin:/usr/bin:/bin
virtualenv python modules
alembic==1.12.1
apache-libcloud==3.8.0
argon2-cffi==23.1.0
argon2-cffi-bindings==21.2.0
asciitree==0.3.3
astroalign==2.5.1
astropy==5.3.4
astropy-healpix==1.0.0
astroscrappy==1.1.0
bcrypt==4.0.1
blinker==1.7.0
boto3==1.29.6
botocore==1.32.6
Bottleneck==1.3.7
cachetools==5.3.2
ccdproc==2.4.1
certifi==2023.11.17
cffi==1.16.0
charset-normalizer==3.3.2
click==8.1.7
cloudpickle==3.0.0
cryptography==41.0.5
Cython==3.0.5
dask==2023.11.0
dbus-python==1.3.2
ephem==4.1.5
fasteners==0.19
Flask==3.0.0
Flask-Login==0.6.3
Flask-Migrate==4.0.5
Flask-SQLAlchemy==3.1.1
Flask-WTF==1.2.1
fsspec==2023.10.0
google-api-core==2.14.0
google-auth==2.23.4
google-cloud-core==2.3.3
google-cloud-storage==2.13.0
google-crc32c==1.5.0
google-resumable-media==2.6.0
googleapis-common-protos==1.61.0
greenlet==3.0.1
gunicorn==21.2.0
idna==3.6
imageio==2.33.0
imageio-ffmpeg==0.4.9
importlib-metadata==6.8.0
inotify==0.2.10
is-safe-url==1.0
itsdangerous==2.1.2
Jinja2==3.1.2
jmespath==1.0.1
lazy_loader==0.3
locket==1.0.0
lxml==4.9.3
Mako==1.3.0
MarkupSafe==2.1.3
mysql-connector-python==8.2.0
networkx==3.2.1
nose==1.3.7
numcodecs==0.12.1
numpy==1.26.2
opencv-python-headless==4.8.1.78
packaging==23.2
paho-mqtt==1.6.1
paramiko==3.3.1
partd==1.4.1
passlib==1.7.4
piexif==1.1.3
Pillow==10.1.0
prettytable==3.9.0
protobuf==4.21.12
psutil==5.9.6
pyasn1==0.5.1
pyasn1-modules==0.3.0
pycparser==2.21
pycurl==7.45.2
pyerfa==2.0.1.1
pygifsicle==1.0.7
pyindi-client @ git+https://github.com/indilib/pyindi-client.git@6f8fa8042f60f7b3079f6d66a0b8ee720af09d64
PyMySQL==1.1.0
PyNaCl==1.5.0
python-dateutil==2.8.2
PyYAML==6.0.1
rawpy==0.18.1
reproject==0.13.0
requests==2.31.0
rsa==4.9
s3transfer==0.7.0
scikit-image==0.22.0
scipy==1.11.4
semantic-version==2.10.0
sep==1.2.1
setuptools-rust==1.8.1
shapely==2.0.2
six==1.16.0
SQLAlchemy==2.0.23
support-developer==1.0.5
tifffile==2023.9.26
toolz==0.12.0
typing_extensions==4.8.0
urllib3==2.0.7
wcwidth==0.2.12
Werkzeug==3.0.1
WTForms==3.1.1
zarr==2.16.1
zipp==3.17.0

#################################
###     end support info      ###
#################################

I do sometime see API authentication failures in the log throughout the day, however the vast majority upload fine as in the 2nd logs below.

cat /var/log/indi-allsky/webapp-indi-allsky.log | grep ERROR
Dec 14 01:35:00 allskynas [ERROR] MainProcess-239 syncapi_views.dispatch_request() #70: Authentication failure: Unable to authenticate API key
Dec 14 03:55:00 allskynas [ERROR] MainProcess-239 syncapi_views.dispatch_request() #70: Authentication failure: Unable to authenticate API key
Dec 14 04:30:03 allskynas [ERROR] MainProcess-239 syncapi_views.dispatch_request() #70: Authentication failure: Unable to authenticate API key
Dec 14 04:49:20 allskynas [ERROR] MainProcess-239 syncapi_views.dispatch_request() #70: Authentication failure: Unable to authenticate API key
Dec 14 05:01:37 allskynas [ERROR] MainProcess-239 syncapi_views.dispatch_request() #70: Authentication failure: Unable to authenticate API key
Dec 14 05:50:00 allskynas [ERROR] MainProcess-239 syncapi_views.dispatch_request() #70: Authentication failure: Unable to authenticate API key
Dec 14 05:55:01 allskynas [ERROR] MainProcess-239 syncapi_views.dispatch_request() #70: Authentication failure: Unable to authenticate API key
Dec 14 07:35:02 allskynas [ERROR] MainProcess-239 syncapi_views.dispatch_request() #70: Authentication failure: Unable to authenticate API key
Dec 14 07:39:59 allskynas [ERROR] MainProcess-239 syncapi_views.dispatch_request() #70: Authentication failure: Unable to authenticate API key

Dec 14 04:56:22 allskynas [INFO] MainProcess-239 syncapi_views.processPost() #502: Uploaded image: /var/www/html/allsky/images/ccd_6440f61c-0c55-47e9-bb50-d3c7cadb5e8e/20231214/day/14_04/ccd1_20231214_045620.jpg
Dec 14 04:56:38 allskynas [INFO] MainProcess-239 syncapi_views.processPost() #502: Uploaded image: /var/www/html/allsky/images/ccd_6440f61c-0c55-47e9-bb50-d3c7cadb5e8e/20231214/day/14_04/ccd1_20231214_045635.jpg
Dec 14 04:56:52 allskynas [INFO] MainProcess-239 syncapi_views.processPost() #502: Uploaded image: /var/www/html/allsky/images/ccd_6440f61c-0c55-47e9-bb50-d3c7cadb5e8e/20231214/day/14_04/ccd1_20231214_045650.jpg
Dec 14 04:57:06 allskynas [INFO] MainProcess-239 syncapi_views.processPost() #502: Uploaded image: /var/www/html/allsky/images/ccd_6440f61c-0c55-47e9-bb50-d3c7cadb5e8e/20231214/day/14_04/ccd1_20231214_045705.jpg
Dec 14 04:57:21 allskynas [INFO] MainProcess-239 syncapi_views.processPost() #502: Uploaded image: /var/www/html/allsky/images/ccd_6440f61c-0c55-47e9-bb50-d3c7cadb5e8e/20231214/day/14_04/ccd1_20231214_045720.jpg
Dec 14 04:57:22 allskynas [INFO] MainProcess-239 syncapi_views.processPost() #239: Uploaded file: /var/www/html/allsky/images/ccd_6440f61c-0c55-47e9-bb50-d3c7cadb5e8e/20231213/allsky-keogram_ccd1_20231213_night.jpg
Dec 14 04:57:26 allskynas [INFO] MainProcess-239 syncapi_views.processPost() #239: Uploaded file: /var/www/html/allsky/images/ccd_6440f61c-0c55-47e9-bb50-d3c7cadb5e8e/20231213/allsky-startrail_ccd1_20231213_night.jpg
Dec 14 05:01:37 allskynas [ERROR] MainProcess-239 syncapi_views.dispatch_request() #70: Authentication failure: Unable to authenticate API key
Dec 14 05:01:39 allskynas [INFO] MainProcess-239 syncapi_views.processPost() #502: Uploaded image: /var/www/html/allsky/images/ccd_6440f61c-0c55-47e9-bb50-d3c7cadb5e8e/20231214/day/14_04/ccd1_20231214_045735.jpg
Dec 14 05:01:40 allskynas [INFO] MainProcess-239 syncapi_views.processPost() #502: Uploaded image: /var/www/html/allsky/images/ccd_6440f61c-0c55-47e9-bb50-d3c7cadb5e8e/20231214/day/14_04/ccd1_20231214_045750.jpg
Dec 14 05:01:41 allskynas [INFO] MainProcess-239 syncapi_views.processPost() #502: Uploaded image: /var/www/html/allsky/images/ccd_6440f61c-0c55-47e9-bb50-d3c7cadb5e8e/20231214/day/14_04/ccd1_20231214_045805.jpg
Dec 14 05:01:42 allskynas [INFO] MainProcess-239 syncapi_views.processPost() #502: Uploaded image: /var/www/html/allsky/images/ccd_6440f61c-0c55-47e9-bb50-d3c7cadb5e8e/20231214/day/14_04/ccd1_20231214_045820.jpg
Dec 14 05:01:44 allskynas [INFO] MainProcess-239 syncapi_views.processPost() #502: Uploaded image: /var/www/html/allsky/images/ccd_6440f61c-0c55-47e9-bb50-d3c7cadb5e8e/20231214/day/14_04/ccd1_20231214_045835.jpg
Dec 14 05:01:45 allskynas [INFO] MainProcess-239 syncapi_views.processPost() #502: Uploaded image: /var/www/html/allsky/images/ccd_6440f61c-0c55-47e9-bb50-d3c7cadb5e8e/20231214/day/14_04/ccd1_20231214_045850.jpg
Dec 14 05:01:46 allskynas [INFO] MainProcess-239 syncapi_views.processPost() #502: Uploaded image: /var/www/html/allsky/images/ccd_6440f61c-0c55-47e9-bb50-d3c7cadb5e8e/20231214/day/14_04/ccd1_20231214_045905.jpg
Dec 14 05:01:47 allskynas [INFO] MainProcess-239 syncapi_views.processPost() #502: Uploaded image: /var/www/html/allsky/images/ccd_6440f61c-0c55-47e9-bb50-d3c7cadb5e8e/20231214/day/14_04/ccd1_20231214_045920.jpg
Dec 14 05:01:49 allskynas [INFO] MainProcess-239 syncapi_views.processPost() #502: Uploaded image: /var/www/html/allsky/images/ccd_6440f61c-0c55-47e9-bb50-d3c7cadb5e8e/20231214/day/14_04/ccd1_20231214_045935.jpg
Dec 14 05:01:50 allskynas [INFO] MainProcess-239 syncapi_views.processPost() #502: Uploaded image: /var/www/html/allsky/images/ccd_6440f61c-0c55-47e9-bb50-d3c7cadb5e8e/20231214/day/14_04/ccd1_20231214_045950.jpg

from indi-allsky.

keno242 commented on June 8, 2024

Ahh - I looked at the logs on the camera connected allsky and the timing of these entries match perfectly... so it's related to Authentication?

~$ cat /var/log/indi-allsky/indi-allsky.log | grep MainProcess-2573 | grep fail
2023-12-14T01:35:01.975774+08:00 allsky [ERROR] MainProcess-2573/Upload-8 uploader.processUpload() #411: Tranfer failure: Sync error: 400
2023-12-14T03:55:00.880859+08:00 allsky [ERROR] MainProcess-2573/Upload-8 uploader.processUpload() #411: Tranfer failure: Sync error: 400
2023-12-14T04:30:05.230182+08:00 allsky [ERROR] MainProcess-2573/Upload-8 uploader.processUpload() #411: Tranfer failure: Sync error: 400
2023-12-14T04:49:21.745002+08:00 allsky [ERROR] MainProcess-2573/Upload-8 uploader.processUpload() #411: Tranfer failure: Sync error: 400
2023-12-14T05:01:39.249530+08:00 allsky [ERROR] MainProcess-2573/Upload-8 uploader.processUpload() #411: Tranfer failure: Sync error: 400
2023-12-14T05:50:02.293152+08:00 allsky [ERROR] MainProcess-2573/Upload-8 uploader.processUpload() #411: Tranfer failure: Sync error: 400
2023-12-14T05:55:03.034341+08:00 allsky [ERROR] MainProcess-2573/Upload-8 uploader.processUpload() #411: Tranfer failure: Sync error: 400
2023-12-14T07:35:03.637277+08:00 allsky [ERROR] MainProcess-2573/Upload-8 uploader.processUpload() #411: Tranfer failure: Sync error: 400
2023-12-14T07:40:01.510468+08:00 allsky [ERROR] MainProcess-2573/Upload-8 uploader.processUpload() #411: Tranfer failure: Sync error: 400

from indi-allsky.

aaronwmorris commented on June 8, 2024

I have been reviewing my code and the API authentication is supposed to happen at the beginning of the connection and not the end. It does not make sense that the API authentication would fail near the end, unless the beginning of the server transaction does not occur until all of the files have been uploaded. I am still investigating.

from indi-allsky.

aaronwmorris commented on June 8, 2024

I think I have confirmed that the processing on the server side does not start until the file upload has completed, since I am not using streamed uploads. So that makes sense.

from indi-allsky.

aaronwmorris commented on June 8, 2024

Merged #1065 to fix.

I am not sure you are on the latest code, but recently I also merged an update to separate the connect timeout from the read timeout.

from indi-allsky.

keno242 commented on June 8, 2024

Yes I was using the version with both timeouts - but also just upgraded both systems to 7.8 to test the latest code.

Same issue I'm afraid - I did try disabling syncapi for images to check if maybe starting a new image transfer, while a video one was still in progress might be causing issues but it made no difference.

python upload_sync.py --no-syncapi-images sync
2023-12-15 12:59:57,979 [INFO] MainProcess upload_sync._get_entry_status() #537: IndiAllSkyDbVideoTable uploading disabled
2023-12-15 12:59:57,980 [INFO] MainProcess upload_sync._get_entry_status() #537: IndiAllSkyDbKeogramTable uploading disabled
2023-12-15 12:59:57,980 [INFO] MainProcess upload_sync._get_entry_status() #537: IndiAllSkyDbStarTrailsTable uploading disabled
2023-12-15 12:59:57,980 [INFO] MainProcess upload_sync._get_entry_status() #537: IndiAllSkyDbStarTrailsVideoTable uploading disabled
2023-12-15 12:59:57,980 [INFO] MainProcess upload_sync._get_entry_status() #568: S3 uploading disabled (IndiAllSkyDbVideoTable)
2023-12-15 12:59:57,980 [INFO] MainProcess upload_sync._get_entry_status() #568: S3 uploading disabled (IndiAllSkyDbKeogramTable)
2023-12-15 12:59:57,981 [INFO] MainProcess upload_sync._get_entry_status() #568: S3 uploading disabled (IndiAllSkyDbStarTrailsTable)
2023-12-15 12:59:57,981 [INFO] MainProcess upload_sync._get_entry_status() #568: S3 uploading disabled (IndiAllSkyDbStarTrailsVideoTable)
2023-12-15 12:59:57,981 [INFO] MainProcess upload_sync._get_entry_status() #568: S3 uploading disabled (IndiAllSkyDbImageTable)
2023-12-15 12:59:57,981 [INFO] MainProcess upload_sync._get_entry_status() #578: S3 uploading disabled (IndiAllSkyDbFitsImageTable)
2023-12-15 12:59:57,981 [INFO] MainProcess upload_sync._get_entry_status() #588: S3 uploading disabled (IndiAllSkyDbRawImageTable)
2023-12-15 12:59:57,988 [INFO] MainProcess upload_sync._get_entry_status() #616: syncapi disabled (IndiAllSkyDbImageTable)
+---------+----------------------------------+----------+---------+
|   Type  |              Table               | Uploaded | Missing |
+---------+----------------------------------+----------+---------+
|  upload |      IndiAllSkyDbVideoTable      |    -     |    -    |
|  upload |     IndiAllSkyDbKeogramTable     |    -     |    -    |
|  upload |   IndiAllSkyDbStarTrailsTable    |    -     |    -    |
|  upload | IndiAllSkyDbStarTrailsVideoTable |    -     |    -    |
|    s3   |      IndiAllSkyDbVideoTable      |    -     |    -    |
|    s3   |     IndiAllSkyDbKeogramTable     |    -     |    -    |
|    s3   |   IndiAllSkyDbStarTrailsTable    |    -     |    -    |
|    s3   | IndiAllSkyDbStarTrailsVideoTable |    -     |    -    |
|    s3   |      IndiAllSkyDbImageTable      |    -     |    -    |
|    s3   |    IndiAllSkyDbFitsImageTable    |    -     |    -    |
|    s3   |    IndiAllSkyDbRawImageTable     |    -     |    -    |
| syncapi |      IndiAllSkyDbVideoTable      |    48    |    52   |
| syncapi |     IndiAllSkyDbKeogramTable     |    30    |    67   |
| syncapi |   IndiAllSkyDbStarTrailsTable    |    19    |    69   |
| syncapi | IndiAllSkyDbStarTrailsVideoTable |    13    |    64   |
| syncapi |      IndiAllSkyDbImageTable      |    -     |    -    |
+---------+----------------------------------+----------+---------+
2023-12-15 13:00:03,121 [INFO] MainProcess upload_sync.sync() #162: Entries to upload: 252
2023-12-15 13:00:03,122 [INFO] MainProcess upload_sync.addUploadEntries() #205: Adding 7 upload entries (245 remaining)
2023-12-15 13:00:03,383 [INFO] MainProcess upload_sync._fileUploadWorkerStart() #720: Starting FileUploader process 1
2023-12-15 13:00:03,406 [INFO] MainProcess generic.connect() #50: Connecting to https://allskynas.discus-ray.ts.net/indi-allsky/sync/v1/video (443) as foobar123 with requests_syncapi_v1
2023-12-15 13:00:03,406 [INFO] MainProcess generic.put() #60: Uploading /var/www/html/allsky/images/ccd_6440f61c-0c55-47e9-bb50-d3c7cadb5e8e/20231214/allsky-timelapse_ccd1_20231214_night.mp4
2023-12-15 13:01:46,549 [INFO] MainProcess requests_syncapi_v1.put() #172: File transferred in 103.1413 s (953.62 kB/s)
2023-12-15 13:01:46,568 [INFO] MainProcess uploader.processUpload() #443: Upload transaction completed in 103.1617 s
2023-12-15 13:01:46,587 [INFO] MainProcess generic.connect() #50: Connecting to https://allskynas.discus-ray.ts.net/indi-allsky/sync/v1/video (443) as foobar123 with requests_syncapi_v1
2023-12-15 13:01:46,587 [INFO] MainProcess generic.put() #60: Uploading /var/www/html/allsky/images/ccd_6440f61c-0c55-47e9-bb50-d3c7cadb5e8e/20231214/allsky-timelapse_ccd1_20231214_day.mp4
2023-12-15 13:07:49,641 [ERROR] MainProcess uploader.processUpload() #412: Tranfer failure: Sync error: 400
2023-12-15 13:07:49,654 [WARNING] MainProcess miscDb.addNotification() #719: Not adding existing notification
2023-12-15 13:07:49,707 [INFO] MainProcess generic.connect() #50: Connecting to https://allskynas.discus-ray.ts.net/indi-allsky/sync/v1/video (443) as foobar123 with requests_syncapi_v1
2023-12-15 13:07:49,707 [INFO] MainProcess generic.put() #60: Uploading /var/www/html/allsky/images/ccd_6440f61c-0c55-47e9-bb50-d3c7cadb5e8e/20231213/allsky-timelapse_ccd1_20231213_night.mp4
^C2023-12-15 13:08:27,244 [WARNING] MainProcess upload_sync.sigint_handler_main() #119: Caught INT signal, shutting down

cat /var/log/indi-allsky/indi-allsky.log | grep fail
2023-12-15T02:10:03.311967+08:00 allsky [ERROR] MainProcess-2573/Upload-8 uploader.processUpload() #411: Tranfer failure: Sync error: 400
2023-12-15T04:48:28.333786+08:00 allsky [ERROR] MainProcess-2573/Upload-8 uploader.processUpload() #411: Tranfer failure: Sync error: 400
2023-12-15T05:00:21.019174+08:00 allsky [ERROR] MainProcess-2573/Upload-8 uploader.processUpload() #411: Tranfer failure: Sync error: 400
2023-12-15T06:40:02.772613+08:00 allsky [ERROR] MainProcess-2573/Upload-8 uploader.processUpload() #411: Tranfer failure: Sync error: 400
2023-12-15T06:45:01.008997+08:00 allsky [ERROR] MainProcess-2573/Upload-8 uploader.processUpload() #411: Tranfer failure: Sync error: 400
2023-12-15T11:40:00.865237+08:00 allsky [ERROR] MainProcess-2573/Upload-8 uploader.processUpload() #411: Tranfer failure: Sync error: 400

And on remote (non camera system)

cat /var/log/indi-allsky/webapp-indi-allsky.log | grep ERROR
Dec 15 02:10:02 allskynas [ERROR] MainProcess-239 syncapi_views.dispatch_request() #70: Authentication failure: Unable to authenticate API key
Dec 15 04:48:26 allskynas [ERROR] MainProcess-239 syncapi_views.dispatch_request() #70: Authentication failure: Unable to authenticate API key
Dec 15 05:00:19 allskynas [ERROR] MainProcess-239 syncapi_views.dispatch_request() #70: Authentication failure: Unable to authenticate API key
Dec 15 06:40:01 allskynas [ERROR] MainProcess-239 syncapi_views.dispatch_request() #70: Authentication failure: Unable to authenticate API key
Dec 15 06:44:59 allskynas [ERROR] MainProcess-239 syncapi_views.dispatch_request() #70: Authentication failure: Unable to authenticate API key
Dec 15 11:39:59 allskynas [ERROR] MainProcess-239 syncapi_views.dispatch_request() #70: Authentication failure: Unable to authenticate API key
Dec 15 12:46:13 allskynas [ERROR] MainProcess-239 syncapi_views.dispatch_request() #70: Authentication failure: Unable to authenticate API key
Dec 15 12:53:22 allskynas [ERROR] MainProcess-239 syncapi_views.dispatch_request() #70: Authentication failure: Unable to authenticate API key
Dec 15 13:07:48 allskynas [ERROR] MainProcess-239 syncapi_views.dispatch_request() #70: Authentication failure: Unable to authenticate API key

from indi-allsky.

aaronwmorris commented on June 8, 2024

Just for giggles, can you check to see if there is a time delta between both the local and remote servers?

from indi-allsky.

keno242 commented on June 8, 2024

Yes did check that and rechecked now pretty much within a second of each other...

And you will have noticed I did get one large file transfer through. I know you said you don't using the stream large files
feature, but I've noticed other around the web having issues with large files - is it worth a shot?

from indi-allsky.

aaronwmorris commented on June 8, 2024

To be perfectly honest, I have been having trouble understanding the full implications of streaming the uploads. It is difficult to find examples of what I am trying. Most examples only use one file, but the syncapi transfers are actually two files each: one file is a json file containing metadata and the second file is the actual image/movie.

I have made a first pass at this, but I do not expect this to really stream the way we need it to.

Merged #1067

from indi-allsky.

keno242 commented on June 8, 2024

Thanks for having a go at this...

But still getting after updating on both sites....

2023-12-16 09:30:05,166 [INFO] MainProcess upload_sync._fileUploadWorkerStart() #720: Starting FileUploader process 1
2023-12-16 09:30:05,190 [INFO] MainProcess generic.connect() #50: Connecting to https://allskynas.discus-ray.ts.net/indi-allsky/sync/v1/video (443) as foobar123 with requests_syncapi_v1
2023-12-16 09:30:05,190 [INFO] MainProcess generic.put() #60: Uploading /var/www/html/allsky/images/ccd_6440f61c-0c55-47e9-bb50-d3c7cadb5e8e/20231215/allsky-timelapse_ccd1_20231215_day.mp4
2023-12-16 09:36:30,554 [ERROR] MainProcess uploader.processUpload() #412: Tranfer failure: Sync error: 400
2023-12-16 09:36:30,569 [WARNING] MainProcess miscDb.addNotification() #719: Not adding existing notification
2023-12-16 09:36:30,622 [INFO] MainProcess generic.connect() #50: Connecting to https://allskynas.discus-ray.ts.net/indi-allsky/sync/v1/video (443) as foobar123 with requests_syncapi_v1
2023-12-16 09:36:30,622 [INFO] MainProcess generic.put() #60: Uploading /var/www/html/allsky/images/ccd_6440f61c-0c55-47e9-bb50-d3c7cadb5e8e/20231214/allsky-timelapse_ccd1_20231214_day.mp4
2023-12-16 09:43:42,050 [ERROR] MainProcess uploader.processUpload() #412: Tranfer failure: Sync error: 400

with corresponding Authentication error on the remote (non camera) site.

ec 16 08:59:59 allskynas [ERROR] MainProcess-239 syncapi_views.dispatch_request() #70: Authentication failure: Unable to authenticate API key
Dec 16 09:04:59 allskynas [ERROR] MainProcess-239 syncapi_views.dispatch_request() #70: Authentication failure: Unable to authenticate API key
Dec 16 09:25:27 allskynas [ERROR] MainProcess-239 syncapi_views.dispatch_request() #70: Authentication failure: Unable to authenticate API key
Dec 16 09:36:28 allskynas [ERROR] MainProcess-273 syncapi_views.dispatch_request() #70: Authentication failure: Unable to authenticate API key
Dec 16 09:43:40 allskynas [ERROR] MainProcess-273 syncapi_views.dispatch_request() #70: Authentication failure: Unable to authenticate API key

Not sure if this is relevant but it references 'We have a web application which accepts (large) files together with some meta data' psf/requests#1584 and the recommendation is to use this https://github.com/requests/toolbelt

However - I'm open to trying to achieve what I want in another way - basically a local (to me) / public instance (with a fast internet connection) of the allsky data (images and timelapses) taken from a remote observatory. I could even manually (via python trigger a scp file copy of the videos as long as I knew how to update the metadata on the other send tell it has a new file.

from indi-allsky.

aaronwmorris commented on June 8, 2024

I made another commit to extend the HMAC signature forward in time. The previous just extended backwards. I do not think this will fix it, but maybe I had my thinking backwards. Let me know.

Merged #1068

from indi-allsky.

keno242 commented on June 8, 2024

Yea same issue... after updating on both systems...

+---------+----------------------------------+----------+---------+
2023-12-16 11:22:21,888 [INFO] MainProcess upload_sync.sync() #162: Entries to upload: 20
2023-12-16 11:22:21,889 [INFO] MainProcess upload_sync.addUploadEntries() #205: Adding 7 upload entries (13 remaining)
2023-12-16 11:22:22,156 [INFO] MainProcess upload_sync._fileUploadWorkerStart() #720: Starting FileUploader process 1
2023-12-16 11:22:22,181 [INFO] MainProcess generic.connect() #50: Connecting to https://allskynas.discus-ray.ts.net/indi-allsky/sync/v1/video (443) as foobar123 with requests_syncapi_v1
2023-12-16 11:22:22,181 [INFO] MainProcess generic.put() #60: Uploading /var/www/html/allsky/images/ccd_6440f61c-0c55-47e9-bb50-d3c7cadb5e8e/20231215/allsky-timelapse_ccd1_20231215_day.mp4
2023-12-16 11:29:02,500 [ERROR] MainProcess uploader.processUpload() #412: Tranfer failure: Sync error: 400

Dec 16 11:29:00 allskynas [ERROR] MainProcess-287 syncapi_views.dispatch_request() #70: Authentication failure: Unable to authenticate API key

from indi-allsky.

keno242 commented on June 8, 2024

BOOM! Nice work Aaron - 3 out of 3 ran fine - thanks for persisting with this!

2023-12-16 17:18:20,408 [INFO] MainProcess upload_sync.sync() #162: Entries to upload: 20
2023-12-16 17:18:20,410 [INFO] MainProcess upload_sync.addUploadEntries() #205: Adding 7 upload entries (13 remaining)
2023-12-16 17:18:20,688 [INFO] MainProcess upload_sync._fileUploadWorkerStart() #720: Starting FileUploader process 1
2023-12-16 17:18:20,712 [INFO] MainProcess generic.connect() #50: Connecting to https://allskynas.discus-ray.ts.net/indi-allsky/sync/v1/video (443) as foobar123 with requests_syncapi_v1
2023-12-16 17:18:20,712 [INFO] MainProcess generic.put() #60: Uploading /var/www/html/allsky/images/ccd_6440f61c-0c55-47e9-bb50-d3c7cadb5e8e/20231215/allsky-timelapse_ccd1_20231215_day.mp4
2023-12-16 17:24:26,835 [INFO] MainProcess requests_syncapi_v1.put() #179: File transferred in 366.1209 s (959.61 kB/s)
2023-12-16 17:24:26,838 [INFO] MainProcess uploader.processUpload() #443: Upload transaction completed in 366.1262 s
2023-12-16 17:24:26,861 [INFO] MainProcess generic.connect() #50: Connecting to https://allskynas.discus-ray.ts.net/indi-allsky/sync/v1/video (443) as foobar123 with requests_syncapi_v1
2023-12-16 17:24:26,861 [INFO] MainProcess generic.put() #60: Uploading /var/www/html/allsky/images/ccd_6440f61c-0c55-47e9-bb50-d3c7cadb5e8e/20231214/allsky-timelapse_ccd1_20231214_day.mp4
2023-12-16 17:34:05,369 [INFO] MainProcess requests_syncapi_v1.put() #179: File transferred in 578.5065 s (616.36 kB/s)
2023-12-16 17:34:05,374 [INFO] MainProcess uploader.processUpload() #443: Upload transaction completed in 578.5132 s
2023-12-16 17:34:05,389 [INFO] MainProcess generic.connect() #50: Connecting to https://allskynas.discus-ray.ts.net/indi-allsky/sync/v1/video (443) as foobar123 with requests_syncapi_v1
2023-12-16 17:34:05,389 [INFO] MainProcess generic.put() #60: Uploading /var/www/html/allsky/images/ccd_6440f61c-0c55-47e9-bb50-d3c7cadb5e8e/20231213/allsky-timelapse_ccd1_20231213_day.mp4
2023-12-16 17:44:34,481 [INFO] MainProcess requests_syncapi_v1.put() #179: File transferred in 629.0906 s (570.91 kB/s)
2023-12-16 17:44:34,486 [INFO] MainProcess uploader.processUpload() #443: Upload transaction completed in 629.0963 s

Dec 16 17:24:23 allskynas [INFO] MainProcess-267 syncapi_views.processPost() #242: Uploaded file: /var/www/html/allsky/images/ccd_6440f61c-0c55-47e9-bb50-d3c7cadb5e8e/20231215/allsky-timelapse_ccd1_20231215_day.mp4
Dec 16 17:34:02 allskynas [INFO] MainProcess-267 syncapi_views.processPost() #242: Uploaded file: /var/www/html/allsky/images/ccd_6440f61c-0c55-47e9-bb50-d3c7cadb5e8e/20231214/allsky-timelapse_ccd1_20231214_day.mp4
Dec 16 17:44:31 allskynas [INFO] MainProcess-267 syncapi_views.processPost() #242: Uploaded file: /var/www/html/allsky/images/ccd_6440f61c-0c55-47e9-bb50-d3c7cadb5e8e/20231213/allsky-timelapse_ccd1_20231213_day.mp4

from indi-allsky.

syncapi timelapse video transfers failing but images OK even with 900s timeouts about indi-allsky HOT 17 CLOSED

Comments (17)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent