Rebuilding db wiki about indi-allsky HOT 32 CLOSED

Okay, pull down the latest code and give it a go.

from indi-allsky.

Yep, I'll get this fixed tonight or tomorrow.

from indi-allsky.

libcamera_imx378 is the name I would expect you should use.

from indi-allsky.

I have made some changes in #953. It should fix the ADU related error and allow you to override the temperature.

from indi-allsky.

The dark frames are standard bias + dark frames like you have in traditional astrophotography.

My concept of a Bad Pixel Map might differ from others. I created the BPM due to my ASI385MC. It had one or two hot pixels that would "pulse". The pixels would slowly increase (then decrease) from barely visible to max value over several frames. The regular dark frames using averages or sigma clipping would not eliminate these.

from indi-allsky.

I updated the instructions, but let me run some tests here. I am sure the image import is probably not working in its current condition.

from indi-allsky.

Sadly, install fails:

ERROR [flask_migrate] Error: Target database is not up to date.

The script exited abnormally, please try to run again...

from indi-allsky.

Sorry, this one is on me. I forgot to merge my PR in GitHub.

#947

I updated the instructions to stop more services. Its possible the database is getting recreated by the web process, so that needs to be stopped

from indi-allsky.

Thanks @aaronwmorris, updated again and retrying the rebuild instructions.

Getting -bash: flask: command not found at step 4?

from indi-allsky.

...actually, I missed that the setup script failed again - same error ERROR [flask_migrate] Error: Target database is not up to date.

from indi-allsky.

@aaronwmorris very strange. Reattempted again this morning: no updates from git pull, however the installation/setup did complete this time.

However, exactly the same when coming to rebuild the database in step 4: -bash: flask: command not found

...so, installed via sudo pip install flask, but again step 4 errors out:

pi@pi:~/indi-allsky $  flask db revision --autogenerate
Error: While importing 'app', an ImportError was raised:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/flask/cli.py", line 219, in locate_app
    __import__(module_name)
  File "/home/pi/indi-allsky/app.py", line 3, in <module>
    from indi_allsky.flask import create_app
  File "/home/pi/indi-allsky/indi_allsky/flask/__init__.py", line 7, in <module>
    from flask_sqlalchemy import SQLAlchemy
ModuleNotFoundError: No module named 'flask_sqlalchemy'


Usage: flask [OPTIONS] COMMAND [ARGS]...
Try 'flask --help' for help.

Error: No such command 'db'.

...and...

pi@pi:~/indi-allsky $  flask db upgrade head
Error: While importing 'app', an ImportError was raised:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/flask/cli.py", line 219, in locate_app
    __import__(module_name)
  File "/home/pi/indi-allsky/app.py", line 3, in <module>
    from indi_allsky.flask import create_app
  File "/home/pi/indi-allsky/indi_allsky/flask/__init__.py", line 7, in <module>
    from flask_sqlalchemy import SQLAlchemy
ModuleNotFoundError: No module named 'flask_sqlalchemy'


Usage: flask [OPTIONS] COMMAND [ARGS]...
Try 'flask --help' for help.

Error: No such command 'db'.

from indi-allsky.

Make sure you are activating the virtualenv when running these commands.

source virtualenv/indi-allsky/bin/activate

I have made some additional clarifications in the wiki page. I appreciate you going through this with me.

from indi-allsky.

Ahhhhh - brilliant, thank you - yes, much better now.

Re; Step 7 - camera driver name. Is there a quick way to determine this on the command line?

from indi-allsky.

If you are using an INDI based camera, this should work:

indi_getprop | grep DRIVER_NAME | awk -F= '{print $2}'

from indi-allsky.

Thanks @aaronwmorris, i think I cracked it - at least, I'm up and running again (but, sadly have lost all settings - should have jotted them all down) and seem to have my timelapses etc in the library.

I'd not clicked the above line was for retrieving the driver, but think there's a problem with this - in the scenario someone would be doing this rebuild, they'll likely not have any of the services running (e.g. if they're stuck like I was - sql corruption and the services won't start successfully... though the issue for me may have been more than I'd tried to update/failed - as this line only returned the CCD and Telescope simulator entries).

I used (guessed at) "libcamera_imx378", but now we're running it looks like it should have been libcamera-still.

ps - loving all the new informational / tool pages 👍

from indi-allsky.

Yeah, It is tough to predict the correct camera name. Normally, this is a string returned by INDI. ZWO cameras are usually very standard. I think it was QHY sometimes adds part of the serial number to the name. Svbony will sometimes append a "0" (like an index if you have multiple cameras).

The worst that can happen if you get the name wrong is you end up with two cameras in the web interface. If that happens, you can just rebuild the database again with the correct name.

from indi-allsky.

@aaronwmorris noticed that it's not using my dark or bad pixel library since the sql failure and update...

All my images, star trails, keograms, timelapses etc. all imported successfully though?

That command line doesn't work for me - checking now with the camera working still only returns the simulators:

pi@Pi:~/indi-allsky $ source virtualenv/indi-allsky/bin/activate
(indi-allsky) pi@Pi:~/indi-allsky $ indi_getprop | grep DRIVER_NAME | awk -F= '{print $2}'
CCD Simulator
Telescope Simulator
(indi-allsky) pi@Pi:~/indi-allsky $

Any suggestions how to verify the correct camera driver to use and I'll try another dB rebuild?

Do I remember seeing somewhere another method where we can re-index these? Installed on the roof, I took a comprehensive set and don't fancy going through all that again.

Thinking out-loud - this isn't linked to the previous thing I mentioned to you about the GUID style naming for camera folders - I now have 3 after this latest rebuild:

from indi-allsky.

There is a different script for re-importing the dark frames. ./misc/import_darks_frames.py Since the metadata is so specific with dark frames, I created a different script to re-import these.

from indi-allsky.

@aaronwmorris brilliant - thank you - just trying this (shame it can't regex / strpos the first three letters to determine the image type automatically?)

EDIT:

It's also struggling with the temperature:

File: /var/www/html/allsky/images/darks/bpm_ccd2_16bit_60s_gain16_bin1_25c_20230315_153551.fit

What type of frame?

0 - Dark Frame
1 - Bad Pixel Map
2 - Skip
? 1

Exposure:    60.0
Gain:        16
Bin mode:    1
Temperature: -273.1
Bit depth:   16

from indi-allsky.

...there's no override for settings so I can't import them?

from indi-allsky.

I will look at making some enhancements. However, if you do not have temperature matched darks, the actual temperature recorded does not matter.

from indi-allsky.

These were temperature matched - spent a couple of days with the freezer, fridge and outside, so be useful to get them indexed again properly.

I can easily regex/str-match process them myself or just with excel if I can import directly into the db?

from indi-allsky.

Thanks @aaronwmorris

To just get going I decided to just ignore the temperature, but it's failed:

File: /var/www/html/allsky/images/darks/bpm_ccd2_16bit_60s_gain16_bin1_25c_20230315_153551.fit

What type of frame?

0 - Dark Frame
1 - Bad Pixel Map
2 - Skip
? 1

Exposure:    60.0
Gain:        16
Bin mode:    1
Temperature: -273.1
Bit depth:   16

Import frame with the above parameters?

0 - No
1 - Yes
? 1
[INFO]: Adding bad pixel map /var/www/html/allsky/images/darks/bpm_ccd2_16bit_60s_gain16_bin1_25c_20230315_153551.fit to DB
Traceback (most recent call last):
  File "/home/pi/indi-allsky/./misc/import_darks_frames.py", line 362, in <module>
    idf.main()
  File "/home/pi/indi-allsky/./misc/import_darks_frames.py", line 279, in main
    self._miscDb.addBadPixelMap(
  File "/home/pi/indi-allsky/indi_allsky/flask/miscDb.py", line 357, in addBadPixelMap
    adu=metadata['adu'],
KeyError: 'adu'

Looking at this and the script, I'm guessing this is relying on meta you've subsequently added to the fits header, which I presume I don't have on these (captured in March 2023)?

On a separate, but related topic... is there a way for us to backup our settings/config once tweaked? I've lost all the settings and it's taken a long long time to get back to this point - now to realise I need to start all over (as still tweaking I'd still not jotted them all down).

(I realised just after I started rebuilding the dB... D'Oh!)

from indi-allsky.

@aaronwmorris thank you, really appreciate that.

from indi-allsky.

@aaronwmorris sorry...

Earlier I re-ran the db rebuild (trying to get these darks) and now (giving up on the dark script) have gone back to the web interface to check it's up again...

libcamera-still (the driver) doesn't appear to be the correct option to use in the db-rebuild - I now don't have a functional camera.

...so, I presume in the case of libcamera users they should use the camera name that they select during the initial setup? (in my case libcamera_imx378 as I'd originally tried)

from indi-allsky.

Thanks @aaronwmorris, it's not crashing out now... but not sure the correct procedure for the temperature override....

For the step: "Would you like to override the temperature?"
Answering with the correct temperature doesn't change it
Answering "yes" gives no options, E.g. below

For info: it's also not detecting between darks and bad pixel maps correctly: detecting all as "Dark Frames".

[INFO]: Detected frame type: Dark Frame
[INFO]: Detected camera: libcamera
[INFO]: Detected exposure: 60.0
[INFO]: Detected gain: 16
[INFO]: Detected bin mode: 1
[INFO]: Detected temperature: -273.1
[INFO]: Detected bit depth: 16
[INFO]: Detected bayer pattern: BGGR
[WARNING]: Date not logged
##########################################

File: /var/www/html/allsky/images/darks/bpm_ccd2_16bit_60s_gain16_bin1_25c_20230315_153551.fit

What type of frame?

0 - Dark Frame
1 - Bad Pixel Map
2 - Skip
? 1

Would you like to override the temperature? yes

Size:        4056 x 3040
Exposure:    60.0
Gain:        16
Bin mode:    1
Temperature: -273.1
Bit depth:   16
Avg ADU:     0.00

Import frame with the above parameters?

0 - No
1 - Yes
?

EDIT: sorry, it requires a response of "Y" specifically - I'm working through them now.

from indi-allsky.

Sorry, back again...

When it comes to the next image it doesn't allow you to override the temperature and instead uses the value you gave for the previous image:

[Continued from above]
Import frame with the above parameters?

0 - No
1 - Yes
? 1
[INFO]: Adding dark frame /var/www/html/allsky/images/darks/dark_ccd2_16bit_60s_gain16_bin1_25c_20230315_153551.fit to DB
[INFO]: Found fits: /var/www/html/allsky/images/darks/bpm_ccd2_16bit_55s_gain16_bin1_23c_20230315_154614.fit
[INFO]: Detected frame type: Dark Frame
[INFO]: Detected camera: libcamera
[INFO]: Detected exposure: 55.0
[INFO]: Detected gain: 16
[INFO]: Detected bin mode: 1
[INFO]: Detected temperature: 25.0
[INFO]: Detected bit depth: 16
[INFO]: Detected bayer pattern: BGGR
[WARNING]: Date not logged
##########################################

File: /var/www/html/allsky/images/darks/bpm_ccd2_16bit_55s_gain16_bin1_23c_20230315_154614.fit

What type of frame?

0 - Dark Frame
1 - Bad Pixel Map
2 - Skip
? 1

Size:        4056 x 3040
Exposure:    55.0
Gain:        16
Bin mode:    1
Temperature: 25.0
Bit depth:   16
Avg ADU:     0.00

Import frame with the above parameters?

0 - No
1 - Yes
? 0

from indi-allsky.

Hmmm... plot thickens.

I'm just analysing the actual FITS headers and they all seem to contain the correct information?

The only misnomer is that they're all of IMAGETYP 'Dark Frame' (not sure if "Bad Pixel Map" is a valid entry?), but besides that they have the correct values:

• CCD-TEMP
• GAIN
• EXPTIME
• BAYERPAT
• BITPIX
• NAXIS1
• NAXIS2
• BSCALE

from indi-allsky.

I used a third-party tool to update and resave the FITS header details and, after this, the script has successfully executed - other than missing the binning details. However, a very quick 'UPDATE darkframe SET binmode = X WHERE filename LIKE '%binX%' has tidied all this up.

Thanks for all your help on this @aaronwmorris, really appreciate it.

Just a suggestion - for those of us with darks/bpms from earlier builds (I assume there's something not quite right with the older headers?) - would it be possible to add a script to update the reference files so future imports just run straight through without the user needing to manually check each file? Similarly, is there a way for us to export/backup our configuration?

from indi-allsky.

As for the frame types... The FITS standard only recognizes 4 frame types: Lights, Darks, Flats, and Bias. Both the indi-allsky dark and bad pixel map frames are technically a kind of dark frame and they are actually merged before being subtracted. I supposed I could have merged the result in to a master dark, but I did not. :-) Having the correct label is actually not necessary, and the import script only logs the type, but does not rely on it.

The script to import the darks needs to be as flexible as possible. There were some people that requested the ability to import darks that were not generated by indi-allsky.

I have a wiki page on backup and recovery:
https://github.com/aaronwmorris/indi-allsky/wiki/Backup-and-Recovery

I also plan on writing a script to automate performing a full backup.

If you just want to export the configuration, you can use ./config.py dump > config.json and ./config.py -c config.json load to load.

from indi-allsky.

Thanks again @aaronwmorris.

Aren't the bpms essentially bias frames? That's how I've pictured them anyway - for astrophotography I capture them in pure dark at the different gain (ISO) levels that I'm using, then combine with lights, darks, flats for my final processing - with the bias removing residual dead/hot/cold pixels, darks correcting hot/cold pixels and noise based on the exposure and the flat correcting any vignette, dust, dirt or other optical effects from the image chain, with the darks and bias frames temperature calibrated, where possible.

Absolutely loving all the changes and new features recently - but, just to flag I think there must have been a minor typo introduced in one of the latest builds as the ADU history page is no longer accessible from the direct url or from the new menu section.

from indi-allsky.

Closing this for now. Let me know if I missed something here.

from indi-allsky.