juanep97 / iop4 Goto Github PK

Sometimes a negative flux is saved in the database as aperphot result. This gives an error downstream as in:

2023-10-10 20:42:11 - iop4lib.telescopes.cahat220 [cahat220.py:343] - DEBUG - Computing relative polarimetry for <AstroSource S4 0954+65 4>.                                                                2023-10-10 20:42:11 - iop4lib.telescopes.cahat220 [cahat220.py:417] - WARNING - polarimetry_group=[ReducedFit.objects.get(id=52269), ReducedFit.objects.get(id=52272), ReducedFit.objects.get(id=52271), ReducedFit.objects.get(id=52270)]: negative flux mean encountered while relative polarimetry for astrosource=AstroSource.objects.get(name='S4 0954+65 4') ??!! It will be nan, but maybe we should look into th
is...                                                                                                                                                                                                       
Traceback (most recent call last):                                                                                                                                                                          
  File "/home/vhega/miniconda3/envs/iop4/bin/iop4", line 8, in <module>                                                                                                                                     
    sys.exit(main())                                                                                                                                                                                        
  File "/home/vhega/iop4/iop4lib/iop4.py", line 195, in main                                                                                                                                                
    process_epochs(epochs_to_process, args.force_rebuild, check_remote_list=~args.skip_remote_file_list)                                                                                                    
  File "/home/vhega/iop4/iop4lib/iop4.py", line 65, in process_epochs                                                                                                                                       
    epoch.compute_relative_polarimetry()                                                                                                                                                                    
  File "/home/vhega/iop4/iop4lib/db/epoch.py", line 674, in compute_relative_polarimetry                                                                                                                    
    return list(map(f, clusters_L))                                                                                                                                                                         
  File "/home/vhega/iop4/iop4lib/db/epoch.py", line 672, in <lambda>                                                                                                                                        
    f = lambda x: Telescope.by_name(self.telescope).compute_relative_polarimetry(x, *args, **kwargs)                                                                                                        
  File "/home/vhega/iop4/iop4lib/telescopes/cahat220.py", line 419, in compute_relative_polarimetry                                                                                                         
    mag_inst = -2.5 * math.log10(flux_mean)                                                                                                                                                                 
ValueError: math domain error                                                                                                                                                                               
(iop4) vhega@quasar:~$     

Also, this error is not catched anywhere and breaks the pipeline execution.

Ocurrences (IOP4 version main @ d9d65f2).

• During relative photo-polarimetry reduction for source "S4 0954+65 4" in CAHA-T220/2023-10-09/caf-20231010-03:36:55-sci-agui.fit.

should be iop4.py instead of iop4, but I would suggest to change the bin name to that without extension everywhere

When computing the zero point to be used in the relative photometry of a given source, all calibrators in the field are used, not only those that are calibrators to the source:

        # 3. Average the zero points

        calib_mag_zp_array = np.array([result.mag_zp or np.nan for result in photopolresult_L if result.astrosource.srctype == SRCTYPES.CALIBRATOR]) # else it fills with None also and the dtype becomes object
        calib_mag_zp_array = calib_mag_zp_array[~np.isnan(calib_mag_zp_array)]

        calib_mag_zp_array_err = np.array([result.mag_zp_err or np.nan for result in photopolresult_L if result.astrosource.srctype == SRCTYPES.CALIBRATOR])
        calib_mag_zp_array_err = calib_mag_zp_array_err[~np.isnan(calib_mag_zp_array_err)]

These lines taken from IOP3 code are not very clear.

            Theta_0 = 0
        
            if Q_I >= 0:
                Theta_0 = math.pi 
                if U_I > 0:
                    Theta_0 = -1 * math.pi
                # if Q_I < 0:
                #     Theta_0 = math.pi / 2
                
            Theta = 0.5 * math.degrees(math.atan(U_I / Q_I) + Theta_0)
            dTheta = 0.5 * 180.0 / math.pi * (1 / (1 + (U_I/Q_I) ** 2)) * math.sqrt( (dU_I/Q_I)**2 + (U_I*dQ_I/Q_I**2)**2 )

I think they can be just substituted by the same expressions that DIPOL class uses.

These functions are only used by the iop4admin and should be moved there or at least refactored out of the model, e.g. moved into the iop4lib.utils package.

Sometimes there have been errors or different conventions in source names.
We should rename other_name to other_names and allow it to be a comma separated list of names.
It will allow IOP4 to work with the different conventions used along the history of -several- programs within one catalog.

E.g. 2145+067 a.k.a B 2145+067 was wrongly named B2135 in the catalog, but was observed with that name during some time due to an error in the observers spreadsheet. We could include the wrong name in the catalog (and a indication that the name is only because of an error, to reduce the files with that name).

Do we want to enable automatic checks for format, syntax and linting?
It would require to make a lot of modifications and cleanup of the code

they are hard to follow

Still not clear to me how to set the configuration file (path to real DB, astrometry cache, etc).

I guess related it's related to knowing if these two scripts actually work

    $ python manage.py makemigrations
    $ python manage.py migrate

The first one returns No changes detected

The second one seems to run fine (Running migration messages and so on)

Then, the command python manage.py dumpdata --natural-primary --natural-foreign --format=yaml > priv.dumps.yaml returns:

CommandError: Unable to serialize database: no such table: iop4api_astrosource

Also the test test_build_reduced_multiproc takes too long, I guess because it's trying to download astrometry files.

Even if they are installed using the env yml, they should be specified in setup.py so they are installed when installing iop4 with pip

Since we are using SQLite and it has some trouble with high concurrency, using multiprocessing sometimes gives "Database is locked" errors.

In production, we could think of moving to a different database backend such as postgre. But tests should use SQLite, and this error makes the bulk reduce with multiprocessing test fail.

Add post-installation steps:

• Install js9
• Basic configuration of paths in config.yml

Explore if these steps can be done in pyproject.toml (related to #15) or have to be done in setup.py/.cfg instead.

Installed iop4 in edit mode, downloaded the tests tarball iop4testdata.tar.gz and placed it in my home directory. Then run pytest. It tries to use the config file .../iop4/config/config.yaml. Shouldn't it use instead Path('~/.iop4tests/').expanduser() / "config.tests.yaml"

In principle, there is no showstopper to run iop4 with the latest Python version 3.12 (3.13 coming already in a few months) other than some requirements above that need to be updated as well to make installation possible. I ran iop4 tests with Python 3.12 leaving dependency versions free (e.g. setting "matplotlib>=3.7.1" instead of "matplotlib==3.7.1"), and it worked.

Still to be thoroughly checked though. But something to keep in mind.

For uploading it to zenodo, you also have to do it @juanep97, linking the repository to your zenodo page.

Using iop4 -o caha_user={{other user}} -o caha_password={{other password}} does not have the intended effect because the telescope class variables are defined at import time

This warning is repeated in the github CI Actions many times during tests

 /opt/hostedtoolcache/Python/3.11.6/x64/lib/python3.11/site-packages/setuptools_scm/git.py:163: UserWarning: "/home/runner/work/iop4/iop4" is shallow and may cause errors
  warnings.warn(f'"{wd.path}" is shallow and may cause errors')
/opt/hostedtoolcache/Python/3.11.6/x64/lib/python3.11/site-packages/setuptools_scm/git.py:163: UserWarning: "/home/runner/work/iop4/iop4" is shallow and may cause errors
  warnings.warn(f'"{wd.path}" is shallow and may cause errors')
/opt/hostedtoolcache/Python/3.11.6/x64/lib/python3.11/site-packages/setuptools_scm/git.py:163: UserWarning: "/home/runner/work/iop4/iop4" is shallow and may cause errors
  warnings.warn(f'"{wd.path}" is shallow and may cause errors')

https://github.com/juanep97/iop4/actions/runs/7034575267/job/19142940434?pr=39

E.g, this is the github pages after deployment (docs.yml) of last version (v1.1.0).

Example notebooks are also not generating the images.

This is local result of make docs-sphinx:

So:

1. The version is not correctly shown
2. The examples are not working.

Reduced Fit sizes appear to be much bigger (x4?) than the corresponding rawfit.

Let's investigate why, and how to avoid this.

The config file given as an example does not work out of the box. It would be great that iop4 help and basic functionality would work out of the box after setup installation

Hi, since this software is meant to be published in a refereed publication and so on, it always looks better if it is contained in a "permanent" organization instead of a personal github repo.
I suggest that we can create the "vhega" organization and migrate libraries like this one (maybe also MUTIS) The migration should be transparent for owners/admins and so on, but it will just be contained under a different URL that will make it look more permanent.

Implement histogram equalization during astrometric calibration, it might lead to a general speed up of the process; specially for images with sources of very different brightness (such us those containing bright stars). Those kind of images currently take a longer time to calibrate, or even fail; histogram equalization might allow their calibration. Some tests will be needed to decide whether it should be the default.

https://scikit-image.org/docs/stable/auto_examples/color_exposure/plot_equalize.html

I think they still point to a previous version. I do not see the example gallery.

We could also link that docs page to the readme

E.g. Mrk 421

Many of these images do get correctly calibrated, we need to check the parameters that are working better.

Related to #6

SRCTYPES should correspond to physical source types. Whether an object is a calibrator should be defined automatically using its calibrates field. This can be implemented with a custom model manager that annotates the objects with a is_calibrator field.

From #67

We should find a way of systematically setting the values of the radii of the annulus used for photometry aperture:

During these days I saw that for some specific sources and nights, it was behaving better. In particular for CAFOS2.2 2023-12-20 1101+384, using 3.0r made the two points inconsistent with OSN observations, and made the photometry uncertainty very large. Using 6.0r made the uncertainties sensible AND compatible with the other instruments. It also didn't affect significantly other results, as you can see, it only increases slightly the discrepancy with the test reference... but that reference comes from IOP3 and is not written in stone... so until we replace it with something more solid, we can allow a bit more of discrepancy, I think.

Anyway, we should open an issue so we don't forget to look into this. These values have changed already several times in an erratic way. We should investigate it systematically. At some point, we will definitely need to implement a way to handle several cases as in #66 for the astrometry calibration parameters.

It is currently impossible to run all tests in GitHub CI Actions because a) it needs a currently private dataset (~22MB) and b) astrometry index files to test the astrometric calibration (~36GB, although it can probably be reduced to less than half of it).

Suggestions?

This will help us to identify parts of the code not tested

Since we have historical blazar monitoring data, might be a good idea to have some kind of alert trigger for sources showing exceptional brightness with respect to their usual flux. It could be used as a ToO alert system.

I think it is time to make the first IOP4 release to have a tagged version to compare later on, for bookkeeping, provenance, etc. Could be 0.1.0

Right now, RawFit.mdata, ReducedFit.mdata are properties that read and return the data from the files every time they are used. There could be two alternatives with different advantages and drawbacks:

• Make them cached properties so that they read the file only one, saving quite a bit of disk read access. This would make some functions and views slightly faster.
• Make RawFit.mdata a cached property that reads data only one. Make ReducedFit.mdata a cached property that computes the value on the fly, applying the master calibrations frames on the fly to the rawfit. This would make it the pipeline processes only slightly slower, however, for the iop4admin views using the data, such as the stats on the detailed views, it might add some seconds to the page load. Some tests are needed to check if these differences are significant (applying the calibration frames, once they are built, is quite fast).

The main advantage of this point is that disk space would be reduced by a factor of x5. Right now, since the reduced FITS are saved using float64 to keep precision (#30 ), they take x4 times the size of the raw images (uint16 usually). This method would completely eliminate these files from the disk. the .build_file would only build and save the WCS header to disk (a few kilobytes at most), and the .header would read this.

Might be interesting to use sigma clipping by default, it will make results more resilant to image defects and possibly bad calibrators.

In PR #23 .

Related discussions:

• #93 (comment)
• #30