juanep97 / iop4 Goto Github PK
View Code? Open in Web Editor NEWIOP4, the Interactive Optical Photo-Polarimetric Python Pipeline.
Home Page: https://juanep97.github.io/iop4/
License: BSD 3-Clause "New" or "Revised" License
IOP4, the Interactive Optical Photo-Polarimetric Python Pipeline.
Home Page: https://juanep97.github.io/iop4/
License: BSD 3-Clause "New" or "Revised" License
Sometimes a negative flux is saved in the database as aperphot result. This gives an error downstream as in:
2023-10-10 20:42:11 - iop4lib.telescopes.cahat220 [cahat220.py:343] - DEBUG - Computing relative polarimetry for <AstroSource S4 0954+65 4>. 2023-10-10 20:42:11 - iop4lib.telescopes.cahat220 [cahat220.py:417] - WARNING - polarimetry_group=[ReducedFit.objects.get(id=52269), ReducedFit.objects.get(id=52272), ReducedFit.objects.get(id=52271), ReducedFit.objects.get(id=52270)]: negative flux mean encountered while relative polarimetry for astrosource=AstroSource.objects.get(name='S4 0954+65 4') ??!! It will be nan, but maybe we should look into th
is...
Traceback (most recent call last):
File "/home/vhega/miniconda3/envs/iop4/bin/iop4", line 8, in <module>
sys.exit(main())
File "/home/vhega/iop4/iop4lib/iop4.py", line 195, in main
process_epochs(epochs_to_process, args.force_rebuild, check_remote_list=~args.skip_remote_file_list)
File "/home/vhega/iop4/iop4lib/iop4.py", line 65, in process_epochs
epoch.compute_relative_polarimetry()
File "/home/vhega/iop4/iop4lib/db/epoch.py", line 674, in compute_relative_polarimetry
return list(map(f, clusters_L))
File "/home/vhega/iop4/iop4lib/db/epoch.py", line 672, in <lambda>
f = lambda x: Telescope.by_name(self.telescope).compute_relative_polarimetry(x, *args, **kwargs)
File "/home/vhega/iop4/iop4lib/telescopes/cahat220.py", line 419, in compute_relative_polarimetry
mag_inst = -2.5 * math.log10(flux_mean)
ValueError: math domain error
(iop4) vhega@quasar:~$
Also, this error is not catched anywhere and breaks the pipeline execution.
Ocurrences (IOP4 version main @ d9d65f2).
CAHA-T220/2023-10-09/caf-20231010-03:36:55-sci-agui.fit
.Line 41 in ee4ce30
should be iop4.py
instead of iop4
, but I would suggest to change the bin name to that without extension everywhere
When computing the zero point to be used in the relative photometry of a given source, all calibrators in the field are used, not only those that are calibrators to the source:
iop4/iop4lib/instruments/instrument.py
Line 595 in d8dfafa
# 3. Average the zero points
calib_mag_zp_array = np.array([result.mag_zp or np.nan for result in photopolresult_L if result.astrosource.srctype == SRCTYPES.CALIBRATOR]) # else it fills with None also and the dtype becomes object
calib_mag_zp_array = calib_mag_zp_array[~np.isnan(calib_mag_zp_array)]
calib_mag_zp_array_err = np.array([result.mag_zp_err or np.nan for result in photopolresult_L if result.astrosource.srctype == SRCTYPES.CALIBRATOR])
calib_mag_zp_array_err = calib_mag_zp_array_err[~np.isnan(calib_mag_zp_array_err)]
These lines taken from IOP3 code are not very clear.
Theta_0 = 0
if Q_I >= 0:
Theta_0 = math.pi
if U_I > 0:
Theta_0 = -1 * math.pi
# if Q_I < 0:
# Theta_0 = math.pi / 2
Theta = 0.5 * math.degrees(math.atan(U_I / Q_I) + Theta_0)
dTheta = 0.5 * 180.0 / math.pi * (1 / (1 + (U_I/Q_I) ** 2)) * math.sqrt( (dU_I/Q_I)**2 + (U_I*dQ_I/Q_I**2)**2 )
I think they can be just substituted by the same expressions that DIPOL class uses.
These functions are only used by the iop4admin and should be moved there or at least refactored out of the model, e.g. moved into the iop4lib.utils package.
Sometimes there have been errors or different conventions in source names.
We should rename other_name to other_names and allow it to be a comma separated list of names.
It will allow IOP4 to work with the different conventions used along the history of -several- programs within one catalog.
E.g. 2145+067 a.k.a B 2145+067 was wrongly named B2135 in the catalog, but was observed with that name during some time due to an error in the observers spreadsheet. We could include the wrong name in the catalog (and a indication that the name is only because of an error, to reduce the files with that name).
Do we want to enable automatic checks for format, syntax and linting?
It would require to make a lot of modifications and cleanup of the code
Line 21 in ee4ce30
they are hard to follow
Still not clear to me how to set the configuration file (path to real DB, astrometry cache, etc).
I guess related it's related to knowing if these two scripts actually work
$ python manage.py makemigrations
$ python manage.py migrate
The first one returns No changes detected
The second one seems to run fine (Running migration messages and so on)
Then, the command python manage.py dumpdata --natural-primary --natural-foreign --format=yaml > priv.dumps.yaml
returns:
CommandError: Unable to serialize database: no such table: iop4api_astrosource
Also the test test_build_reduced_multiproc
takes too long, I guess because it's trying to download astrometry files.
Even if they are installed using the env yml, they should be specified in setup.py
so they are installed when installing iop4 with pip
Since we are using SQLite and it has some trouble with high concurrency, using multiprocessing sometimes gives "Database is locked" errors.
In production, we could think of moving to a different database backend such as postgre. But tests should use SQLite, and this error makes the bulk reduce with multiprocessing test fail.
Add post-installation steps:
config.yml
Explore if these steps can be done in pyproject.toml
(related to #15) or have to be done in setup.py/.cfg
instead.
Installed iop4 in edit mode, downloaded the tests tarball iop4testdata.tar.gz
and placed it in my home directory. Then run pytest. It tries to use the config file .../iop4/config/config.yaml
. Shouldn't it use instead Path('~/.iop4tests/').expanduser() / "config.tests.yaml"
Lines 20 to 40 in 090cc97
In principle, there is no showstopper to run iop4 with the latest Python version 3.12 (3.13 coming already in a few months) other than some requirements above that need to be updated as well to make installation possible. I ran iop4 tests with Python 3.12 leaving dependency versions free (e.g. setting "matplotlib>=3.7.1"
instead of "matplotlib==3.7.1"
), and it worked.
Still to be thoroughly checked though. But something to keep in mind.
For uploading it to zenodo, you also have to do it @juanep97, linking the repository to your zenodo page.
Using iop4 -o caha_user={{other user}} -o caha_password={{other password}}
does not have the intended effect because the telescope class variables are defined at import time
iop4/iop4lib/telescopes/cahat220.py
Line 54 in 75a1db1
This warning is repeated in the github CI Actions many times during tests
/opt/hostedtoolcache/Python/3.11.6/x64/lib/python3.11/site-packages/setuptools_scm/git.py:163: UserWarning: "/home/runner/work/iop4/iop4" is shallow and may cause errors
warnings.warn(f'"{wd.path}" is shallow and may cause errors')
/opt/hostedtoolcache/Python/3.11.6/x64/lib/python3.11/site-packages/setuptools_scm/git.py:163: UserWarning: "/home/runner/work/iop4/iop4" is shallow and may cause errors
warnings.warn(f'"{wd.path}" is shallow and may cause errors')
/opt/hostedtoolcache/Python/3.11.6/x64/lib/python3.11/site-packages/setuptools_scm/git.py:163: UserWarning: "/home/runner/work/iop4/iop4" is shallow and may cause errors
warnings.warn(f'"{wd.path}" is shallow and may cause errors')
https://github.com/juanep97/iop4/actions/runs/7034575267/job/19142940434?pr=39
Reduced Fit sizes appear to be much bigger (x4?) than the corresponding rawfit.
Let's investigate why, and how to avoid this.
The config file given as an example does not work out of the box. It would be great that iop4 help and basic functionality would work out of the box after setup installation
Hi, since this software is meant to be published in a refereed publication and so on, it always looks better if it is contained in a "permanent" organization instead of a personal github repo.
I suggest that we can create the "vhega" organization and migrate libraries like this one (maybe also MUTIS) The migration should be transparent for owners/admins and so on, but it will just be contained under a different URL that will make it look more permanent.
Implement histogram equalization during astrometric calibration, it might lead to a general speed up of the process; specially for images with sources of very different brightness (such us those containing bright stars). Those kind of images currently take a longer time to calibrate, or even fail; histogram equalization might allow their calibration. Some tests will be needed to decide whether it should be the default.
https://scikit-image.org/docs/stable/auto_examples/color_exposure/plot_equalize.html
I think they still point to a previous version. I do not see the example gallery.
We could also link that docs page to the readme
E.g. Mrk 421
Many of these images do get correctly calibrated, we need to check the parameters that are working better.
Related to #6
SRCTYPES should correspond to physical source types. Whether an object is a calibrator should be defined automatically using its calibrates
field. This can be implemented with a custom model manager that annotates the objects with a is_calibrator
field.
From #67
We should find a way of systematically setting the values of the radii of the annulus used for photometry aperture:
During these days I saw that for some specific sources and nights, it was behaving better. In particular for CAFOS2.2 2023-12-20 1101+384, using 3.0r made the two points inconsistent with OSN observations, and made the photometry uncertainty very large. Using 6.0r made the uncertainties sensible AND compatible with the other instruments. It also didn't affect significantly other results, as you can see, it only increases slightly the discrepancy with the test reference... but that reference comes from IOP3 and is not written in stone... so until we replace it with something more solid, we can allow a bit more of discrepancy, I think.
Anyway, we should open an issue so we don't forget to look into this. These values have changed already several times in an erratic way. We should investigate it systematically. At some point, we will definitely need to implement a way to handle several cases as in #66 for the astrometry calibration parameters.
It is currently impossible to run all tests in GitHub CI Actions because a) it needs a currently private dataset (~22MB) and b) astrometry index files to test the astrometric calibration (~36GB, although it can probably be reduced to less than half of it).
Suggestions?
This will help us to identify parts of the code not tested
Since we have historical blazar monitoring data, might be a good idea to have some kind of alert trigger for sources showing exceptional brightness with respect to their usual flux. It could be used as a ToO alert system.
I think it is time to make the first IOP4 release to have a tagged version to compare later on, for bookkeeping, provenance, etc. Could be 0.1.0
Right now, RawFit.mdata, ReducedFit.mdata are properties that read and return the data from the files every time they are used. There could be two alternatives with different advantages and drawbacks:
The main advantage of this point is that disk space would be reduced by a factor of x5. Right now, since the reduced FITS are saved using float64 to keep precision (#30 ), they take x4 times the size of the raw images (uint16 usually). This method would completely eliminate these files from the disk. the .build_file would only build and save the WCS header to disk (a few kilobytes at most), and the .header would read this.
iop4/iop4lib/instruments/instrument.py
Line 595 in d8dfafa
Might be interesting to use sigma clipping by default, it will make results more resilant to image defects and possibly bad calibrators.
In PR #23 .
Related discussions:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.