Add MIRI reduction capabilities (intermediate) about eureka HOT 14 CLOSED

I've merged the MIRI_LRS branch into main so everyone should run git pull to make sure they're up-to-date.

from eureka.

For Determining the source position, Sarah Kendrew informed me that each MRI/LRS observation will come with a direct image (like how HST/WFC3 operates). This could be an indirect way to determine the position of the spectrum on the detector. Ultimately, it could be used as a form of validation for our own implementation.

from eureka.

That's a fun bug... or feature?!?!

from eureka.

Sebastian Zieba, Megan Mansfield, and I will be working on this in the miri_lrs branch - others are welcome to join!

from eureka.

TODO for MIRI: New stuff and things that are not instrument-agnostic in s3_reduce.py and have to be changed:

• Determining the source position:
  It looks like MIRI doesn’t include the source position ('SRCXPOS' and 'SRCYPOS') in the science header. s3_reduce.py however currently assumes it exists:
  meta.src_xpos = data.shdr['SRCXPOS'] - meta.xwindow[0]
  That has to be moved, therefore. I would propose that s3_reduce.py should call “inst” here (like miri.py) and check if it is in the header. If not, call a new python file (like source_pos.py) and determine the position (by for example determining the brightest row in the spatial direction or fitting a gaussian).

• Check axis:
  MIRI seems to be rotated by 90° compared to NIRCam. There’s a “Dispersion Direction” note in the header. I would suggest inst (like miri.py) should check this value (it is axis=2 for MIRI and axis=1 for NIRCam) when the data is read in. If axis=2, transpose the data so that it works with the current setup of s3_reduce.py.

• Unit conversion:
  currently: s3_reduce.py -> inst (like nircam.py) -> bright2flux.py -> bright2dn. But: data.mhdr['PUPIL'] doesn't exist in MIRI. Todo: Make that instrument-specific and make it work for miri. (Not really doable right now because the MIRI stage 2 calints files don't include wavelength information and you need that in order to correct for the wavelength-dependent response function).

• Other things which are not necessarily MIRI specific (but Megan, Taylor, and I discussed this during our meeting):

  • Check non-zero Data Quality entries and mask them. Give these pixels flux values back by interpolating in space or time. Could be done around here: data.submask = util.check_nans(data.subdata, data.submask, log) (this part could be better too. Just something like: data.submask = util.check_nans(data, log) instead. util.check_nans is currently called three times after another in s3_reduce.py). It might be nice imo, to have a diagnostic DQ plot too, showing the data quality values. So, plot the image and color in pixels with different colors depending on the DQ value.

(I think that's everything we talked about)

from eureka.

Cool! I wrote just a quick first pass on finding the source position based on just locating the brightest row. I can do something fancier like fitting a Gaussian but figured I'd start small. However, I'm having some problems trying to run the code to test it, and the problems seem to not even be on the files I've edited. I might just be making a dumb mistake somewhere, or having some other problem unique to my machine, but could someone else try testing the MIRI stuff? I'm about to push changes, including the new source_pos.py file.
Also, I'm pretty new to writing code for other people to use, so Sebastian I just did my best to try and follow the structure you suggested above. But feel free to restructure anything!

from eureka.

A flux-weighted mean centering method might be a good option to explore too (I've added one to Megan's file). With HST/STIS, the spectra could end up sloped with respect to the expected dispersion axis, so that's potentially something we may need to worry about for future iterations.

I was able to successfully run the NIRCam code, but for MIRI I get many RuntimeWarning: invalid value encountered in true_divide error messages from line 360 of optspex.py, so perhaps MIRI data need more tweaks before they will work with this function. Alternatively, this may be due to the lack of conversion from MJy/sr -> e- at the moment.

from eureka.

That's a new error to me. I'm guessing there might be a NaN or INF lurking in the median aperture array. That should be easy enough to check.

from eureka.

That's great, thanks!

from eureka.

@kevin218, what subarray region did you end up using for MIRI Stage 3? I'm going through and testing whether my Stage 2 outputs are compatible with Stage 3+, and the changes to Stage 3 have broken the old subarray region I used to use in my local ecf file. I remembered that you'd mentioned on Slack that you had spent some time trying to figure out some math for the subarray region after having mirrored the wavelength axis

from eureka.

@taylorbell57 See attached ECF file for MIRI
S3_wasp43b.ecf.zip

from eureka.

Ahh, my ecf was fine - it was that my MIRI simulated data had been broken down into multiple segments and the code you added kept switching the x and y windows back and forth every time a file was read in which broke the code after the first file ran through and the code started on the second file. I've patched this locally and will be making some pull requests soon related to this and my work on S2

from eureka.

@kevin218 I would propose we close this issue as NIRCam, NIRSpec, and MIRI are now equally supported. Thoughts?

from eureka.

Agreed!

from eureka.