kevin218 / eureka Goto Github PK

I think it would be useful to have some kind of flow chart or visual guide to all of the moving pieces in eureka, because there are a lot! It doesn't have to be super fancy, just something that outlines what's done in S1, S2, S3, and what tools are available in S4 and S5.

When testing the new jwst v1.3.3 I noticed that the background subtraction step in S3 (here) was requiring 6 seconds per iteration with 14 cpu threads, but when I tried to run it with only 1 cpu thread it only took 2 seconds per iteration. I can imagine the runtime not scaling linearly with the cpu threads, but running slower seems very unintuitive and is obviously a bad sign. We'll need to see if this is related to my other multithreading issue (#92) or if the there is a larger issue with multithreading here.

When I ran
pytest

to test my Eureka! installation, I got the following error message:

AttributeError: module 'eureka.S5_lightcurve_fitting.models' has no attribute 'ExponentialModel'

(out of the 10 tests, it is the only one that failed).

jwst version 1.1.0 was previously required by the MIRISim version used to create the simulations provided by the ERS simulated data team, but current MIRISim version 2.4.1 is compatible with the jwst pipeline version 1.3.3. This will potentially be important for the March 2022 data challenge which will include new simulations which will most likely make use of the newest MIRISim version. Updating jwst to a newer version will have non-negligible impacts on other instruments and code in stages 1 and 2 though, so this will need to be a carefully planned transition when the time comes.

Whenever Eureka is imported into a file, attempting to call plt.show() for a matplotlib.pyplot plot results in the error copied below.

If Eureka is simply not imported, but the same anaconda environment is used, no such error appears. It's important to note that you don't have to actually use Eureka / specify any latex for this to appear. Simply importing the package whilst trying to call plt.show() results in this error.

Temporary workaround is to add the following lines at some point in the code prior to any calls to plt.show()

from matplotlib import rc
rc('text', usetex=False)

However, this isn't the best long term solution. Current thought is that the error is being raised by the use of latex in label text within the Eureka package, and whether 'r' should be placed before the label text itself. I expect we need to discuss whether the 'r' is important or not, personally I've never had to include it and any latex commands (e.g. $\mu$) have been interpreted correctly by matplotlib.

Final note, it seems this issue isn't unique to me, and at least 3 other people experienced this during the ERS hackathon.

Mac OS Catalina 10.15.7
Python 3.9.5 Anaconda Environment

Traceback (most recent call last):
  File "/Users/acarter/anaconda3/envs/eureka/lib/python3.9/site-packages/matplotlib/texmanager.py", line 252, in _run_checked_subprocess
    report = subprocess.check_output(
  File "/Users/acarter/anaconda3/envs/eureka/lib/python3.9/subprocess.py", line 424, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/Users/acarter/anaconda3/envs/eureka/lib/python3.9/subprocess.py", line 505, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/Users/acarter/anaconda3/envs/eureka/lib/python3.9/subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/Users/acarter/anaconda3/envs/eureka/lib/python3.9/subprocess.py", line 1821, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'latex'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/acarter/anaconda3/envs/eureka/lib/python3.9/site-packages/matplotlib/backends/backend_macosx.py", line 45, in _draw
    self.figure.draw(renderer)
  File "/Users/acarter/anaconda3/envs/eureka/lib/python3.9/site-packages/matplotlib/artist.py", line 74, in draw_wrapper
    result = draw(artist, renderer, *args, **kwargs)
  File "/Users/acarter/anaconda3/envs/eureka/lib/python3.9/site-packages/matplotlib/artist.py", line 51, in draw_wrapper
    return draw(artist, renderer, *args, **kwargs)
  File "/Users/acarter/anaconda3/envs/eureka/lib/python3.9/site-packages/matplotlib/figure.py", line 2780, in draw
    mimage._draw_list_compositing_images(
  File "/Users/acarter/anaconda3/envs/eureka/lib/python3.9/site-packages/matplotlib/image.py", line 132, in _draw_list_compositing_images
    a.draw(renderer)
  File "/Users/acarter/anaconda3/envs/eureka/lib/python3.9/site-packages/matplotlib/artist.py", line 51, in draw_wrapper
    return draw(artist, renderer, *args, **kwargs)
  File "/Users/acarter/anaconda3/envs/eureka/lib/python3.9/site-packages/matplotlib/_api/deprecation.py", line 431, in wrapper
    return func(*inner_args, **inner_kwargs)
  File "/Users/acarter/anaconda3/envs/eureka/lib/python3.9/site-packages/matplotlib/axes/_base.py", line 2921, in draw
    mimage._draw_list_compositing_images(renderer, self, artists)
  File "/Users/acarter/anaconda3/envs/eureka/lib/python3.9/site-packages/matplotlib/image.py", line 132, in _draw_list_compositing_images
    a.draw(renderer)
  File "/Users/acarter/anaconda3/envs/eureka/lib/python3.9/site-packages/matplotlib/artist.py", line 51, in draw_wrapper
    return draw(artist, renderer, *args, **kwargs)
  File "/Users/acarter/anaconda3/envs/eureka/lib/python3.9/site-packages/matplotlib/axis.py", line 1137, in draw
    ticklabelBoxes, ticklabelBoxes2 = self._get_tick_bboxes(ticks_to_draw,
  File "/Users/acarter/anaconda3/envs/eureka/lib/python3.9/site-packages/matplotlib/axis.py", line 1063, in _get_tick_bboxes
    return ([tick.label1.get_window_extent(renderer)
  File "/Users/acarter/anaconda3/envs/eureka/lib/python3.9/site-packages/matplotlib/axis.py", line 1063, in <listcomp>
    return ([tick.label1.get_window_extent(renderer)
  File "/Users/acarter/anaconda3/envs/eureka/lib/python3.9/site-packages/matplotlib/text.py", line 903, in get_window_extent
    bbox, info, descent = self._get_layout(self._renderer)
  File "/Users/acarter/anaconda3/envs/eureka/lib/python3.9/site-packages/matplotlib/text.py", line 306, in _get_layout
    _, lp_h, lp_d = renderer.get_text_width_height_descent(
  File "/Users/acarter/anaconda3/envs/eureka/lib/python3.9/site-packages/matplotlib/backends/backend_agg.py", line 229, in get_text_width_height_descent
    w, h, d = texmanager.get_text_width_height_descent(
  File "/Users/acarter/anaconda3/envs/eureka/lib/python3.9/site-packages/matplotlib/texmanager.py", line 399, in get_text_width_height_descent
    dvifile = self.make_dvi(tex, fontsize)
  File "/Users/acarter/anaconda3/envs/eureka/lib/python3.9/site-packages/matplotlib/texmanager.py", line 291, in make_dvi
    self._run_checked_subprocess(
  File "/Users/acarter/anaconda3/envs/eureka/lib/python3.9/site-packages/matplotlib/texmanager.py", line 256, in _run_checked_subprocess
    raise RuntimeError(
RuntimeError: Failed to process string with tex because latex could not be found

Before we start a Stage 6 (Issue #98), we need to save all the detailed outputs from S5 which we currently don't do. One issue is that Parameter objects aren't pickleable (as will be many other things), so we'll need to work out how to cleanly save the outputs

need to implement relative paths so python doesn't look in the wrong directory for scripts. For example, I have models.py in my POET directory and the lightcurve_fitting directory for eureka. I implemented a hacky fix where eureka looks in the existing directory first, but that only works if you want to run the code from inside the demos directory.

During my recent attempt at analyzing the new MIRI/LRS phase curve simulations (starting from uncal files) I got the following error message on the last file:

Starting file 43 of 43
Running the Spec2Pipeline

2022-02-17 23:17:07,408 - stpipe.EurekaSpec2Pipeline - INFO - Step EurekaSpec2Pipeline running with args ('/Volumes/DataDrive/MIRI_ERS_DC1/Stage1/S1_2022-02-17_miri_lrs_run1/ERS_NGTS10_2022_seg_042_rateints.fits',).
2022-02-17 23:17:07,431 - stpipe.EurekaSpec2Pipeline - INFO - Step EurekaSpec2Pipeline parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': '/Volumes/DataDrive/MIRI_ERS_DC1/Stage1/S1_2022-02-17_miri_lrs_run1/ERS_NGTS10_2022_seg_041', 'output_dir': '/Volumes/DataDrive/MIRI_ERS_DC1/Stage2/S2_2022-02-17_miri_lrs_run1/', 'output_ext': '.fits', 'output_use_model': True, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '/Volumes/DataDrive/MIRI_ERS_DC1/Stage1/S1_2022-02-17_miri_lrs_run1', 'save_bsub': False, 'fail_on_exception': True, 'steps': {'bkg_subtract': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': 'bkg_subtract', 'search_output_file': True, 'input_dir': '/Volumes/DataDrive/MIRI_ERS_DC1/Stage1/S1_2022-02-17_miri_lrs_run1', 'save_combined_background': False, 'sigma': 3.0, 'maxiters': None}, 'assign_wcs': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': 'assign_wcs', 'search_output_file': True, 'input_dir': '/Volumes/DataDrive/MIRI_ERS_DC1/Stage1/S1_2022-02-17_miri_lrs_run1', 'sip_approx': True, 'sip_max_pix_error': 0.25, 'sip_degree': None, 'sip_max_inv_pix_error': 0.25, 'sip_inv_degree': None, 'sip_npoints': 32, 'slit_y_low': -0.55, 'slit_y_high': 0.55}, 'imprint_subtract': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': 'imprint_subtract', 'search_output_file': True, 'input_dir': '/Volumes/DataDrive/MIRI_ERS_DC1/Stage1/S1_2022-02-17_miri_lrs_run1'}, 'msa_flagging': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': 'msa_flagging', 'search_output_file': True, 'input_dir': '/Volumes/DataDrive/MIRI_ERS_DC1/Stage1/S1_2022-02-17_miri_lrs_run1'}, 'extract_2d': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '/Volumes/DataDrive/MIRI_ERS_DC1/Stage1/S1_2022-02-17_miri_lrs_run1', 'slit_name': None, 'extract_orders': None, 'tsgrism_extract_height': None, 'wfss_extract_half_height': 5, 'grism_objects': None, 'mmag_extract': 99.0}, 'master_background': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': True, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '/Volumes/DataDrive/MIRI_ERS_DC1/Stage1/S1_2022-02-17_miri_lrs_run1', 'force_subtract': False, 'save_background': False, 'user_background': None, 'inverse': False, 'steps': {'flat_field': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '/Volumes/DataDrive/MIRI_ERS_DC1/Stage1/S1_2022-02-17_miri_lrs_run1', 'save_interpolated_flat': False, 'user_supplied_flat': None, 'inverse': False}, 'pathloss': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '/Volumes/DataDrive/MIRI_ERS_DC1/Stage1/S1_2022-02-17_miri_lrs_run1', 'inverse': False, 'source_type': None}, 'barshadow': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '/Volumes/DataDrive/MIRI_ERS_DC1/Stage1/S1_2022-02-17_miri_lrs_run1', 'inverse': False, 'source_type': None}, 'photom': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '/Volumes/DataDrive/MIRI_ERS_DC1/Stage1/S1_2022-02-17_miri_lrs_run1', 'inverse': False, 'source_type': None}}}, 'wavecorr': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '/Volumes/DataDrive/MIRI_ERS_DC1/Stage1/S1_2022-02-17_miri_lrs_run1'}, 'flat_field': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': 'flat_field', 'search_output_file': True, 'input_dir': '/Volumes/DataDrive/MIRI_ERS_DC1/Stage1/S1_2022-02-17_miri_lrs_run1', 'save_interpolated_flat': False, 'user_supplied_flat': None, 'inverse': False}, 'srctype': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': 'srctype', 'search_output_file': True, 'input_dir': '/Volumes/DataDrive/MIRI_ERS_DC1/Stage1/S1_2022-02-17_miri_lrs_run1'}, 'straylight': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': 'straylight', 'search_output_file': True, 'input_dir': '/Volumes/DataDrive/MIRI_ERS_DC1/Stage1/S1_2022-02-17_miri_lrs_run1', 'method': 'ModShepard', 'roi': 50, 'power': 1.0}, 'fringe': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': 'fringe', 'search_output_file': True, 'input_dir': '/Volumes/DataDrive/MIRI_ERS_DC1/Stage1/S1_2022-02-17_miri_lrs_run1'}, 'pathloss': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': 'pathloss', 'search_output_file': True, 'input_dir': '/Volumes/DataDrive/MIRI_ERS_DC1/Stage1/S1_2022-02-17_miri_lrs_run1', 'inverse': False, 'source_type': None}, 'barshadow': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': 'barshadow', 'search_output_file': True, 'input_dir': '/Volumes/DataDrive/MIRI_ERS_DC1/Stage1/S1_2022-02-17_miri_lrs_run1', 'inverse': False, 'source_type': None}, 'photom': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': 'photom', 'search_output_file': True, 'input_dir': '/Volumes/DataDrive/MIRI_ERS_DC1/Stage1/S1_2022-02-17_miri_lrs_run1', 'inverse': False, 'source_type': None}, 'resample_spec': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': True, 'suffix': 's2d', 'search_output_file': True, 'input_dir': '/Volumes/DataDrive/MIRI_ERS_DC1/Stage1/S1_2022-02-17_miri_lrs_run1', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'weight_type': 'ivm', 'pixel_scale_ratio': 1.0, 'single': False, 'blendheaders': True, 'allowed_memory': None}, 'cube_build': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': True, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': False, 'input_dir': '/Volumes/DataDrive/MIRI_ERS_DC1/Stage1/S1_2022-02-17_miri_lrs_run1', 'channel': 'all', 'band': 'all', 'grating': 'all', 'filter': 'all', 'output_type': 'multi', 'scale1': 0.0, 'scale2': 0.0, 'scalew': 0.0, 'weighting': 'emsm', 'coord_system': 'skyalign', 'rois': 0.0, 'roiw': 0.0, 'weight_power': 2.0, 'wavemin': None, 'wavemax': None, 'single': False, 'xdebug': None, 'ydebug': None, 'zdebug': None, 'skip_dqflagging': True}, 'extract_1d': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': 'x1dints', 'search_output_file': True, 'input_dir': '/Volumes/DataDrive/MIRI_ERS_DC1/Stage1/S1_2022-02-17_miri_lrs_run1', 'smoothing_length': None, 'bkg_fit': 'poly', 'bkg_order': None, 'bkg_sigma_clip': 3.0, 'log_increment': 50, 'subtract_background': None, 'use_source_posn': None, 'apply_apcorr': True}}}
2022-02-17 23:17:07,493 - stpipe.EurekaSpec2Pipeline - INFO - Prefetching reference files for dataset: 'ERS_NGTS10_2022_seg_042_rateints.fits' reftypes = ['apcorr', 'camera', 'collimator', 'dflat', 'disperser', 'distortion', 'extract1d', 'fflat', 'filteroffset', 'flat', 'fore', 'fpa', 'ifufore', 'ifupost', 'ifuslicer', 'msa', 'ote', 'regions', 'sflat', 'specwcs', 'wavelengthrange']
2022-02-17 23:17:07,495 - stpipe.EurekaSpec2Pipeline - INFO - Prefetch for APCORR reference file is '/Users/tjbell1/crds_cache/references/jwst/miri/jwst_miri_apcorr_0007.fits'.
2022-02-17 23:17:07,495 - stpipe.EurekaSpec2Pipeline - INFO - Prefetch for CAMERA reference file is 'N/A'.
2022-02-17 23:17:07,495 - stpipe.EurekaSpec2Pipeline - INFO - Prefetch for COLLIMATOR reference file is 'N/A'.
2022-02-17 23:17:07,495 - stpipe.EurekaSpec2Pipeline - INFO - Prefetch for DFLAT reference file is 'N/A'.
2022-02-17 23:17:07,496 - stpipe.EurekaSpec2Pipeline - INFO - Prefetch for DISPERSER reference file is 'N/A'.
2022-02-17 23:17:07,496 - stpipe.EurekaSpec2Pipeline - INFO - Prefetch for DISTORTION reference file is '/Users/tjbell1/crds_cache/references/jwst/miri/jwst_miri_distortion_0028.asdf'.
2022-02-17 23:17:07,496 - stpipe.EurekaSpec2Pipeline - INFO - Prefetch for EXTRACT1D reference file is '/Users/tjbell1/crds_cache/references/jwst/miri/jwst_miri_extract1d_0004.json'.
2022-02-17 23:17:07,496 - stpipe.EurekaSpec2Pipeline - INFO - Prefetch for FFLAT reference file is 'N/A'.
2022-02-17 23:17:07,496 - stpipe.EurekaSpec2Pipeline - INFO - Prefetch for FILTEROFFSET reference file is 'N/A'.
2022-02-17 23:17:07,496 - stpipe.EurekaSpec2Pipeline - INFO - Prefetch for FLAT reference file is '/Users/tjbell1/crds_cache/references/jwst/miri/jwst_miri_flat_0745.fits'.
2022-02-17 23:17:07,496 - stpipe.EurekaSpec2Pipeline - INFO - Prefetch for FORE reference file is 'N/A'.
2022-02-17 23:17:07,496 - stpipe.EurekaSpec2Pipeline - INFO - Prefetch for FPA reference file is 'N/A'.
2022-02-17 23:17:07,496 - stpipe.EurekaSpec2Pipeline - INFO - Prefetch for IFUFORE reference file is 'N/A'.
2022-02-17 23:17:07,496 - stpipe.EurekaSpec2Pipeline - INFO - Prefetch for IFUPOST reference file is 'N/A'.
2022-02-17 23:17:07,496 - stpipe.EurekaSpec2Pipeline - INFO - Prefetch for IFUSLICER reference file is 'N/A'.
2022-02-17 23:17:07,496 - stpipe.EurekaSpec2Pipeline - INFO - Prefetch for MSA reference file is 'N/A'.
2022-02-17 23:17:07,496 - stpipe.EurekaSpec2Pipeline - INFO - Prefetch for OTE reference file is 'N/A'.
2022-02-17 23:17:07,497 - stpipe.EurekaSpec2Pipeline - INFO - Prefetch for REGIONS reference file is 'N/A'.
2022-02-17 23:17:07,497 - stpipe.EurekaSpec2Pipeline - INFO - Prefetch for SFLAT reference file is 'N/A'.
2022-02-17 23:17:07,497 - stpipe.EurekaSpec2Pipeline - INFO - Prefetch for SPECWCS reference file is '/Users/tjbell1/crds_cache/references/jwst/miri/jwst_miri_specwcs_0003.fits'.
2022-02-17 23:17:07,497 - stpipe.EurekaSpec2Pipeline - INFO - Prefetch for WAVELENGTHRANGE reference file is 'N/A'.
2022-02-17 23:17:07,497 - stpipe.EurekaSpec2Pipeline - INFO - Starting calwebb_spec2 ...
2022-02-17 23:17:07,497 - stpipe.EurekaSpec2Pipeline - INFO - Processing product /Volumes/DataDrive/MIRI_ERS_DC1/Stage1/S1_2022-02-17_miri_lrs_run1/ERS_NGTS10_2022_seg_042
2022-02-17 23:17:07,497 - stpipe.EurekaSpec2Pipeline - INFO - Working on input /Volumes/DataDrive/MIRI_ERS_DC1/Stage1/S1_2022-02-17_miri_lrs_run1/ERS_NGTS10_2022_seg_042_rateints.fits ...
2022-02-17 23:17:07,701 - stpipe.EurekaSpec2Pipeline.assign_wcs - INFO - Step assign_wcs running with args (<CubeModel(1, 416, 72) from ERS_NGTS10_2022_seg_042_rateints.fits>,).
2022-02-17 23:17:07,703 - stpipe.EurekaSpec2Pipeline.assign_wcs - INFO - Step assign_wcs parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': 'assign_wcs', 'search_output_file': True, 'input_dir': '/Volumes/DataDrive/MIRI_ERS_DC1/Stage1/S1_2022-02-17_miri_lrs_run1', 'sip_approx': True, 'sip_max_pix_error': 0.25, 'sip_degree': None, 'sip_max_inv_pix_error': 0.25, 'sip_inv_degree': None, 'sip_npoints': 32, 'slit_y_low': -0.55, 'slit_y_high': 0.55}
2022-02-17 23:17:08,047 - stpipe.EurekaSpec2Pipeline.assign_wcs - INFO - Created a MIRI mir_lrs-slitless pipeline with references {'distortion': '/Users/tjbell1/crds_cache/references/jwst/miri/jwst_miri_distortion_0028.asdf', 'filteroffset': None, 'specwcs': '/Users/tjbell1/crds_cache/references/jwst/miri/jwst_miri_specwcs_0003.fits', 'regions': None, 'wavelengthrange': None, 'camera': None, 'collimator': None, 'disperser': None, 'fore': None, 'fpa': None, 'msa': None, 'ote': None, 'ifupost': None, 'ifufore': None, 'ifuslicer': None}
2022-02-17 23:17:08,116 - stpipe.EurekaSpec2Pipeline.assign_wcs - INFO - Update S_REGION to POLYGON ICRS 0.000981120 -0.000094311 0.000981120 0.000098852 359.998963682 0.000098852 359.998963682 -0.000094311
2022-02-17 23:17:08,116 - stpipe.EurekaSpec2Pipeline.assign_wcs - INFO - assign_wcs updated S_REGION to POLYGON ICRS 0.000981120 -0.000094311 0.000981120 0.000098852 359.998963682 0.000098852 359.998963682 -0.000094311
2022-02-17 23:17:08,116 - stpipe.EurekaSpec2Pipeline.assign_wcs - INFO - COMPLETED assign_wcs
2022-02-17 23:17:08,129 - stpipe.EurekaSpec2Pipeline.assign_wcs - INFO - Step assign_wcs done
2022-02-17 23:17:08,299 - stpipe.EurekaSpec2Pipeline.bkg_subtract - INFO - Step bkg_subtract running with args (<CubeModel(1, 416, 72) from ERS_NGTS10_2022_seg_042_rateints.fits>, []).
2022-02-17 23:17:08,301 - stpipe.EurekaSpec2Pipeline.bkg_subtract - INFO - Step bkg_subtract parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': 'bkg_subtract', 'search_output_file': True, 'input_dir': '/Volumes/DataDrive/MIRI_ERS_DC1/Stage1/S1_2022-02-17_miri_lrs_run1', 'save_combined_background': False, 'sigma': 3.0, 'maxiters': None}
2022-02-17 23:17:08,301 - stpipe.EurekaSpec2Pipeline.bkg_subtract - INFO - Step skipped.
2022-02-17 23:17:08,304 - stpipe.EurekaSpec2Pipeline.bkg_subtract - INFO - Step bkg_subtract done
2022-02-17 23:17:08,484 - stpipe.EurekaSpec2Pipeline.imprint_subtract - INFO - Step imprint_subtract running with args (<CubeModel(1, 416, 72) from ERS_NGTS10_2022_seg_042_rateints.fits>, []).
2022-02-17 23:17:08,486 - stpipe.EurekaSpec2Pipeline.imprint_subtract - INFO - Step imprint_subtract parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': 'imprint_subtract', 'search_output_file': True, 'input_dir': '/Volumes/DataDrive/MIRI_ERS_DC1/Stage1/S1_2022-02-17_miri_lrs_run1'}
2022-02-17 23:17:08,486 - stpipe.EurekaSpec2Pipeline.imprint_subtract - INFO - Step skipped.
2022-02-17 23:17:08,489 - stpipe.EurekaSpec2Pipeline.imprint_subtract - INFO - Step imprint_subtract done
2022-02-17 23:17:08,677 - stpipe.EurekaSpec2Pipeline.msa_flagging - INFO - Step msa_flagging running with args (<CubeModel(1, 416, 72) from ERS_NGTS10_2022_seg_042_rateints.fits>,).
2022-02-17 23:17:08,679 - stpipe.EurekaSpec2Pipeline.msa_flagging - INFO - Step msa_flagging parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': 'msa_flagging', 'search_output_file': True, 'input_dir': '/Volumes/DataDrive/MIRI_ERS_DC1/Stage1/S1_2022-02-17_miri_lrs_run1'}
2022-02-17 23:17:08,679 - stpipe.EurekaSpec2Pipeline.msa_flagging - INFO - Step skipped.
2022-02-17 23:17:08,682 - stpipe.EurekaSpec2Pipeline.msa_flagging - INFO - Step msa_flagging done
2022-02-17 23:17:08,863 - stpipe.EurekaSpec2Pipeline.srctype - INFO - Step srctype running with args (<CubeModel(1, 416, 72) from ERS_NGTS10_2022_seg_042_rateints.fits>,).
2022-02-17 23:17:08,865 - stpipe.EurekaSpec2Pipeline.srctype - INFO - Step srctype parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': 'srctype', 'search_output_file': True, 'input_dir': '/Volumes/DataDrive/MIRI_ERS_DC1/Stage1/S1_2022-02-17_miri_lrs_run1'}
2022-02-17 23:17:08,881 - stpipe.EurekaSpec2Pipeline.srctype - INFO - Input EXP_TYPE is MIR_LRS-SLITLESS
2022-02-17 23:17:08,882 - stpipe.EurekaSpec2Pipeline.srctype - INFO - Input SRCTYAPT = None
2022-02-17 23:17:08,882 - stpipe.EurekaSpec2Pipeline.srctype - WARNING - SRCTYAPT keyword not found in input; using SRCTYPE instead
2022-02-17 23:17:08,882 - stpipe.EurekaSpec2Pipeline.srctype - INFO - Input source type is unknown; setting default SRCTYPE = POINT
2022-02-17 23:17:08,887 - stpipe.EurekaSpec2Pipeline.srctype - INFO - Step srctype done
2022-02-17 23:17:09,066 - stpipe.EurekaSpec2Pipeline.flat_field - INFO - Step flat_field running with args (<CubeModel(1, 416, 72) from ERS_NGTS10_2022_seg_042_rateints.fits>,).
2022-02-17 23:17:09,067 - stpipe.EurekaSpec2Pipeline.flat_field - INFO - Step flat_field parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': 'flat_field', 'search_output_file': True, 'input_dir': '/Volumes/DataDrive/MIRI_ERS_DC1/Stage1/S1_2022-02-17_miri_lrs_run1', 'save_interpolated_flat': False, 'user_supplied_flat': None, 'inverse': False}
2022-02-17 23:17:09,174 - stpipe.EurekaSpec2Pipeline.flat_field - WARNING - Keyword CDP_PARTIAL_DATA does not correspond to an existing DQ mnemonic, so will be ignored
2022-02-17 23:17:09,175 - stpipe.EurekaSpec2Pipeline.flat_field - WARNING - Keyword CDP_LOW_QUAL does not correspond to an existing DQ mnemonic, so will be ignored
2022-02-17 23:17:09,175 - stpipe.EurekaSpec2Pipeline.flat_field - WARNING - Keyword CDP_UNRELIABLE_ERROR does not correspond to an existing DQ mnemonic, so will be ignored
2022-02-17 23:17:09,175 - stpipe.EurekaSpec2Pipeline.flat_field - WARNING - Keyword DIFF_PATTERN does not correspond to an existing DQ mnemonic, so will be ignored
2022-02-17 23:17:09,271 - stpipe.EurekaSpec2Pipeline.flat_field - INFO - Step flat_field done
2022-02-17 23:17:09,446 - stpipe.EurekaSpec2Pipeline.straylight - INFO - Step straylight running with args (<CubeModel(1, 416, 72) from ERS_NGTS10_2022_seg_042_rateints.fits>,).
2022-02-17 23:17:09,447 - stpipe.EurekaSpec2Pipeline.straylight - INFO - Step straylight parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': 'straylight', 'search_output_file': True, 'input_dir': '/Volumes/DataDrive/MIRI_ERS_DC1/Stage1/S1_2022-02-17_miri_lrs_run1', 'method': 'ModShepard', 'roi': 50, 'power': 1.0}
2022-02-17 23:17:09,448 - stpipe.EurekaSpec2Pipeline.straylight - INFO - Step skipped.
2022-02-17 23:17:09,451 - stpipe.EurekaSpec2Pipeline.straylight - INFO - Step straylight done
2022-02-17 23:17:09,631 - stpipe.EurekaSpec2Pipeline.fringe - INFO - Step fringe running with args (<CubeModel(1, 416, 72) from ERS_NGTS10_2022_seg_042_rateints.fits>,).
2022-02-17 23:17:09,633 - stpipe.EurekaSpec2Pipeline.fringe - INFO - Step fringe parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': 'fringe', 'search_output_file': True, 'input_dir': '/Volumes/DataDrive/MIRI_ERS_DC1/Stage1/S1_2022-02-17_miri_lrs_run1'}
2022-02-17 23:17:09,633 - stpipe.EurekaSpec2Pipeline.fringe - INFO - Step skipped.
2022-02-17 23:17:09,636 - stpipe.EurekaSpec2Pipeline.fringe - INFO - Step fringe done
2022-02-17 23:17:09,822 - stpipe.EurekaSpec2Pipeline.pathloss - INFO - Step pathloss running with args (<CubeModel(1, 416, 72) from ERS_NGTS10_2022_seg_042_rateints.fits>,).
2022-02-17 23:17:09,823 - stpipe.EurekaSpec2Pipeline.pathloss - INFO - Step pathloss parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': 'pathloss', 'search_output_file': True, 'input_dir': '/Volumes/DataDrive/MIRI_ERS_DC1/Stage1/S1_2022-02-17_miri_lrs_run1', 'inverse': False, 'source_type': None}
2022-02-17 23:17:09,823 - stpipe.EurekaSpec2Pipeline.pathloss - INFO - Step skipped.
2022-02-17 23:17:09,827 - stpipe.EurekaSpec2Pipeline.pathloss - INFO - Step pathloss done
2022-02-17 23:17:10,009 - stpipe.EurekaSpec2Pipeline.barshadow - INFO - Step barshadow running with args (<CubeModel(1, 416, 72) from ERS_NGTS10_2022_seg_042_rateints.fits>,).
2022-02-17 23:17:10,010 - stpipe.EurekaSpec2Pipeline.barshadow - INFO - Step barshadow parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': 'barshadow', 'search_output_file': True, 'input_dir': '/Volumes/DataDrive/MIRI_ERS_DC1/Stage1/S1_2022-02-17_miri_lrs_run1', 'inverse': False, 'source_type': None}
2022-02-17 23:17:10,010 - stpipe.EurekaSpec2Pipeline.barshadow - INFO - Step skipped.
2022-02-17 23:17:10,014 - stpipe.EurekaSpec2Pipeline.barshadow - INFO - Step barshadow done
2022-02-17 23:17:10,196 - stpipe.EurekaSpec2Pipeline.photom - INFO - Step photom running with args (<CubeModel(1, 416, 72) from ERS_NGTS10_2022_seg_042_rateints.fits>,).
2022-02-17 23:17:10,197 - stpipe.EurekaSpec2Pipeline.photom - INFO - Step photom parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': 'photom', 'search_output_file': True, 'input_dir': '/Volumes/DataDrive/MIRI_ERS_DC1/Stage1/S1_2022-02-17_miri_lrs_run1', 'inverse': False, 'source_type': None}
2022-02-17 23:17:10,198 - stpipe.EurekaSpec2Pipeline.photom - INFO - Step skipped.
2022-02-17 23:17:10,201 - stpipe.EurekaSpec2Pipeline.photom - INFO - Step photom done
2022-02-17 23:17:10,394 - stpipe.EurekaSpec2Pipeline.extract_1d - INFO - Step extract_1d running with args (<CubeModel(1, 416, 72) from /Volumes/DataDrive/MIRI_ERS_DC1/Stage2/S2_2022-02-17_miri_lrs_run1/ERS_NGTS10_2022_seg_042_calints.fits>,).
2022-02-17 23:17:10,396 - stpipe.EurekaSpec2Pipeline.extract_1d - INFO - Step extract_1d parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': 'x1dints', 'search_output_file': True, 'input_dir': '/Volumes/DataDrive/MIRI_ERS_DC1/Stage1/S1_2022-02-17_miri_lrs_run1', 'smoothing_length': None, 'bkg_fit': 'poly', 'bkg_order': None, 'bkg_sigma_clip': 3.0, 'log_increment': 50, 'subtract_background': None, 'use_source_posn': None, 'apply_apcorr': True}
2022-02-17 23:17:10,428 - stpipe.EurekaSpec2Pipeline.extract_1d - INFO - Using EXTRACT1D reference file /Users/tjbell1/crds_cache/references/jwst/miri/jwst_miri_extract1d_0004.json
2022-02-17 23:17:10,441 - stpipe.EurekaSpec2Pipeline.extract_1d - INFO - Using APCORR file /Users/tjbell1/crds_cache/references/jwst/miri/jwst_miri_apcorr_0007.fits
2022-02-17 23:17:10,498 - stpipe.EurekaSpec2Pipeline.extract_1d - WARNING - The photom step has not been run.
2022-02-17 23:17:10,498 - stpipe.EurekaSpec2Pipeline.extract_1d - WARNING - spectral_order is None; using 1
2022-02-17 23:17:10,498 - stpipe.EurekaSpec2Pipeline.extract_1d - INFO - Processing spectral order 1
2022-02-17 23:17:10,507 - stpipe.EurekaSpec2Pipeline.extract_1d - INFO - Using extraction limits:
2022-02-17 23:17:10,507 - stpipe.EurekaSpec2Pipeline.extract_1d - INFO - xstart=32.5, xstop=42.5, ystart=7, ystop=397
2022-02-17 23:17:10,508 - stpipe.EurekaSpec2Pipeline.extract_1d - WARNING - /Users/tjbell1/miniconda3/envs/eureka2/lib/python3.9/site-packages/jwst/extract_1d/extract.py:1935: RuntimeWarning: Mean of empty slice
wavelength = np.nanmean(wl[sy0:sy1, sx0:sx1], axis=1)

Traceback (most recent call last):
File "/Users/tjbell1/miniconda3/envs/eureka2/lib/python3.9/site-packages/jwst/pipeline/calwebb_spec2.py", line 113, in process
result = self.process_exposure_product(
File "/Users/tjbell1/miniconda3/envs/eureka2/lib/python3.9/site-packages/jwst/pipeline/calwebb_spec2.py", line 285, in process_exposure_product
x1d = self.extract_1d(resampled)
File "/Users/tjbell1/miniconda3/envs/eureka2/lib/python3.9/site-packages/stpipe/step.py", line 430, in run
step_result = self.process(*args)
File "/Users/tjbell1/miniconda3/envs/eureka2/lib/python3.9/site-packages/jwst/extract_1d/extract_1d_step.py", line 269, in process
result = extract.run_extract1d(
File "/Users/tjbell1/miniconda3/envs/eureka2/lib/python3.9/site-packages/jwst/extract_1d/extract.py", line 2675, in run_extract1d
output_model = do_extract1d(
File "/Users/tjbell1/miniconda3/envs/eureka2/lib/python3.9/site-packages/jwst/extract_1d/extract.py", line 3381, in do_extract1d
b_var_rnoise, b_var_flat, npixels, dq, prev_offset = extract_one_slit(
File "/Users/tjbell1/miniconda3/envs/eureka2/lib/python3.9/site-packages/jwst/extract_1d/extract.py", line 4032, in extract_one_slit
b_var_rnoise, b_var_flat, npixels, dq = extract_model.extract(data, var_poisson, var_rnoise,
File "/Users/tjbell1/miniconda3/envs/eureka2/lib/python3.9/site-packages/jwst/extract_1d/extract.py", line 1987, in extract
image = np.transpose(data, (1, 0))
File "<array_function internals>", line 180, in transpose
File "/Users/tjbell1/miniconda3/envs/eureka2/lib/python3.9/site-packages/numpy/core/fromnumeric.py", line 660, in transpose
return _wrapfunc(a, 'transpose', axes)
File "/Users/tjbell1/miniconda3/envs/eureka2/lib/python3.9/site-packages/numpy/core/fromnumeric.py", line 57, in _wrapfunc
return bound(*args, **kwds)
ValueError: axes don't match array
Traceback (most recent call last):
File "/Users/tjbell1/Research/JWST/ERS/Eureka/demos/JWST/run_eureka.py", line 18, in
s2_meta = s2.calibrateJWST(eventlabel)
File "/Users/tjbell1/Research/JWST/ERS/Eureka/demos/JWST/../../eureka/S2_calibrations/s2_calibrate.py", line 130, in calibrateJWST
pipeline.run_eurekaS2(filename, meta, log)
File "/Users/tjbell1/Research/JWST/ERS/Eureka/demos/JWST/../../eureka/S2_calibrations/s2_calibrate.py", line 236, in run_eurekaS2
self(filename)
File "/Users/tjbell1/miniconda3/envs/eureka2/lib/python3.9/site-packages/stpipe/step.py", line 430, in run
step_result = self.process(*args)
File "/Users/tjbell1/miniconda3/envs/eureka2/lib/python3.9/site-packages/jwst/pipeline/calwebb_spec2.py", line 131, in process
raise RuntimeError('\n'.join(failures))
RuntimeError: Traceback (most recent call last):
File "/Users/tjbell1/miniconda3/envs/eureka2/lib/python3.9/site-packages/jwst/pipeline/calwebb_spec2.py", line 113, in process
result = self.process_exposure_product(
File "/Users/tjbell1/miniconda3/envs/eureka2/lib/python3.9/site-packages/jwst/pipeline/calwebb_spec2.py", line 285, in process_exposure_product
x1d = self.extract_1d(resampled)
File "/Users/tjbell1/miniconda3/envs/eureka2/lib/python3.9/site-packages/stpipe/step.py", line 430, in run
step_result = self.process(*args)
File "/Users/tjbell1/miniconda3/envs/eureka2/lib/python3.9/site-packages/jwst/extract_1d/extract_1d_step.py", line 269, in process
result = extract.run_extract1d(
File "/Users/tjbell1/miniconda3/envs/eureka2/lib/python3.9/site-packages/jwst/extract_1d/extract.py", line 2675, in run_extract1d
output_model = do_extract1d(
File "/Users/tjbell1/miniconda3/envs/eureka2/lib/python3.9/site-packages/jwst/extract_1d/extract.py", line 3381, in do_extract1d
b_var_rnoise, b_var_flat, npixels, dq, prev_offset = extract_one_slit(
File "/Users/tjbell1/miniconda3/envs/eureka2/lib/python3.9/site-packages/jwst/extract_1d/extract.py", line 4032, in extract_one_slit
b_var_rnoise, b_var_flat, npixels, dq = extract_model.extract(data, var_poisson, var_rnoise,
File "/Users/tjbell1/miniconda3/envs/eureka2/lib/python3.9/site-packages/jwst/extract_1d/extract.py", line 1987, in extract
image = np.transpose(data, (1, 0))
File "<array_function internals>", line 180, in transpose
File "/Users/tjbell1/miniconda3/envs/eureka2/lib/python3.9/site-packages/numpy/core/fromnumeric.py", line 660, in transpose
return _wrapfunc(a, 'transpose', axes)
File "/Users/tjbell1/miniconda3/envs/eureka2/lib/python3.9/site-packages/numpy/core/fromnumeric.py", line 57, in _wrapfunc
return bound(*args, **kwds)
ValueError: axes don't match array

Since merging the latest edits and re-running S5, I have noticed several things that I want to flag for others if that can help:

• my S4 outputs obtained with the previous version of the code were not compatible with the new S5 read-in (there was stuff missing in the S4 meta file). I had to re-run S4, then S5 worked fine -> just be aware that S5 is not reverse-compatible with previously-generated S4 outputs
• the template S5 ecf file for parameters would require the time_offset parameter to be specified (even if it is zero), otherwise I get an error.
  For me, it was only a matter of adding one line to the params ecf file. It is a very minor edit which I will include in my next pull request if needed, but it might help others if you run into this error! Another option would be to set it to zero by default if it is not specified in the params file.

As of right now we are directly fitting for the quadratic limb darkening coefficients, but this is not very efficient and is hard to place priors on. We'd be better off using the reparameterized versions (q₁ = (u₁ + u₂)² and q₂ = 0.5*u₁(u₁ + u₂)⁻¹ with uniform priors in the interval [0, 1]) à la Kipping (2013).

Currently Stage 1 does not connect to Stage 2, so the two cannot be called sequentially and users need to specify the exact output folder from Stage 1. Some small tweaks to Stage 2 (mirroring the code in Stages 3+) will allow a more fluid transition

NIRCam's detector is read out row-by-row instead of column-by-column (at least by default, although this can be switched at the detriment of bright star performance). As a result, NIRCam's 1/f noise is expressed along each row instead of along columns, so performing row-by-row background subtraction will better remove 1/f noise in NIRCam data than columnb-by-column. Worse yet, NIRCam has four amplifiers along the dispersion direction which can each have their own constant offset (and maybe even 1/f noise characteristics - I sure hope not though), so it may even be necessary to perform row-by-row background subtraction per amplifier. It will be important for us to look into how we can perform row-by-row background subtraction for NIRCam observations without self-subtracting the spectra - I can ask the NIRCam TSO team to see if they have any tricks they'd be willing to share, or we can try to come up with our own fully-independent solution and see which works better.

When trying to run a shared fit of a 20-bin NIRCam transit with rp 'free'; per 'fixed'; t0 'shared'; inc 'shared'; a 'shared'; ecc 'fixed'; w 'fixed'; limb_dark 'quadratic'; u1 'free'; u2 'free'; and c0 'free' dynesty runs for ~27 minutes and the eff % initially rises to ~36% but then gradually falls to just below 10% (9.996% this time) at which point it seems to stop making any progress (although my CPU continues to work hard). This has happened twice to me with different fitted parameters and priors but with the same data set and still using some kind of shared fit. With this most recent fit I increased the run_nlive to 7627 given the massive dimensionality of the fit I'd attempted earlier (with 4-parameter limb darkening instead of quadratic which gives 123 fitted parameters which gives a minimum run_nlive of 7626). Previously I started from a lsq fitted starting point, but lsq seems to fail so I started instead from my initial guesses directly. For the earlier fit with the 4-parameter limb darkening, dynesty wouldn't even begin to fit and said "Exception: Convergence failure in calculation of scale factor for integration step size".

Encountering some strange behaviour when running Stage 1 on the NIRCam simulated data. As this dataset is split across two files, it should end up producing two separate _rateints.fits files, but it instead only produces one. The confusing thing is that the code definitely runs on each file, and the .log file reflects this, but only the first segmented file appears.

EDIT: The data is actually split over 21 files, but I've been testing on just two for speed.

A little stumped, and was hoping @taylorbell57 / @evamariaa could clone the recent s1 branch and see if the same thing happens on your end?

Cheers!

S1_nirx_template.log

S1_nirx_template.ecf.zip

Just logging this down as I'm about to sign off for the day, but I'll pick things back up next week. Stage 3 seems to be working on the NIRSpec dataset, but crashes during a plotting routine towards the end.

File "/Users/acarter/Documents/TRANSITS/EUREKA/TESTING_NIRSPEC/run_eureka.py", line 14, in <module>
   s3_meta = s3.reduceJWST(eventlabel)#, s2_meta=s2_meta)
 File "/Users/acarter/Documents/TRANSITS/EUREKA/TESTING_NIRSPEC/../eureka_s1/eureka/S3_data_reduction/s3_reduce.py", line 346, in reduceJWST
   plots_s3.optimal_spectrum(data, meta, n)
 File "/Users/acarter/Documents/TRANSITS/EUREKA/TESTING_NIRSPEC/../eureka_s1/eureka/S3_data_reduction/plots_s3.py", line 119, in optimal_spectrum
   plt.errorbar(range(subnx), optspec[n], opterr[n], fmt='-', color='C2', ecolor='C2', label='Optimal Spec')
 File "/Users/acarter/anaconda3/envs/eureka_133/lib/python3.10/site-packages/matplotlib/pyplot.py", line 2509, in errorbar
   return gca().errorbar(
 File "/Users/acarter/anaconda3/envs/eureka_133/lib/python3.10/site-packages/matplotlib/__init__.py", line 1412, in inner
   return func(ax, *map(sanitize_sequence, args), **kwargs)
 File "/Users/acarter/anaconda3/envs/eureka_133/lib/python3.10/site-packages/matplotlib/axes/_axes.py", line 3467, in errorbar
   barcols.append(lines_func(
 File "/Users/acarter/anaconda3/envs/eureka_133/lib/python3.10/site-packages/matplotlib/__init__.py", line 1412, in inner
   return func(ax, *map(sanitize_sequence, args), **kwargs)
 File "/Users/acarter/anaconda3/envs/eureka_133/lib/python3.10/site-packages/matplotlib/axes/_axes.py", line 1142, in vlines
   minx = x.min()
 File "/Users/acarter/anaconda3/envs/eureka_133/lib/python3.10/site-packages/numpy/ma/core.py", line 5703, in min
   result = self.filled(fill_value).min(
 File "/Users/acarter/anaconda3/envs/eureka_133/lib/python3.10/site-packages/numpy/core/_methods.py", line 44, in _amin
   return umr_minimum(a, axis, None, out, keepdims, initial, where)
TypeError: '<=' not supported between instances of 'int' and 'str'

On a clean install of Eureka (48c0443), when running S5 with run_myfuncs set to [transit], plot_fit() in plots_s5.py breaks. The code looks to use a variable called model_sys to plot the detrended lightcurve, and on line 54 selects a subarray of this model. However, if there's no systematics model specified in the ECF, this model_sys is given a default OOT flux of 1.0 in model.syseval(), which pulls the systematics array out of the model object. The default return value here should probably be an ndarray of the same length as the model, with all the values set to 1.0, rather than just a single value.

Hi all, with my new computer I have much more RAM and am now able to process the NIRSpec data. My first attempt at doing so turned out very poorly (see attached image below). I will note that this was reduced with jwst v1.3.3, but I don't think that is likely to be the source of any issues. I've included the ecf files used in Stage 2 and 3 for reference (in a zip file to pacify GitHub). If @AarynnCarter or others have gotten better results with NIRSpec and can suggest better ecf settings, that would be much appreciated.

NIRSpec ECFs.zip

Add a wrapper for the Image2Pipeline and call it instead of the Spec2Pipeline when appropriate

currently, new parameters are added to the Parameters instance like this:

params.a = wasp107b_par['a/Rs'], 'free', 10., 25.

Then, in fitters.py, we access the atrributes like this:
if param[1] == 'free': ...

This is risky!! what happens if the attributes are entered out of order? I suggest reformatting this to make it more fool-proof

Laura to look into what MPFIT does

Currently have only free or fixed model parameters in S5, and would be good to add an option to have a Gaussian prior to use prior constraints from previous high precision observations.

As of right now all Stage 5 models are in one file called models.py, but as we add more models thatt format will become very inconvenient. Instead, we should make a folder called models and then an init.py file inside of that so that we still have the easy access to all the different models but can have different models in different python files.

Hi all, so @cpiaulet had some issues installing Eureka which made me think deeper about our installation instructions. We currently say that people should do python setup.py install which is an outdated and somewhat dangerous/complicated way of having people install a package. The recommended way to install things now is to do pip install . within the downloaded package folder which itself calls the setup.py file but does all the work of checking for dependencies. If we simply require that everyone does the pip install option, then we can clean out a lot of what is in the requirements.py file which is already incomplete and out of date since many of the version requirements are coming from other packages we're installing rather than any requirements of our code. With this option, we only put the packages we actually call with Eureka into the requirements.txt file which should be quite easy. However, down the road someone will likely try running python setup.py install out of habit and the command will fail. If we don't want to have that command fail then we'll need to do a lot of upkeep on the requirements.txt file, otherwise we can just have a note on the readthedocs FAQ page that tells people that they should try installing again with pip install .

What path do you recommend we take?

Need to test newly implemented Stage 2 code on the other instruments. Already I've found that the currently existing code does not work for MIRI and NIRCam as their FITS files do not have a GRATING keyword in the header.

here are some suggested performance metrics to test how Eureka! is performing compared to other codes:

• plot of pointing stability over time (drift in x and y direction relative to first exposure)
• plot of background counts over time (median, or by wavelength channel?)
• from Kevin: MAD calculation that’s in Stage 3:

normspec = optspec / np.mean(optspec, axis=0)
for m in range(n_int):
        ediff[m] = 1e6 * np.median(np.abs(np.ediff1d(normspec[m])))
MAD = np.mean(ediff)

• rms of the light curve fits compared to the photon noise (broadband and spectroscopic light curves)
• Allan deviation plots for each wavelength channel to test for correlated noise (Kevin/I have a code snippet that can do this - ask one of us if you want it)
• goodness of fit for the lightcurves. No metric is perfect, so it would be great to have multiple options: chi2, BIC, AIC, and Bayesian evidence (this last is easy to get with dynesty, which should be v easy to implement now that Sebastian has emcee working)
• from Aarynn: posterior distributions of fit parameters
• from Aarynn: transmission spectrum

There are several instances where a reload statement is present in Eureka!, and while this is normally fine it causes significant issues with trying to make a models folder for S5 rather than a models.py file which contains every single model (Issue #90). The issue is that reload makes completely new classes, so tests that use type(params)==Parameters or isinstance(params,Parameters) where params was made using a reloaded version of Parameters all return False. This is a known bug/feature (e.g. here) and we won't be able to avoid it without removing the reload statements. Simply reloading Parameters in all the files won't work either because they will still all end up being different classes.

Rather than just unilaterally removing all the reload statements (again), I thought I'd open this issue to discuss why others want reload statements to see if we can find a solution that works for all.

Apparently repeated fits to the same data with the lsq fitter were providing very different results. It's unclear to me if this issue still exists as of PR #75, but if so it could potentially be driven by arrays being edited in place.

Get ReadTheDocs running

For some reason, the readECF function won't allow spaces in the fit_method list in the S5 ecf. If there are spaces, it splits the string at the spaces and keeps only the first value. This should be addressed to ensure ease of use

If you set ramp_fit_max_cores to all in the ecf file, the readecf function interprets it as the python built-in all function rather than the string all. Perhaps quotes are needed? Using half works fine without quotes, but I haven't tested it with quotes

Right now it is pretty inconvenient to gather together an emission spectrum and plot it - we should either add a Stage 6 to do this kind of thing or add a final step to Stage 5 to gather everything together.

I've received reports that the Data save files are being overwritten because the file names are not unique for each input file. We should look into this.

If I run S3, then S4 of Eureka! from the S2 jwst pipeline products (I tried NIRCam and NIRSpec so far), I get very small error bars on the points in individual wavelength-dependent light curves.

Is there some other unit conversion that needs to happen on the uncertainties to correct for this? @taylorbell57 tells me this does not occur if the photom step is skipped.

On a related note, do we have plans to add the single-point scatter as a free parameter in the S5 fit? It could be useful if errors are over- or underestimated in previous steps.

Link in README is broken (https://eurekadocs.readthedocs.io/en/)

Write an equivalent of the Stage 1 reduction pipeline that can be executed directly within Eureka!

List of current tasks in order of priority:

1. Default pipeline processing

2. Differenced frames processing

3. Custom weighting functions for the default pipeline.

4. Saving of individual frames for full pipeline processing to allow for finer time resolution / manual inspection.

I and others have found that calling lsq before emcee can be unhelpful as sometimes the estimator fails and starts emcee in a worse place than where we initialized it with the parameter ecf. This should be an optional step. Additionally, lsq is being called before dynesty which is useless since dynesty doesn't take a starting point anyway.

I just switched to using my new Macbook Pro to run Eureka and I have encountered a very strange issue. Stage 2 runs well to completion, and then when Stage 3 is called it runs as expected until multiprocessing.Pool is called here at which point Stage 2 starts running again in as many threads as was provided in the Stage 3 ncpu ecf parameter... I've not encountered this before on Ubuntu and I don't think anything has changed with these relevant files since I last successfully ran S2->S4. I'm not sure how this could possibly be happening, but I'll look into it more later.

After Stage 3 starts and I get the " Performing background subtraction" print statement, I get ncpu copies of this error message:

2022-01-19 15:49:22,337 - stpipe - WARNING - /Users/tjbell1/miniconda3/envs/ers/lib/python3.9/site-packages/photutils/detection/findstars.py:33: AstropyDeprecationWarning: _StarFinderKernel was moved to the photutils.detection._utils module. Please update your import statement.
warnings.warn(f'{name} was moved to the {deprecated[name]} module. '

followed by ncpu copies of this error message:

WARNING (only important for Stage 5): The $EXOCTK_DATA environment variable is not set. Please set the value of this variable to point to the location of the exoctk_data download folder. Users may retreive this folder by clicking the "ExoCTK Data Download" button on the ExoCTK website, or by using the exoctk.utils.download_exoctk_data() function.

which then is followed by "Starting Stage 2 Reduction" which should definitely not happen in Stage 3. Things then get chaotic as ncpu threads all start running Stage 2 and simultaneously logging text to the terminal.

For reference, I have attached a file (eureka_environment.txt) which exactly reproduces my conda environment on Macbooks. Relatedly, the current requirements.txt file was not sufficient to create a new environment on MacOS, so that's something we'll need to solve as well.

This issue pertains the NIRSpec data from the tiny NIRSpec files uploaded by @AarynnCarter in BOX

I'm still unsure about the nature of the issue and whether it is a Eureka error or a data error, but after a conversation with @taylorbell57 and his experience with a similar issue, I'm officially opening this issue for other people to take a look at it.

THE ERROR

THE ISSUE

The error occurs affter running Eureka's S2 wrapper on the JWST pipeline and then trying to use the resulting file on Eureka's Stage 3, the error appears during the S3 run. It seems that an array mismatch happens between the subdata and subgain arrays (the trimmed data and gain arrays). After further tracking the issue, it seems the subgain array only has 2 dimensions (x,y) but the subdata array contains 3 (n,x,y) of which n stands for a single iteration. Eureka is not doing anything to trim that third dimension and therefore the arrays have a mismatch. Yet, even after zeroing in on a single iteration and trimming the subdata array to, say, [0, : , : ] there is still a mismatch in the x dimension between the subdata and subgain arrays that I was unsure how to trace since I am not too familiar with the actual nature of the files themselves. Hence this issue being opened.

THE FILES USED FOR THIS PARTICULAR RUN

• Data Simulation Working Group/NIRSpec/Tiny/jwdata0010010_11010_0001_NRS1_TINY_uncal.fits (used to run Eureka's S2 on it)
• Data Simulation Working Group/NIRSpec/Ancillary Files

HANDWRITTEN NOTES ON THE ISSUE

IN-DEPTH NOTES ON ISSUE

Leonardo dos Santos to start work on this

The Stage 4 cross-correlation step (used to correct for drift/jitter in the 1D spectra) fails on every frame when run on the MIRI simulated data. While the MIRI data are blackbody simulations (compared to the NIRCam simulated data which have spectral features), I think there should still be plenty of curvature and features in the 1-D spectrum for the cross-correlation step to work. Because the step is run inside a try-catch statement, I don't yet understand why it is failing, but the output cross-correlation strength plot looks like an exponential decay, so it seems like it wants to shift the spectrum unreasonably to the left. I have tried setting sub_mean to True and False, but neither option works. The code does work for NIRCam data (haven't been able to test on NIRSpec yet), so the issue seems to be MIRI specific for some reason.

If others want to try and tackle this, please feel free - otherwise I'll try to get to this in a day or few.

On a clean install of Eureka (48c0443), when running S5 with the polynomial model specified, the fitting steps fail because the PolynomialModel is missing the attribute 'longparamlist'. This attribute is present in TransitModel, but didn't make it over to the PolynomialModel yet.