Jump step refactoring about jwst HOT 9 OPEN

Comment by David Law on JIRA:

Maria Pena-Guerrero Which MIRI file are you using for the above?

from jwst.

Comment by Maria Pena-Guerrero on JIRA:

David Law the file is 
jw01283001001_03101_00001_mirimage_uncal.fits

from jwst.

Comment by David Law on JIRA:

Hm, just to add to this puzzle I tried processing this file myself with all of the latest versions, through a default strun of calwebb_detector1

The entire process completed in about 12 minutes, with jump the longest running step at about 7 minutes.

I've got 64 GB of physical memory in my MacBook Pro; 61 GB of that got used at peak, though I saw the python usage spike up to 153 GB during the jump step (presumably using virtual memory to handle it).

from jwst.

Comment by Maria Pena-Guerrero on JIRA:

David Law more to the mystery.... did you run the Detector1 pipeline with a file or did you open the datamodel first? i.e.

option A  ->  det1.call(uncal_file.fits)

option B  ->  det1.call(uncal_model)

For me the run takes significantly more memory if I do option B

from jwst.

Comment by David Law on JIRA:

My initial run was just using strun from a terminal command line.  Took 12 minutes, peaked at 153 GB of python memory (61 GB physical), and when it finished RAM usage returned to the normal baseline.

Using your option A (i.e., running on a file) within a jupyter notebook it took 12 minutes, peaked at 153 GB (with a brief excursion to 172 GB that I may have missed before) and 61 GB physical.  When it finished though physical RAM usage stayed high at 34 GB until I explicitly halted the notebook.

Using your option B (i.e, running on a datamodel) within a jupyter notebook it peaked at 182 GB used, and stalled out in the jump step.  After 20 minutes in the 'two point difference' alone I killed it.

So, I'm also seeing a big difference in whether I pass in a file or a datamodel.  Datamodel presumably required enough additional resources that it stalled out for me as well.

from jwst.

Comment by Tyler Pauly on JIRA:

It may be worth testing in an environment with the dev version of stpipe - some of the memory issues using file vs. datamodel may be linked to the log referencing bug found and fixed here: spacetelescope/stpipe#171

I believe a patch release of stpipe is planned, but I don't know its status.

from jwst.

Comment by David Law on JIRA:

Just repeated the test as suggested by Tyler Pauly with the dev version of stpipe.

Major memory difference.  Option A ran in 11 minutes, maxing at 60 GB used by python vs the 172 GB it used to take.  Option B maxed at 94 GB vs previous 182 GB, but still hung in the jump step, where I killed it after taking over 20 min in the 2-pt-difference section that took option A just 2.5 minutes to complete.

So the memory has improved, but running with a datamodel input is still hanging despite the memory usage being far less than what ran successfully in the original option A.  Also still about 12 GB hanging around in memory until the kernel is restarted, even with option A.

from jwst.

Comment by Tyler Pauly on JIRA:

Opened a PR on the first steps toward reducing memory usage in jump - flamegraph shows a run of standalone jump step on a refpix output. More work to be done, but for this input file it's a quick 10GB reduction.

from jwst.

Comment by Maria Pena-Guerrero on JIRA:

for documentation purposes, I am putting the link to a Help Desk ticket related to their machine being killed at the jump step due to memory issues: https://stsci.service-now.com/nav_to.do?uri=%2Fincident.do%3Fsys_id%3Da878bdf483e14e102c4f1a226daad3f1%26sysparm_record_list%3Dassignment_group%3D8dbea402dbf2620033b55dd5ce9619ba%5Estate%3D2%5EORstate%3D3%5EORstate%3D1%5Ecompany%3D%5EORwatch_listCONTAINSa89f418ddb8f620042685434ce961990%5EORcompany%3Dc3478b5edb013e0042685434ce961951%5EORcaller_id%3Da89f418ddb8f620042685434ce961990%5EORopened_by%3Da89f418ddb8f620042685434ce961990%5EORDERBYstate%26sysparm_record_row%3D8%26sysparm_record_rows%3D47%26sysparm_record_target%3Dincident%26sysparm_view%3DJWST%26sysparm_view_forced%3Dtrue

from jwst.