Ramps-to-slopes: Detector1Pipeline (MIRI CALDETECTOR1)¶
The Detector1Pipeline takes DMS Level 1B data and applies basic detector-level corrections to all exposure types. It is applied to one exposure at a time. It is sometimes referred to as “ramps-to-slopes” processing, because the input raw data are in the form of one or more ramps (integrations) containing accumulating counts from the non-destructive detector readouts and the output is a corrected countrate (slope) image.
Official documentation for Detector1Pipeline can be found here:
https://jwst-pipeline.readthedocs.io/en/latest/jwst/pipeline/calwebb_detector1.html
The Detector1Pipeline comprises a linear series of steps to first calibrate the pixel ramps, detect cosmic ray jumps, and fit a linear function to the calibrate ramps to produce a slope or DMS Level 2A image. The steps applied to MIRI data in order are:
Step |
Description |
|---|---|
|
populates the DQ mask for the input dataset |
|
flags saturated pixel values |
|
flags the first group in every integration as bad |
|
flags the last group in every integration as bad |
|
corrects for detector non-linearity |
|
correct for the reset switch charge decay |
|
subtracts dark current data |
|
corrects for pixel drifts using the reference pixels |
|
detects jumps in the ramps by looking for outliers in each integration |
|
determines count rate for each pixel by performing a linear fit to ramps |
For more information and examples of each of the steps click on the links in the side bar.
Input data¶
An example of running the file through the Detector1Pipeline is now shown using a simple simulated observation of a galaxy and some point sources with the MIRI Imager (F1130W filter) produced with MIRISim v2.3.
Python¶
Start by importing what will be used and set the CRDS_CONTEXT
# imports
import os, glob, shutil
import numpy as np
from matplotlib.colors import LogNorm
import matplotlib.pyplot as plt
from subprocess import call
from jwst import datamodels
# set the CRDS_CONTEXT
os.environ["CRDS_CONTEXT"] = "jwst_0641.pmap"
Import Detector1Pipeline and print the docstring to show some information
from jwst.pipeline import Detector1Pipeline
print(Detector1Pipeline.__doc__)
Detector1Pipeline: Apply all calibration steps to raw JWST
ramps to produce a 2-D slope product. Included steps are:
group_scale, dq_init, saturation, ipc, superbias, refpix, rscd,
lastframe, linearity, dark_current, persistence, jump detection,
ramp_fit, and gain_scale.
Set the name of the input file, where the output should be saved and run the pipeline. The output level 2A files will be saved in my_output_dir as _rate.fits. Note that we must explicitly skip the IPC step when running from Python. This is achieved by passing the skip parameter to the step in a Python dict.
Parameters used:
output_use_model : boolean, optional, default=False
propagate the input filename to the output
save_results: boolean, optional, default=False
save the results to file
output_dir : boolean, optional, default is the working directory
the location to save the output
steps : dict, optional
dict of user-defined parameters for individual steps
Note that the Detector1Pipeline will return the rate datamodel so we set this to the dm variable.
# user specified
my_input_file = 'IMA_simulation/det_image_seq1_MIRIMAGE_F1130Wexp1.fits'
my_output_dir = 'IMA_demo_output'
# the output directory should be created if it doesn't exist
if not os.path.exists(my_output_dir):
os.mkdir(my_output_dir)
# run the pipeline
dm = Detector1Pipeline.call(my_input_file, output_use_model=True, save_results=True,
output_dir=my_output_dir, steps={'ipc': {'skip': True},
'refpix': {'skip': True}})
2020-10-14 11:22:46,652 - CRDS - ERROR - Error determining best reference for 'pars-groupscalestep' = Unknown reference type 'pars-groupscalestep'
2020-10-14 11:22:46,658 - CRDS - ERROR - Error determining best reference for 'pars-dqinitstep' = Unknown reference type 'pars-dqinitstep'
2020-10-14 11:22:46,663 - CRDS - ERROR - Error determining best reference for 'pars-saturationstep' = Unknown reference type 'pars-saturationstep'
2020-10-14 11:22:46,668 - CRDS - ERROR - Error determining best reference for 'pars-ipcstep' = Unknown reference type 'pars-ipcstep'
2020-10-14 11:22:46,673 - CRDS - ERROR - Error determining best reference for 'pars-superbiasstep' = Unknown reference type 'pars-superbiasstep'
2020-10-14 11:22:46,677 - CRDS - ERROR - Error determining best reference for 'pars-refpixstep' = Unknown reference type 'pars-refpixstep'
2020-10-14 11:22:46,682 - CRDS - ERROR - Error determining best reference for 'pars-rscd_step' = Unknown reference type 'pars-rscd_step'
2020-10-14 11:22:46,688 - CRDS - ERROR - Error determining best reference for 'pars-firstframestep' = Unknown reference type 'pars-firstframestep'
2020-10-14 11:22:46,692 - CRDS - ERROR - Error determining best reference for 'pars-lastframestep' = Unknown reference type 'pars-lastframestep'
2020-10-14 11:22:46,697 - CRDS - ERROR - Error determining best reference for 'pars-linearitystep' = Unknown reference type 'pars-linearitystep'
2020-10-14 11:22:46,702 - CRDS - ERROR - Error determining best reference for 'pars-darkcurrentstep' = Unknown reference type 'pars-darkcurrentstep'
2020-10-14 11:22:46,707 - CRDS - ERROR - Error determining best reference for 'pars-persistencestep' = Unknown reference type 'pars-persistencestep'
2020-10-14 11:22:46,712 - CRDS - ERROR - Error determining best reference for 'pars-jumpstep' = Unknown reference type 'pars-jumpstep'
2020-10-14 11:22:46,717 - CRDS - ERROR - Error determining best reference for 'pars-rampfitstep' = Unknown reference type 'pars-rampfitstep'
2020-10-14 11:22:46,721 - CRDS - ERROR - Error determining best reference for 'pars-gainscalestep' = Unknown reference type 'pars-gainscalestep'
2020-10-14 11:22:46,724 - CRDS - ERROR - Error determining best reference for 'pars-detector1pipeline' = Unknown reference type 'pars-detector1pipeline'
2020-10-14 11:22:46,733 - stpipe.Detector1Pipeline - INFO - Detector1Pipeline instance created.
2020-10-14 11:22:46,734 - stpipe.Detector1Pipeline.group_scale - INFO - GroupScaleStep instance created.
2020-10-14 11:22:46,736 - stpipe.Detector1Pipeline.dq_init - INFO - DQInitStep instance created.
2020-10-14 11:22:46,738 - stpipe.Detector1Pipeline.saturation - INFO - SaturationStep instance created.
2020-10-14 11:22:46,739 - stpipe.Detector1Pipeline.ipc - INFO - IPCStep instance created.
2020-10-14 11:22:46,741 - stpipe.Detector1Pipeline.superbias - INFO - SuperBiasStep instance created.
2020-10-14 11:22:46,742 - stpipe.Detector1Pipeline.refpix - INFO - RefPixStep instance created.
2020-10-14 11:22:46,744 - stpipe.Detector1Pipeline.rscd - INFO - RSCD_Step instance created.
2020-10-14 11:22:46,746 - stpipe.Detector1Pipeline.firstframe - INFO - FirstFrameStep instance created.
2020-10-14 11:22:46,747 - stpipe.Detector1Pipeline.lastframe - INFO - LastFrameStep instance created.
2020-10-14 11:22:46,749 - stpipe.Detector1Pipeline.linearity - INFO - LinearityStep instance created.
2020-10-14 11:22:46,750 - stpipe.Detector1Pipeline.dark_current - INFO - DarkCurrentStep instance created.
2020-10-14 11:22:46,752 - stpipe.Detector1Pipeline.persistence - INFO - PersistenceStep instance created.
2020-10-14 11:22:46,754 - stpipe.Detector1Pipeline.jump - INFO - JumpStep instance created.
2020-10-14 11:22:46,755 - stpipe.Detector1Pipeline.ramp_fit - INFO - RampFitStep instance created.
2020-10-14 11:22:46,757 - stpipe.Detector1Pipeline.gain_scale - INFO - GainScaleStep instance created.
2020-10-14 11:22:46,837 - stpipe.Detector1Pipeline - INFO - Step Detector1Pipeline running with args ('IMA_simulation/det_image_seq1_MIRIMAGE_F1130Wexp1.fits',).
2020-10-14 11:22:46,847 - stpipe.Detector1Pipeline - INFO - Step Detector1Pipeline parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': 'IMA_demo_output', 'output_ext': '.fits', 'output_use_model': True, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_calibrated_ramp': False, 'steps': {'group_scale': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'dq_init': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'saturation': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'ipc': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'superbias': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'refpix': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'odd_even_columns': True, 'use_side_ref_pixels': True, 'side_smoothing_length': 11, 'side_gain': 1.0, 'odd_even_rows': True}, 'rscd': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'type': 'baseline'}, 'firstframe': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'lastframe': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'linearity': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'dark_current': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'dark_output': None}, 'persistence': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'input_trapsfilled': '', 'flag_pers_cutoff': 40.0, 'save_persistence': False, 'save_trapsfilled': True}, 'jump': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'rejection_threshold': 4.0, 'maximum_cores': 'none', 'flag_4_neighbors': True, 'max_jump_to_flag_neighbors': 200.0, 'min_jump_to_flag_neighbors': 10.0}, 'ramp_fit': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'int_name': '', 'save_opt': False, 'opt_name': '', 'maximum_cores': 'none'}, 'gain_scale': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}}}
2020-10-14 11:22:47,097 - stpipe.Detector1Pipeline - INFO - Prefetching reference files for dataset: 'det_image_seq1_MIRIMAGE_F1130Wexp1.fits' reftypes = ['dark', 'gain', 'linearity', 'mask', 'persat', 'readnoise', 'rscd', 'saturation', 'superbias', 'trapdensity', 'trappars']
2020-10-14 11:22:47,103 - stpipe.Detector1Pipeline - INFO - Prefetch for DARK reference file is '/Users/patrickkavanagh/crds_mirror/references/jwst/miri/jwst_miri_dark_0049.fits'.
2020-10-14 11:22:47,103 - stpipe.Detector1Pipeline - INFO - Prefetch for GAIN reference file is '/Users/patrickkavanagh/crds_mirror/references/jwst/miri/jwst_miri_gain_0008.fits'.
2020-10-14 11:22:47,104 - stpipe.Detector1Pipeline - INFO - Prefetch for LINEARITY reference file is '/Users/patrickkavanagh/crds_mirror/references/jwst/miri/jwst_miri_linearity_0024.fits'.
2020-10-14 11:22:47,105 - stpipe.Detector1Pipeline - INFO - Prefetch for MASK reference file is '/Users/patrickkavanagh/crds_mirror/references/jwst/miri/jwst_miri_mask_0023.fits'.
2020-10-14 11:22:47,106 - stpipe.Detector1Pipeline - INFO - Prefetch for PERSAT reference file is 'N/A'.
2020-10-14 11:22:47,106 - stpipe.Detector1Pipeline - INFO - Prefetch for READNOISE reference file is '/Users/patrickkavanagh/crds_mirror/references/jwst/miri/jwst_miri_readnoise_0059.fits'.
2020-10-14 11:22:47,107 - stpipe.Detector1Pipeline - INFO - Prefetch for RSCD reference file is '/Users/patrickkavanagh/crds_mirror/references/jwst/miri/jwst_miri_rscd_0010.fits'.
2020-10-14 11:22:47,108 - stpipe.Detector1Pipeline - INFO - Prefetch for SATURATION reference file is '/Users/patrickkavanagh/crds_mirror/references/jwst/miri/jwst_miri_saturation_0022.fits'.
2020-10-14 11:22:47,108 - stpipe.Detector1Pipeline - INFO - Prefetch for SUPERBIAS reference file is 'N/A'.
2020-10-14 11:22:47,109 - stpipe.Detector1Pipeline - INFO - Prefetch for TRAPDENSITY reference file is 'N/A'.
2020-10-14 11:22:47,109 - stpipe.Detector1Pipeline - INFO - Prefetch for TRAPPARS reference file is 'N/A'.
2020-10-14 11:22:47,110 - stpipe.Detector1Pipeline - INFO - Starting calwebb_detector1 ...
2020-10-14 11:22:47,464 - stpipe.Detector1Pipeline.group_scale - INFO - Step group_scale running with args (<RampModel(2, 20, 1024, 1032) from det_image_seq1_MIRIMAGE_F1130Wexp1.fits>,).
2020-10-14 11:22:47,465 - stpipe.Detector1Pipeline.group_scale - INFO - Step group_scale parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': 'IMA_simulation'}
2020-10-14 11:22:47,474 - stpipe.Detector1Pipeline.group_scale - INFO - NFRAMES=1 is a power of 2; correction not needed
2020-10-14 11:22:47,474 - stpipe.Detector1Pipeline.group_scale - INFO - Step will be skipped
2020-10-14 11:22:47,475 - stpipe.Detector1Pipeline.group_scale - INFO - Step group_scale done
2020-10-14 11:22:47,526 - stpipe.Detector1Pipeline.dq_init - INFO - Step dq_init running with args (<RampModel(2, 20, 1024, 1032) from det_image_seq1_MIRIMAGE_F1130Wexp1.fits>,).
2020-10-14 11:22:47,527 - stpipe.Detector1Pipeline.dq_init - INFO - Step dq_init parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': 'IMA_simulation'}
2020-10-14 11:22:47,538 - stpipe.Detector1Pipeline.dq_init - INFO - Using MASK reference file /Users/patrickkavanagh/crds_mirror/references/jwst/miri/jwst_miri_mask_0023.fits
2020-10-14 11:22:47,832 - stpipe.Detector1Pipeline.dq_init - INFO - Step dq_init done
2020-10-14 11:22:47,885 - stpipe.Detector1Pipeline.saturation - INFO - Step saturation running with args (<RampModel(2, 20, 1024, 1032) from det_image_seq1_MIRIMAGE_F1130Wexp1.fits>,).
2020-10-14 11:22:47,886 - stpipe.Detector1Pipeline.saturation - INFO - Step saturation parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': 'IMA_simulation'}
2020-10-14 11:22:47,897 - stpipe.Detector1Pipeline.saturation - INFO - Using SATURATION reference file /Users/patrickkavanagh/crds_mirror/references/jwst/miri/jwst_miri_saturation_0022.fits
2020-10-14 11:22:48,460 - stpipe.Detector1Pipeline.saturation - INFO - Step saturation done
2020-10-14 11:22:48,536 - stpipe.Detector1Pipeline.ipc - INFO - Step ipc running with args (<RampModel(2, 20, 1024, 1032) from det_image_seq1_MIRIMAGE_F1130Wexp1.fits>,).
2020-10-14 11:22:48,538 - stpipe.Detector1Pipeline.ipc - INFO - Step ipc parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': 'IMA_simulation'}
2020-10-14 11:22:48,538 - stpipe.Detector1Pipeline.ipc - INFO - Step skipped.
2020-10-14 11:22:48,539 - stpipe.Detector1Pipeline.ipc - INFO - Step ipc done
2020-10-14 11:22:48,599 - stpipe.Detector1Pipeline.firstframe - INFO - Step firstframe running with args (<RampModel(2, 20, 1024, 1032) from det_image_seq1_MIRIMAGE_F1130Wexp1.fits>,).
2020-10-14 11:22:48,600 - stpipe.Detector1Pipeline.firstframe - INFO - Step firstframe parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': 'IMA_simulation'}
2020-10-14 11:22:48,821 - stpipe.Detector1Pipeline.firstframe - INFO - Step firstframe done
2020-10-14 11:22:48,918 - stpipe.Detector1Pipeline.lastframe - INFO - Step lastframe running with args (<RampModel(2, 20, 1024, 1032) from det_image_seq1_MIRIMAGE_F1130Wexp1.fits>,).
2020-10-14 11:22:48,919 - stpipe.Detector1Pipeline.lastframe - INFO - Step lastframe parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': 'IMA_simulation'}
2020-10-14 11:22:49,138 - stpipe.Detector1Pipeline.lastframe - INFO - Step lastframe done
2020-10-14 11:22:49,213 - stpipe.Detector1Pipeline.linearity - INFO - Step linearity running with args (<RampModel(2, 20, 1024, 1032) from det_image_seq1_MIRIMAGE_F1130Wexp1.fits>,).
2020-10-14 11:22:49,214 - stpipe.Detector1Pipeline.linearity - INFO - Step linearity parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': 'IMA_simulation'}
2020-10-14 11:22:49,225 - stpipe.Detector1Pipeline.linearity - INFO - Using Linearity reference file /Users/patrickkavanagh/crds_mirror/references/jwst/miri/jwst_miri_linearity_0024.fits
2020-10-14 11:22:49,926 - stpipe.Detector1Pipeline.linearity - INFO - Step linearity done
2020-10-14 11:22:50,011 - stpipe.Detector1Pipeline.rscd - INFO - Step rscd running with args (<RampModel(2, 20, 1024, 1032) from det_image_seq1_MIRIMAGE_F1130Wexp1.fits>,).
2020-10-14 11:22:50,012 - stpipe.Detector1Pipeline.rscd - INFO - Step rscd parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': 'IMA_simulation', 'type': 'baseline'}
2020-10-14 11:22:50,024 - stpipe.Detector1Pipeline.rscd - INFO - Using RSCD reference file /Users/patrickkavanagh/crds_mirror/references/jwst/miri/jwst_miri_rscd_0010.fits
2020-10-14 11:22:50,281 - stpipe.Detector1Pipeline.rscd - INFO - Step rscd done
2020-10-14 11:22:50,376 - stpipe.Detector1Pipeline.dark_current - INFO - Step dark_current running with args (<RampModel(2, 20, 1024, 1032) from det_image_seq1_MIRIMAGE_F1130Wexp1.fits>,).
2020-10-14 11:22:50,377 - stpipe.Detector1Pipeline.dark_current - INFO - Step dark_current parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': 'IMA_demo_output', 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': 'IMA_simulation', 'dark_output': None}
2020-10-14 11:22:50,388 - stpipe.Detector1Pipeline.dark_current - INFO - Using DARK reference file /Users/patrickkavanagh/crds_mirror/references/jwst/miri/jwst_miri_dark_0049.fits
2020-10-14 11:22:51,858 - stpipe.Detector1Pipeline.dark_current - INFO - Science data nints=2, ngroups=20, nframes=1, groupgap=0
2020-10-14 11:22:51,859 - stpipe.Detector1Pipeline.dark_current - INFO - Dark data nints=2, ngroups=45, nframes=1, groupgap=0
2020-10-14 11:22:52,228 - stpipe.Detector1Pipeline.dark_current - INFO - Step dark_current done
2020-10-14 11:22:52,370 - stpipe.Detector1Pipeline.refpix - INFO - Step refpix running with args (<RampModel(2, 20, 1024, 1032) from det_image_seq1_MIRIMAGE_F1130Wexp1.fits>,).
2020-10-14 11:22:52,371 - stpipe.Detector1Pipeline.refpix - INFO - Step refpix parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': 'IMA_simulation', 'odd_even_columns': True, 'use_side_ref_pixels': True, 'side_smoothing_length': 11, 'side_gain': 1.0, 'odd_even_rows': True}
2020-10-14 11:22:52,372 - stpipe.Detector1Pipeline.refpix - INFO - Step skipped.
2020-10-14 11:22:52,373 - stpipe.Detector1Pipeline.refpix - INFO - Step refpix done
2020-10-14 11:22:52,424 - stpipe.Detector1Pipeline.jump - INFO - Step jump running with args (<RampModel(2, 20, 1024, 1032) from det_image_seq1_MIRIMAGE_F1130Wexp1.fits>,).
2020-10-14 11:22:52,425 - stpipe.Detector1Pipeline.jump - INFO - Step jump parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': 'IMA_simulation', 'rejection_threshold': 4.0, 'maximum_cores': 'none', 'flag_4_neighbors': True, 'max_jump_to_flag_neighbors': 200.0, 'min_jump_to_flag_neighbors': 10.0}
2020-10-14 11:22:52,433 - stpipe.Detector1Pipeline.jump - INFO - CR rejection threshold = 4 sigma
2020-10-14 11:22:52,437 - stpipe.Detector1Pipeline.jump - INFO - Using GAIN reference file: /Users/patrickkavanagh/crds_mirror/references/jwst/miri/jwst_miri_gain_0008.fits
2020-10-14 11:22:52,465 - stpipe.Detector1Pipeline.jump - INFO - Using READNOISE reference file: /Users/patrickkavanagh/crds_mirror/references/jwst/miri/jwst_miri_readnoise_0059.fits
2020-10-14 11:22:52,489 - stpipe.Detector1Pipeline.jump - INFO - Found 8 possible cores to use for jump detection
2020-10-14 11:22:52,774 - stpipe.Detector1Pipeline.jump - INFO - Executing two-point difference method
2020-10-14 11:22:53,404 - stpipe.Detector1Pipeline.jump - INFO - working on integration 1
2020-10-14 11:22:54,589 - stpipe.Detector1Pipeline.jump - INFO - From highest outlier Two point found 30960 pixels with at least one CR
2020-10-14 11:22:55,565 - stpipe.Detector1Pipeline.jump - INFO - working on integration 2
2020-10-14 11:22:56,639 - stpipe.Detector1Pipeline.jump - INFO - From highest outlier Two point found 30142 pixels with at least one CR
2020-10-14 11:22:58,741 - stpipe.Detector1Pipeline.jump - INFO - Total elapsed time = 5.96583 sec
2020-10-14 11:22:58,758 - stpipe.Detector1Pipeline.jump - INFO - The execution time in seconds: 6.324742
2020-10-14 11:22:58,760 - stpipe.Detector1Pipeline.jump - INFO - Step jump done
2020-10-14 11:22:58,853 - stpipe.Detector1Pipeline.ramp_fit - INFO - Step ramp_fit running with args (<RampModel(2, 20, 1024, 1032) from det_image_seq1_MIRIMAGE_F1130Wexp1.fits>,).
2020-10-14 11:22:58,854 - stpipe.Detector1Pipeline.ramp_fit - INFO - Step ramp_fit parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': 'IMA_demo_output', 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': 'IMA_simulation', 'int_name': '', 'save_opt': False, 'opt_name': '', 'maximum_cores': 'none'}
2020-10-14 11:22:58,869 - stpipe.Detector1Pipeline.ramp_fit - INFO - Using READNOISE reference file: /Users/patrickkavanagh/crds_mirror/references/jwst/miri/jwst_miri_readnoise_0059.fits
2020-10-14 11:22:58,880 - stpipe.Detector1Pipeline.ramp_fit - INFO - Using GAIN reference file: /Users/patrickkavanagh/crds_mirror/references/jwst/miri/jwst_miri_gain_0008.fits
2020-10-14 11:22:58,891 - stpipe.Detector1Pipeline.ramp_fit - INFO - Using algorithm = ols
2020-10-14 11:22:58,892 - stpipe.Detector1Pipeline.ramp_fit - INFO - Using weighting = optimal
2020-10-14 11:22:59,021 - stpipe.Detector1Pipeline.ramp_fit - INFO - Number of leading groups that are flagged as DO_NOT_USE: 1
2020-10-14 11:22:59,022 - stpipe.Detector1Pipeline.ramp_fit - INFO - MIRI dataset has all pixels in the final group flagged as DO_NOT_USE.
2020-10-14 11:23:35,560 - stpipe.Detector1Pipeline.ramp_fit - INFO - Number of groups per integration: 20
2020-10-14 11:23:35,561 - stpipe.Detector1Pipeline.ramp_fit - INFO - Number of integrations: 2
2020-10-14 11:23:35,647 - stpipe.Detector1Pipeline.ramp_fit - INFO - Step ramp_fit done
2020-10-14 11:23:35,751 - stpipe.Detector1Pipeline.gain_scale - INFO - Step gain_scale running with args (<ImageModel(1024, 1032) from det_image_seq1_MIRIMAGE_F1130Wexp1.fits>,).
2020-10-14 11:23:35,752 - stpipe.Detector1Pipeline.gain_scale - INFO - Step gain_scale parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': 'gain_scale', 'search_output_file': True, 'input_dir': 'IMA_simulation'}
2020-10-14 11:23:35,771 - stpipe.Detector1Pipeline.gain_scale - INFO - GAINFACT not found in gain reference file
2020-10-14 11:23:35,772 - stpipe.Detector1Pipeline.gain_scale - INFO - Step will be skipped
2020-10-14 11:23:35,773 - stpipe.Detector1Pipeline.gain_scale - INFO - Step gain_scale done
2020-10-14 11:23:35,828 - stpipe.Detector1Pipeline.gain_scale - INFO - Step gain_scale running with args (<CubeModel(2, 1024, 1032) from det_image_seq1_MIRIMAGE_F1130Wexp1.fits>,).
2020-10-14 11:23:35,829 - stpipe.Detector1Pipeline.gain_scale - INFO - Step gain_scale parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': 'gain_scaleints', 'search_output_file': True, 'input_dir': 'IMA_simulation'}
2020-10-14 11:23:35,850 - stpipe.Detector1Pipeline.gain_scale - INFO - GAINFACT not found in gain reference file
2020-10-14 11:23:35,850 - stpipe.Detector1Pipeline.gain_scale - INFO - Step will be skipped
2020-10-14 11:23:35,852 - stpipe.Detector1Pipeline.gain_scale - INFO - Step gain_scale done
2020-10-14 11:23:36,157 - stpipe.Detector1Pipeline - INFO - Saved model in IMA_demo_output/det_image_seq1_MIRIMAGE_F1130Wexp1_rateints.fits
2020-10-14 11:23:36,158 - stpipe.Detector1Pipeline - INFO - ... ending calwebb_detector1
2020-10-14 11:23:36,324 - stpipe.Detector1Pipeline - INFO - Saved model in IMA_demo_output/det_image_seq1_MIRIMAGE_F1130Wexp1_rate.fits
2020-10-14 11:23:36,325 - stpipe.Detector1Pipeline - INFO - Step Detector1Pipeline done
We can plot the before (last frame of first integration) and after images
# open the input image as a jwst data model
with datamodels.open(my_input_file) as in_dm:
fig, axs = plt.subplots(1, 2, figsize=(14, 7), sharey=True)
axs[0].imshow(in_dm.data[0,-1,:,:], cmap='jet', interpolation='nearest', origin='lower', norm=LogNorm(vmin=1e4,vmax=6e4))
axs[0].annotate('DMS Level 1B (ramp)', xy=(0.0, 1.02), xycoords='axes fraction', fontsize=12, fontweight='bold', color='k')
axs[0].set_facecolor('black')
axs[1].imshow(dm.data, cmap='jet', interpolation='nearest', origin='lower', norm=LogNorm(vmin=10, vmax=100))
axs[1].annotate('DMS Level 2A (slope)', xy=(0.0, 1.02), xycoords='axes fraction', fontsize=12, fontweight='bold', color='k')
axs[1].set_facecolor('black')
plt.tight_layout()
plt.show()
Command line¶
To achieve the same result from the command line there are a couple of options.
Option 1:
Run the Detector1Pipeline class using the strun command:
mkdir demo_output
strun jwst.pipeline.Detector1Pipeline det_image_seq1_MIRIMAGE_F1130Wexp1.fits --output_dir demo_output
This will produce the same output file in the user-defined --output_dir
Option 2:
Collect the pipeline configuration files in your working directory using collect_pipeline_configs and then run the Detector1Pipeline using the strun command with the associated calwebb_detector1.cfg file. This option is a little more flexible as one can create edit the cfg files, use them again, etc.
mkdir demo_output
collect_pipeline_cfgs cfgs/
strun cfgs/calwebb_detector1.cfg det_image_seq1_MIRIMAGE_F1130Wexp1.fits --output_dir demo_output
This will produce the same output file in the user-defined --output_dir