Image3Pipeline (MIRI CALIMAGE3)¶
Image3Pipeline combines the calibrated data from multiple exposures (e.g. a dither or mosaic pattern) into a single rectified (distortion corrected) product. Before being combined, the exposures receive additional corrections for the purpose of astrometric alignment, background matching, and outlier rejection.
Official documentation for Image3Pipeline can be found here:
https://jwst-pipeline.readthedocs.io/en/latest/jwst/pipeline/calwebb_image3.html
The Image3Pipeline comprises a linear series of steps. The steps applied to MIRI data in order are:
Step |
Description |
|---|---|
|
refine the WCS solution by comparing source catalogues from input images |
|
compute sky values in a collection of input images that contain both sky and source signal |
|
identification of bad pixels or cosmic-rays that remain in each of the input images |
|
resample using WCS and distortion information into an undistorted product |
|
resample using WCS and distortion information into an undistorted product |
Input data¶
An example of running the file through the Image3Pipeline is now shown using a simple simulated 4-point dither observation of a galaxy with the MIRI Imager (F1130W filter) produced with MIRISim v2.3 and brought to level 2B using the Detector1Pipeline and Image2Pipeline modules.
Python¶
Start by importing what will be used and set the CRDS_CONTEXT
# imports
import os, glob, shutil
import numpy as np
from matplotlib.colors import LogNorm
import matplotlib.pyplot as plt
from subprocess import call
from jwst import datamodels
from jwst.associations.lib.member import Member
from jwst.associations.asn_from_list import asn_from_list
from astropy.table import Table
from photutils import CircularAperture
# set the CRDS_CONTEXT
os.environ["CRDS_CONTEXT"] = "jwst_0641.pmap"
Image3Pipeline requires an association file as input that lists the exposures to be combined. The content of the association file is printed for inspection. It is assumed the files are in ‘IMA_science’
# load the science and background files to lists
my_science_files = glob.glob('IMA_science/*cal.fits')
my_science_files = sorted(my_science_files)
# set the association name
asn_name = 'my_galaxy'
# create an association
asn = asn_from_list(my_science_files, product_name=asn_name)
# set some metadata
asn['asn_pool'] = asn_name + '_pool'
asn['asn_type'] = 'image3'
# print the association and save to file
name, ser = asn.dump()
print(ser)
asn_file = asn_name + '_lvl3_asn.json'
with open(asn_file, 'w') as f:
f.write(ser)
{
"asn_type": "image3",
"asn_rule": "DMS_Level3_Base",
"version_id": null,
"code_version": "0.17.1",
"degraded_status": "No known degraded exposures in association.",
"program": "noprogram",
"constraints": "No constraints",
"asn_id": "a3001",
"target": "none",
"asn_pool": "my_galaxy_pool",
"products": [
{
"name": "my_galaxy",
"members": [
{
"expname": "IMA_science/my_galaxy_dither1_cal.fits",
"exptype": "science"
},
{
"expname": "IMA_science/my_galaxy_dither2_cal.fits",
"exptype": "science"
},
{
"expname": "IMA_science/my_galaxy_dither3_cal.fits",
"exptype": "science"
},
{
"expname": "IMA_science/my_galaxy_dither4_cal.fits",
"exptype": "science"
}
]
}
]
}
Import Image2Pipeline and print the docstring to show some information
from jwst.pipeline import Image3Pipeline
print(Image3Pipeline.__doc__)
Image3Pipeline: Applies level 3 processing to imaging-mode data from
any JWST instrument.
Included steps are:
assign_mtwcs
tweakreg
skymatch
outlier_detection
resample
source_catalog
The association file is passed as input. The output level 2B files will be saved in my_output_dir as _cal.fits.
Parameters used:
save_results: boolean, optional, default=False
save the results to file
output_dir : boolean, optional, default is the working directory
the location to save the output
There are still some issues with the tweakreg step so this is skipped for now. We also set the source detection parameters and explicitly save the output from the source_catalog step.
# user specified
my_output_dir = 'demo_output'
# the output directory should be created if it doesn't exist
if not os.path.exists(my_output_dir):
os.mkdir(my_output_dir)
# run the pipeline
Image3Pipeline.call(asn_file, save_results=True, output_dir=my_output_dir,
steps={'skymatch':{'skip':True},
'source_catalog':{'save_results':True, 'kernel_fwhm': 3.0, 'snr_threshold': 5.}})
2020-10-14 14:34:49,784 - stpipe - WARNING - /Users/patrickkavanagh/anaconda3/anaconda3/envs/jwst7.6/lib/python3.8/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.
and should_run_async(code)
2020-10-14 14:34:50,117 - CRDS - ERROR - Error determining best reference for 'pars-assignmtwcsstep' = Unknown reference type 'pars-assignmtwcsstep'
2020-10-14 14:34:50,264 - CRDS - ERROR - Error determining best reference for 'pars-tweakregstep' = Unknown reference type 'pars-tweakregstep'
2020-10-14 14:34:50,534 - CRDS - ERROR - Error determining best reference for 'pars-skymatchstep' = Unknown reference type 'pars-skymatchstep'
2020-10-14 14:34:50,687 - CRDS - ERROR - Error determining best reference for 'pars-outlierdetectionstep' = Unknown reference type 'pars-outlierdetectionstep'
2020-10-14 14:34:50,844 - CRDS - ERROR - Error determining best reference for 'pars-resamplestep' = Unknown reference type 'pars-resamplestep'
2020-10-14 14:34:51,008 - CRDS - ERROR - Error determining best reference for 'pars-sourcecatalogstep' = Unknown reference type 'pars-sourcecatalogstep'
2020-10-14 14:34:51,168 - CRDS - ERROR - Error determining best reference for 'pars-image3pipeline' = Unknown reference type 'pars-image3pipeline'
2020-10-14 14:34:51,174 - stpipe.Image3Pipeline - INFO - Image3Pipeline instance created.
2020-10-14 14:34:51,176 - stpipe.Image3Pipeline.assign_mtwcs - INFO - AssignMTWcsStep instance created.
2020-10-14 14:34:51,178 - stpipe.Image3Pipeline.tweakreg - INFO - TweakRegStep instance created.
2020-10-14 14:34:51,180 - stpipe.Image3Pipeline.skymatch - INFO - SkyMatchStep instance created.
2020-10-14 14:34:51,183 - stpipe.Image3Pipeline.outlier_detection - INFO - OutlierDetectionStep instance created.
2020-10-14 14:34:51,185 - stpipe.Image3Pipeline.resample - INFO - ResampleStep instance created.
2020-10-14 14:34:51,187 - stpipe.Image3Pipeline.source_catalog - INFO - SourceCatalogStep instance created.
2020-10-14 14:34:51,270 - stpipe.Image3Pipeline - INFO - Step Image3Pipeline running with args ('my_galaxy_lvl3_asn.json',).
2020-10-14 14:34:51,276 - stpipe.Image3Pipeline - INFO - Step Image3Pipeline parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': 'demo_output', 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'steps': {'assign_mtwcs': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': True, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': 'assign_mtwcs', 'search_output_file': True, 'input_dir': ''}, 'tweakreg': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_catalogs': False, 'catalog_format': 'ecsv', 'kernel_fwhm': 2.5, 'snr_threshold': 10.0, 'brightest': 1000, 'peakmax': None, 'enforce_user_order': False, 'expand_refcat': False, 'minobj': 15, 'searchrad': 1.0, 'use2dhist': True, 'separation': 0.5, 'tolerance': 1.0, 'xoffset': 0.0, 'yoffset': 0.0, 'fitgeometry': 'general', 'nclip': 3, 'sigma': 3.0, 'align_to_gaia': False, 'gaia_catalog': 'GAIADR2', 'min_gaia': 5, 'save_gaia_catalog': False}, 'skymatch': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'skymethod': 'global+match', 'match_down': True, 'subtract': False, 'stepsize': None, 'skystat': 'mode', 'dqbits': '0', 'lower': None, 'upper': None, 'nclip': 5, 'lsigma': 4.0, 'usigma': 4.0, 'binwidth': 0.1}, 'outlier_detection': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': False, 'input_dir': '', 'weight_type': 'exptime', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'nlow': 0, 'nhigh': 0, 'maskpt': 0.7, 'grow': 1, 'snr': '4.0 3.0', 'scale': '0.5 0.4', 'backg': 0.0, 'save_intermediate_results': False, 'resample_data': True, 'good_bits': '~DO_NOT_USE', 'scale_detection': False}, 'resample': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'weight_type': 'exptime', 'single': False, 'blendheaders': True}, 'source_catalog': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': 'cat', 'search_output_file': True, 'input_dir': '', 'bkg_boxsize': 100, 'kernel_fwhm': 3.0, 'snr_threshold': 5.0, 'npixels': 5, 'deblend': False, 'aperture_ee1': 30, 'aperture_ee2': 50, 'aperture_ee3': 70, 'ci1_star_threshold': 2.0, 'ci2_star_threshold': 1.8}}}
2020-10-14 14:34:51,451 - stpipe.Image3Pipeline - INFO - Prefetching reference files for dataset: 'my_galaxy_dither1_cal.fits' reftypes = ['abvegaoffset', 'apcorr', 'drizpars']
2020-10-14 14:34:51,454 - stpipe.Image3Pipeline - INFO - Prefetch for ABVEGAOFFSET reference file is '/Users/patrickkavanagh/crds_mirror/references/jwst/miri/jwst_miri_abvegaoffset_0001.asdf'.
2020-10-14 14:34:51,455 - stpipe.Image3Pipeline - INFO - Prefetch for APCORR reference file is '/Users/patrickkavanagh/crds_mirror/references/jwst/miri/jwst_miri_apcorr_0005.fits'.
2020-10-14 14:34:51,456 - stpipe.Image3Pipeline - INFO - Prefetch for DRIZPARS reference file is '/Users/patrickkavanagh/crds_mirror/references/jwst/miri/jwst_miri_drizpars_0001.fits'.
2020-10-14 14:34:51,457 - stpipe.Image3Pipeline - INFO - Starting calwebb_image3 ...
2020-10-14 14:34:52,170 - stpipe.Image3Pipeline.tweakreg - INFO - Step tweakreg running with args (<ModelContainer>,).
2020-10-14 14:34:52,172 - stpipe.Image3Pipeline.tweakreg - INFO - Step tweakreg parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_catalogs': False, 'catalog_format': 'ecsv', 'kernel_fwhm': 2.5, 'snr_threshold': 10.0, 'brightest': 1000, 'peakmax': None, 'enforce_user_order': False, 'expand_refcat': False, 'minobj': 15, 'searchrad': 1.0, 'use2dhist': True, 'separation': 0.5, 'tolerance': 1.0, 'xoffset': 0.0, 'yoffset': 0.0, 'fitgeometry': 'general', 'nclip': 3, 'sigma': 3.0, 'align_to_gaia': False, 'gaia_catalog': 'GAIADR2', 'min_gaia': 5, 'save_gaia_catalog': False}
2020-10-14 14:34:52,514 - stpipe.Image3Pipeline.tweakreg - WARNING - /Users/patrickkavanagh/anaconda3/anaconda3/envs/jwst7.6/lib/python3.8/site-packages/photutils/detection/findstars.py:971: NoDetectionsWarning: Sources were found, but none pass the sharpness and roundness criteria.
warnings.warn('Sources were found, but none pass the sharpness '
2020-10-14 14:34:52,517 - stpipe.Image3Pipeline.tweakreg - WARNING - No sources found in my_galaxy_dither1_cal.fits.
2020-10-14 14:34:52,885 - stpipe.Image3Pipeline.tweakreg - INFO - Detected 1 sources in my_galaxy_dither2_cal.fits.
2020-10-14 14:34:53,319 - stpipe.Image3Pipeline.tweakreg - INFO - Detected 1 sources in my_galaxy_dither3_cal.fits.
2020-10-14 14:34:53,653 - stpipe.Image3Pipeline.tweakreg - WARNING - /Users/patrickkavanagh/anaconda3/anaconda3/envs/jwst7.6/lib/python3.8/site-packages/photutils/detection/findstars.py:940: NoDetectionsWarning: No sources were found.
warnings.warn('No sources were found.', NoDetectionsWarning)
2020-10-14 14:34:53,655 - stpipe.Image3Pipeline.tweakreg - WARNING - No sources found in my_galaxy_dither4_cal.fits.
2020-10-14 14:34:53,670 - stpipe.Image3Pipeline.tweakreg - INFO -
2020-10-14 14:34:53,670 - stpipe.Image3Pipeline.tweakreg - INFO - Number of image groups to be aligned: 4.
2020-10-14 14:34:53,671 - stpipe.Image3Pipeline.tweakreg - INFO - Image groups:
2020-10-14 14:34:53,719 - stpipe.Image3Pipeline.tweakreg - WARNING - /Users/patrickkavanagh/anaconda3/anaconda3/envs/jwst7.6/lib/python3.8/site-packages/gwcs/wcs.py:973: DeprecationWarning: Indexing a WCS.pipeline step is deprecated. Use the `frame` and `transform` attributes instead.
warnings.warn("Indexing a WCS.pipeline step is deprecated. "
2020-10-14 14:34:53,719 - stpipe.Image3Pipeline.tweakreg - INFO - * Images in GROUP 'my_galaxy_dither1_cal':
2020-10-14 14:34:53,720 - stpipe.Image3Pipeline.tweakreg - INFO - my_galaxy_dither1_cal
2020-10-14 14:34:53,761 - stpipe.Image3Pipeline.tweakreg - INFO - * Images in GROUP 'my_galaxy_dither2_cal':
2020-10-14 14:34:53,762 - stpipe.Image3Pipeline.tweakreg - INFO - my_galaxy_dither2_cal
2020-10-14 14:34:53,804 - stpipe.Image3Pipeline.tweakreg - INFO - * Images in GROUP 'my_galaxy_dither3_cal':
2020-10-14 14:34:53,804 - stpipe.Image3Pipeline.tweakreg - INFO - my_galaxy_dither3_cal
2020-10-14 14:34:53,845 - stpipe.Image3Pipeline.tweakreg - INFO - * Images in GROUP 'my_galaxy_dither4_cal':
2020-10-14 14:34:53,845 - stpipe.Image3Pipeline.tweakreg - INFO - my_galaxy_dither4_cal
2020-10-14 14:34:53,846 - stpipe.Image3Pipeline.tweakreg - INFO -
2020-10-14 14:34:53,846 - stpipe.Image3Pipeline.tweakreg - INFO -
2020-10-14 14:34:53,847 - stpipe.Image3Pipeline.tweakreg - INFO - ***** tweakwcs.imalign.align_wcs() started on 2020-10-14 14:34:53.846778
2020-10-14 14:34:53,848 - stpipe.Image3Pipeline.tweakreg - INFO - Version 0.6.4
2020-10-14 14:34:53,848 - stpipe.Image3Pipeline.tweakreg - INFO -
2020-10-14 14:34:53,872 - stpipe.Image3Pipeline.tweakreg - WARNING - Group with ID 'my_galaxy_dither1_cal' will not be aligned: empty source catalog
2020-10-14 14:34:53,892 - stpipe.Image3Pipeline.tweakreg - WARNING - Group with ID 'my_galaxy_dither4_cal' will not be aligned: empty source catalog
2020-10-14 14:34:53,892 - stpipe.Image3Pipeline.tweakreg - INFO - Selected image 'GROUP ID: my_galaxy_dither2_cal' as reference image
2020-10-14 14:34:53,897 - stpipe.Image3Pipeline.tweakreg - INFO - Aligning image catalog 'GROUP ID: my_galaxy_dither3_cal' to the reference catalog.
2020-10-14 14:34:54,024 - stpipe.Image3Pipeline.tweakreg - INFO - Matching sources from 'my_galaxy_dither3_cal' catalog with sources from the reference 'my_galaxy_dither2_cal' catalog.
2020-10-14 14:34:54,025 - stpipe.Image3Pipeline.tweakreg - INFO - Computing initial guess for X and Y shifts...
2020-10-14 14:34:54,026 - stpipe.Image3Pipeline.tweakreg - INFO - Found initial X and Y shifts of 0, 0 based on a single non-zero bin and 1 matches
2020-10-14 14:34:54,028 - stpipe.Image3Pipeline.tweakreg - INFO - Found 1 matches for 'GROUP ID: my_galaxy_dither3_cal'...
2020-10-14 14:34:54,029 - stpipe.Image3Pipeline.tweakreg - WARNING - Not enough matches (< 1) found for image catalog 'GROUP ID: my_galaxy_dither3_cal'.
2020-10-14 14:34:54,029 - stpipe.Image3Pipeline.tweakreg - INFO -
2020-10-14 14:34:54,030 - stpipe.Image3Pipeline.tweakreg - INFO - ***** tweakwcs.imalign.align_wcs() ended on 2020-10-14 14:34:54.029592
2020-10-14 14:34:54,030 - stpipe.Image3Pipeline.tweakreg - INFO - ***** tweakwcs.imalign.align_wcs() TOTAL RUN TIME: 0:00:00.182814
2020-10-14 14:34:54,031 - stpipe.Image3Pipeline.tweakreg - INFO -
2020-10-14 14:34:54,034 - stpipe.Image3Pipeline.tweakreg - INFO - Step tweakreg done
2020-10-14 14:34:54,129 - stpipe.Image3Pipeline.skymatch - INFO - Step skymatch running with args (<ModelContainer>,).
2020-10-14 14:34:54,131 - stpipe.Image3Pipeline.skymatch - INFO - Step skymatch parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'skymethod': 'global+match', 'match_down': True, 'subtract': False, 'stepsize': None, 'skystat': 'mode', 'dqbits': '0', 'lower': None, 'upper': None, 'nclip': 5, 'lsigma': 4.0, 'usigma': 4.0, 'binwidth': 0.1}
2020-10-14 14:34:54,132 - stpipe.Image3Pipeline.skymatch - INFO - Step skipped.
2020-10-14 14:34:54,134 - stpipe.Image3Pipeline.skymatch - INFO - Step skymatch done
2020-10-14 14:34:54,201 - stpipe.Image3Pipeline.outlier_detection - INFO - Step outlier_detection running with args (<ModelContainer>,).
2020-10-14 14:34:54,203 - stpipe.Image3Pipeline.outlier_detection - INFO - Step outlier_detection parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': 'crf', 'search_output_file': False, 'input_dir': '', 'weight_type': 'exptime', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'nlow': 0, 'nhigh': 0, 'maskpt': 0.7, 'grow': 1, 'snr': '4.0 3.0', 'scale': '0.5 0.4', 'backg': 0.0, 'save_intermediate_results': False, 'resample_data': True, 'good_bits': '~DO_NOT_USE', 'scale_detection': False}
2020-10-14 14:34:54,206 - stpipe.Image3Pipeline.outlier_detection - INFO - Performing outlier detection on 4 inputs
2020-10-14 14:34:54,835 - stpipe.Image3Pipeline.outlier_detection - INFO - Drizzling (1024, 1032) --> (1142, 1111)
2020-10-14 14:34:55,918 - stpipe.Image3Pipeline.outlier_detection - INFO - Drizzling (1024, 1032) --> (1142, 1111)
2020-10-14 14:34:56,954 - stpipe.Image3Pipeline.outlier_detection - INFO - Drizzling (1024, 1032) --> (1142, 1111)
2020-10-14 14:34:58,041 - stpipe.Image3Pipeline.outlier_detection - INFO - Drizzling (1024, 1032) --> (1142, 1111)
2020-10-14 14:34:58,893 - stpipe.Image3Pipeline.outlier_detection - INFO - Generating median from 4 images
2020-10-14 14:34:59,159 - stpipe.Image3Pipeline.outlier_detection - WARNING - /Users/patrickkavanagh/anaconda3/anaconda3/envs/jwst7.6/lib/python3.8/site-packages/numpy/lib/nanfunctions.py:1113: RuntimeWarning: All-NaN slice encountered
r, k = function_base._ureduce(a, func=_nanmedian, axis=axis, out=out,
2020-10-14 14:34:59,244 - stpipe.Image3Pipeline.outlier_detection - INFO - Blotting median...
2020-10-14 14:34:59,800 - stpipe.Image3Pipeline.outlier_detection - INFO - Blotting (1024, 1032) <-- (1142, 1111)
2020-10-14 14:35:00,411 - stpipe.Image3Pipeline.outlier_detection - INFO - Blotting (1024, 1032) <-- (1142, 1111)
2020-10-14 14:35:01,024 - stpipe.Image3Pipeline.outlier_detection - INFO - Blotting (1024, 1032) <-- (1142, 1111)
2020-10-14 14:35:01,646 - stpipe.Image3Pipeline.outlier_detection - INFO - Blotting (1024, 1032) <-- (1142, 1111)
2020-10-14 14:35:02,020 - stpipe.Image3Pipeline.outlier_detection - WARNING - /Users/patrickkavanagh/anaconda3/anaconda3/envs/jwst7.6/lib/python3.8/site-packages/jwst/outlier_detection/outlier_detection.py:424: RuntimeWarning: overflow encountered in square
ta = np.sqrt(np.abs(blot_data + subtracted_background) + err_data ** 2)
2020-10-14 14:35:03,018 - stpipe.Image3Pipeline.outlier_detection - INFO - Saved model in demo_output/my_galaxy_dither1_a3001_crf.fits
2020-10-14 14:35:03,431 - stpipe.Image3Pipeline.outlier_detection - INFO - Saved model in demo_output/my_galaxy_dither2_a3001_crf.fits
2020-10-14 14:35:03,847 - stpipe.Image3Pipeline.outlier_detection - INFO - Saved model in demo_output/my_galaxy_dither3_a3001_crf.fits
2020-10-14 14:35:04,280 - stpipe.Image3Pipeline.outlier_detection - INFO - Saved model in demo_output/my_galaxy_dither4_a3001_crf.fits
2020-10-14 14:35:04,281 - stpipe.Image3Pipeline.outlier_detection - INFO - Step outlier_detection done
2020-10-14 14:35:04,531 - stpipe.Image3Pipeline.resample - INFO - Step resample running with args (<ModelContainer>,).
2020-10-14 14:35:04,533 - stpipe.Image3Pipeline.resample - INFO - Step resample parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': 'i2d', 'search_output_file': True, 'input_dir': '', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'weight_type': 'exptime', 'single': False, 'blendheaders': True}
2020-10-14 14:35:04,538 - stpipe.Image3Pipeline.resample - INFO - Drizpars reference file: /Users/patrickkavanagh/crds_mirror/references/jwst/miri/jwst_miri_drizpars_0001.fits
2020-10-14 14:35:04,647 - stpipe.Image3Pipeline.resample - INFO - Blending metadata for my_galaxy
2020-10-14 14:35:06,026 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1111)
2020-10-14 14:35:07,078 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1111)
2020-10-14 14:35:08,165 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1111)
2020-10-14 14:35:09,313 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1111)
2020-10-14 14:35:09,880 - stpipe.Image3Pipeline.resample - INFO - Update S_REGION to POLYGON ICRS 0.021513048 -0.020377659 0.024586201 0.014641639 359.990548179 0.017628679 359.987475025 -0.017390618
2020-10-14 14:35:10,653 - stpipe.Image3Pipeline.resample - INFO - Saved model in demo_output/my_galaxy_i2d.fits
2020-10-14 14:35:10,654 - stpipe.Image3Pipeline.resample - INFO - Step resample done
2020-10-14 14:35:10,719 - stpipe.Image3Pipeline.source_catalog - INFO - Step source_catalog running with args (<ImageModel(1142, 1111) from my_galaxy_i2d.fits>,).
2020-10-14 14:35:10,720 - stpipe.Image3Pipeline.source_catalog - INFO - Step source_catalog parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': 'cat', 'search_output_file': True, 'input_dir': '', 'bkg_boxsize': 100, 'kernel_fwhm': 3.0, 'snr_threshold': 5.0, 'npixels': 5, 'deblend': False, 'aperture_ee1': 30, 'aperture_ee2': 50, 'aperture_ee3': 70, 'ci1_star_threshold': 2.0, 'ci2_star_threshold': 1.8}
2020-10-14 14:35:10,730 - stpipe.Image3Pipeline.source_catalog - INFO - Using APCORR reference file /Users/patrickkavanagh/crds_mirror/references/jwst/miri/jwst_miri_apcorr_0005.fits
2020-10-14 14:35:10,732 - stpipe.Image3Pipeline.source_catalog - INFO - Using ABVEGAOFFSET reference file /Users/patrickkavanagh/crds_mirror/references/jwst/miri/jwst_miri_abvegaoffset_0001.asdf
2020-10-14 14:35:10,733 - stpipe.Image3Pipeline.source_catalog - INFO - Instrument: MIRI
2020-10-14 14:35:10,733 - stpipe.Image3Pipeline.source_catalog - INFO - Detector: MIRIMAGE
2020-10-14 14:35:10,734 - stpipe.Image3Pipeline.source_catalog - INFO - Filter: F1130W
2020-10-14 14:35:10,734 - stpipe.Image3Pipeline.source_catalog - INFO - Subarray: FULL
2020-10-14 14:35:10,774 - stpipe.Image3Pipeline.source_catalog - INFO - AB to Vega magnitude offset 5.49349
2020-10-14 14:35:12,686 - stpipe.Image3Pipeline.source_catalog - INFO - Detected 47 sources
2020-10-14 14:35:13,408 - stpipe.Image3Pipeline.source_catalog - WARNING - /Users/patrickkavanagh/anaconda3/anaconda3/envs/jwst7.6/lib/python3.8/site-packages/astropy/io/misc/yaml.py:144: DeprecationWarning: tostring() is deprecated. Use tobytes() instead.
data_b64 = base64.b64encode(obj.tostring())
2020-10-14 14:35:13,458 - stpipe.Image3Pipeline.source_catalog - INFO - Wrote source catalog: demo_output/my_galaxy_cat.ecsv
2020-10-14 14:35:13,466 - stpipe.Image3Pipeline.source_catalog - INFO - Step source_catalog done
2020-10-14 14:35:13,469 - stpipe.Image3Pipeline - INFO - Step Image3Pipeline done
We can plot the before (rate) and after (cal) images for the first dither position
driz_dm = datamodels.open(os.path.join(my_output_dir,'my_galaxy_i2d.fits'))
# plot--------------------------------------
fig, axs = plt.subplots(1, 1, figsize=(8, 8))
# show last frame of first integration
axs.imshow(driz_dm.data, cmap='jet', interpolation='nearest', origin='lower', norm=LogNorm(vmin=10,vmax=500))
axs.annotate('Drizzled image', xy=(0.0, 1.02), xycoords='axes fraction', fontsize=12, fontweight='bold', color='k')
axs.set_facecolor('black')
plt.tight_layout()
plt.show()
2020-10-14 14:40:37,490 - stpipe - WARNING - /Users/patrickkavanagh/anaconda3/anaconda3/envs/jwst7.6/lib/python3.8/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.
and should_run_async(code)
my_catalog = os.path.join(my_output_dir,'my_galaxy_cat.ecsv')
sources = Table.read(my_catalog, format='ascii.ecsv')
print(sources.columns) # print all available column names
print()
sources['id', 'xcentroid', 'ycentroid', 'sky_centroid', 'aper30_flux'].pprint(max_width=200)
<TableColumns names=('id','xcentroid','ycentroid','sky_centroid','aper_bkg_flux','aper_bkg_flux_err','aper30_flux','aper30_flux_err','aper50_flux','aper50_flux_err','aper70_flux','aper70_flux_err','aper_total_flux','aper_total_flux_err','aper30_abmag','aper30_abmag_err','aper50_abmag','aper50_abmag_err','aper70_abmag','aper70_abmag_err','aper_total_abmag','aper_total_abmag_err','aper30_vegamag','aper30_vegamag_err','aper50_vegamag','aper50_vegamag_err','aper70_vegamag','aper70_vegamag_err','aper_total_vegamag','aper_total_vegamag_err','CI_30_50','CI_50_70','CI_30_70','is_star','sharpness','roundness','nn_dist','nn_abmag','isophotal_flux','isophotal_flux_err','isophotal_abmag','isophotal_abmag_err','isophotal_vegamag','isophotal_vegamag_err','isophotal_area','semimajor_sigma','semiminor_sigma','ellipticity','orientation','sky_orientation','sky_bbox_ll','sky_bbox_ul','sky_bbox_lr','sky_bbox_ur')>
id xcentroid ycentroid sky_centroid aper30_flux
pix pix deg,deg Jy
--- --------- --------- ------------------------------------------- -------------
1 469.9220 54.2437 0.007248914722059638,-0.01744971065714288 1.295216e-04
2 696.2326 80.7708 0.00038051187933409224,-0.0160272516694967 1.442850e-04
3 591.1478 190.5680 0.0038983912527062363,-0.012943117841751456 1.290441e-04
4 738.6110 230.5144 359.9994839469423,-0.011321339783567972 1.014409e-04
5 695.9396 288.1143 0.000947463105526589,-0.009669874720335625 1.277817e-04
6 844.8914 326.5756 359.9964833750726,-0.008089629301227494 9.848953e-05
7 940.8762 435.2375 359.9938324240641,-0.0044992275787256 1.065251e-04
8 351.3610 484.1433 0.012041445537664171,-0.004585939549639497 1.219468e-04
9 581.3696 592.3515 0.005279450326242856,-0.0006487877277036434 1.193683e-04
10 1072.0149 590.2738 359.990228278093,0.0006078406314121922 9.908035e-05
... ... ... ... ...
38 992.9896 1020.0000 359.9938079836952,0.013572685360032446 -2.050312e-05
39 1071.1200 1062.8683 359.9915274827764,0.015097488347460787 6.945693e-05
40 1044.3592 1067.4549 359.9923604446804,0.015166120328598723 -8.578647e-05
41 1036.2449 1077.6618 359.9926367354973,0.015457277527749147 -9.771649e-05
42 1029.9704 1079.9556 359.99283531443257,0.01551073282752823 1.875869e-05
43 199.5609 1089.4041 0.01832515355077436,0.013565815141783798 1.590341e-04
44 1037.1692 1098.0000 359.9926631224109,0.016083434575418797 -2.417245e-05
45 774.1114 1130.9421 0.0008184054050327275,0.01638570381518864 1.341135e-04
46 500.7521 1136.2409 0.009215198170058501,0.015812576058425707 1.469640e-04
47 798.2018 1138.8243 0.00010088824422353018,0.016692240119965878 1.262678e-04
Length = 47 rows
# read source X,Y positions from the table, show with circles of radius 10pix
positions = (sources['xcentroid'], sources['ycentroid'])
apertures = CircularAperture(positions, r=15.)
# plot--------------------------------------
fig, axs = plt.subplots(1, 1, figsize=(8, 8))
# show last frame of first integration
axs.imshow(driz_dm.data, cmap='jet', interpolation='nearest', origin='lower', norm=LogNorm(vmin=10,vmax=500))
apertures.plot(color='yellow', lw=2.0, alpha=0.8)
axs.annotate('Drizzled image with detected sources', xy=(0.0, 1.02), xycoords='axes fraction', fontsize=12, fontweight='bold', color='k')
axs.set_facecolor('black')
plt.tight_layout()
plt.show()
Command line¶
To achieve the same result from the command line there are a couple of options. However, one must still generate the association file. In this case, it is best to copy the template above into a text file and save it to my_galaxy_lvl3_asn.json. The content is printed here for convenience.
{
"asn_type": "image3",
"asn_rule": "DMS_Level3_Base",
"version_id": null,
"code_version": "0.13.7",
"degraded_status": "No known degraded exposures in association.",
"program": "noprogram",
"constraints": "No constraints",
"asn_id": "a3001",
"target": "none",
"asn_pool": "my_galaxy_pool",
"products": [
{
"name": "my_galaxy",
"members": [
{
"expname": "IMA_science/my_galaxy_dither1_cal.fits",
"exptype": "science"
},
{
"expname": "IMA_science/my_galaxy_dither2_cal.fits",
"exptype": "science"
},
{
"expname": "IMA_science/my_galaxy_dither3_cal.fits",
"exptype": "science"
},
{
"expname": "IMA_science/my_galaxy_dither4_cal.fits",
"exptype": "science"
}
]
}
]
}
Option 1:
Run the Image3Pipeline class using the strun command:
mkdir demo_output
strun jwst.pipeline.Image3Pipeline my_galaxy_lvl3_asn.json --output_dir demo_output
This will produce the same output file in the user-defined --output_dir
Option 2:
Collect the pipeline configuration files in your working directory (if they are not already there) using collect_pipeline_configs and then run the Image3Pipeline using the strun command with the associated calwebb_image3.cfg file. This option is a little more flexible as one can create edit the cfg files, use them again, etc.
mkdir demo_output
collect_pipeline_cfgs cfgs/
strun cfgs/calwebb_image3.cfg my_galaxy_lvl3_asn.json --output_dir demo_output
This will produce the same output file in the user-defined --output_dir