PRIMUS 2d Reduction Status

From NYU CCPP Wiki

This is a page about the PRIMUS project.

This page is meant to document the current status of the 2d reductions and the ongoing efforts to improve the code. This document is by no means complete and will provide a skeleton outline of the current version of the pipeline and identify current weakness.

The current workhorse for the 2d pipeline is run_reduction. All of the steps outlined here are completed by this program.


  1. Generation of a test slit model
    1. A frame of the input images is displayed in ds9 and is prompted to use a ds9 region to indicate one of the slit footprints in the image.
    2. This model is saved and will later be used when locating the slits on the full mask frame
    3. Code used : extract_model
      1. There is more information we can take from the .SMF file. The tags .dx, .dy1 and .dy2 refer to the slit width (.dx in mm) and the .dy1 and dy2 is the upper and lower extent of the slit relative to the object. So slit_len and slit_width are given in mm for each aperture.
      2. We can also use the spacing of the slit pairs in the .SMF to deduce the relative pixel shift in both spatial and spectral directions. This should match basically with the RA-NOD and DEC-NOD cards given in the image FITS header.
  2. Fit Helium Arcs
    1. If there were helium arc images taken through this mask, they are extracted and fit
    2. This fit is used to generate a scale factor to apply to calibrate the wavelengths of the spectra as a function of location on the focal plane.
    3. After this fit has been preformed, the solution is saved as a fits file in the calibration directory for the rerun and a qa plot is generated that illustrates the continuity of the solution.
    4. Code used : fit_helium_arc, arcfit_qaplot
  3. Generate Bad pixel masks and Halo correction terms
    1. If multiple images are specified (Richard needs to make this statement true --currently, the code will try and coadd a single exposure if the user asks it to), they are medianed together and this median is used to find cosmic rays in each of the input images and add these to a bad pixel mask.
    2. The final coadded image is taken as the average of the bad pixel -masked input images
    3. The final coadded image is then used to generate a model for the scattered light halo of the field. This model is generated using a fixed normalization model. We might try and solve for the normalization later.
      1. TODO: From what I could tell the halo shape and normalization may have changed slightly between the spring 2006 runs and the fall 2006 runs, possibly related to the changes in the IMACS dewar. Someone should check this and possibly alter/fit for the halo profile. --Burles
      2. In November, we took some arc images through a V filter to limit the number of lines. These may be a good test of the halo or a good place to fit for a new halo profile / normalization. -- --Cool ; 4 December 2006 14:13
    4. The halo model is written out to be used at a later time
    5. Code used : coadd_images_agg
      1. TODO: It would be worthwhile to run optimal extraction on the combined image first. We don't have to use the spectral exptractions (but we could compare to the individual extractions for QA). The main reason is to get a very good estimate of slit positions from the sky profile fitting of the coadded frame. We would also get a good initial guess for the object profile in each slit, and a S/N estimate for the plate. Let's think of a good naming convention for a coadded extraction, maybe one already exists. This extraction would be run after extract_slits is done. --Burles 17:17, 1 December 2006 (EST)
  4. Find the slit locatons:
    1. Using the example of the slit footprint provided by the user in the first step, the images are cross-correlated to find the slits on the mask.
    2. These slit locations are used to generate a bulk shift between the actual slit locations and the locations predicted from the SMF files.
    3. After the shifts are calculated, the _slits_ files are used to locate the slit for each object targeted. These slit locations are stored for later use
    4. We should probably incorporate Scott's tweak the slit finder to make these positions more robust
    5. Code Used : extract_slits
  5. Extract Spectra :
    1. Update - instead of the users specifying the NOD and SHUFFLE for the reductions, these are now retrieved out of the headers during the run_reduction preparations for reductions. Also, the code checks that there are not imags with more than one N&S configuration included and stops if it finds any images that seem out of place.-- Cool 10:28, 5 December 2006
    2. The two-d images from the mask are first correted for the contribution of the halo and then the pixel-to-pixel flat field generated from sky flats is applied to correct for sensitivity variations.
    3. The four slits are then traced with linear functions and are extracted with 6 pixel simple boxcar extractions after performing a simple nod and shuffle subtraction. We have shown that this extract is not ideal and need to incorporate many possibilities.
    4. We should probably fit for the trace more correctly
    5. We shoud use the psf in the extractions and extract using profiles
    6. The spectra from the two nod & shuffle positions are combined by shifting them by the number of pixels in the nod and directly averaging. This can likely be improved as well.
    7. Code used: extract_spectra
  6. Coadd multiple exposures :
    1. After each of the images for a mask are extracted, the spectra from both position A and position B are coadded to make the coadded spectrum at each location. These two coadded spectra are then shifted by the nod amount and coadded to get a final coadded spectrum for the mask.
    2. The individual images are coadded with weight according to the signal to noise in each input exposure.
    3. Code used : coadd_spectra
      1. I think we should change the ordering of the co-addition: A&B spectra should be coadded with their respective subsequent exposures, i.e. A spectra only combine with A spectra. At the end, we can decide whether it is worth coadding the super-A with the super-B. But it might be better just to fit the templates to 2 individual spectra. This will likely help us recover from cases where one of the two slits was compromised somehow --Burles 17:17, 1 December 2006 (EST)
  7. QA plots :
    1. After the data are reduced a number of quality assurance plots are created to validate the process these include:
    2. S/N versus aperture magnitude
    3. S/N in both apertures compared
    4. QA for the fits to the slit traces
  8. Create photometric and spectroscopic linematch fles:
    1. The final extracted files are compared to the _slits_ files generated with maskgen and the two files are matched line-by-lineand the _spec and _phot files are output


Outstanding todos:

  • Anyone:
    • Generate a list of normalized flat fields from all the data obtained to-date. Characterize any variations in differences from run to run and passband to passband.
    • Try to make sure all errors generated in the code given by various text error messages are recorded as flag bits in the 2d outputs.
  • Richard:
    • Incorportate coadd_images_agg correctly
      • DONE - Now, 3 or more images must be specified in order to the images to be coadded. If less than 3 are spcified, not coadding is done and no halo correction image is generated.--Cool 12:52, 5 December 2006 (EST)
    • Can you take the first part of the code that does the tracing and adds the new slits (in test_skyprofile) and put this into extract_slits? I think we want the apertures totally identified by this step. But we should add a flag in the aperture structure for the ones that were shifted and the new ones which were found near the edges. I think we want to skip or at least be cautious in both cases. --Burles 15:16, 29 November 2006 (EST)
      • Done - this has been included in extract_slits (specifically in find_slits). This will likely be superceded by hogg_find_slits
    • Check to ensure that all new revisions to the code work for the various nod&shuffle values and check that none of these values are hardwired Cool 14:38, 30 November 2006 (MST)
    • Add all of the needed files to be ready for the Dec and Jan runs
  • Scott
    • Use slit parameters in SMF file to form a better guess for the spatial/spectral nod separations for each slit pair. Right now we just assume uniform nod values at integer pixels.
    • Finish first iteration of optimal extraction, using a gaussian of fixed width and position relative to the slit, which is chopped by the slit_edges and smoothed with a fixed IMACS spatial PSF.
    • Need to allow for some variation in the profiles, brighter targets will have some biases without.
    • Need to incorporate the variation of the IMACS spatial PSF with focal position and wavelength.
    • Need to incorporate the updated slit-edge positions from the sky profile fitting.
    • Need to double-check the application of the flat-field. The best case is a bright target that falls on an obvious pixel feature in the flat field. Show that the extracted sky and object look correct.
    • We could add some weak constraints on the extracted sky profile in subsequent iterations that it either looks like the globally predicted one or the median profile from the first round of extraction or both.
  • Adam
    • Work on the global fitting of the sky profile parameters (slit_height, slit_separation, and slit_position vs. pixel and PSF vs. pixel for each of the sky pairs).
  • Hogg
    • Look at avg sky and sky as a fn of a/b, ccd number, field position to test uniformity of throughput.
  • Others
    • fill me in
Retrieved from "http://howdy.physics.nyu.edu/index.php/PRIMUS_2d_Reduction_Status"