PRIMUS Data Model
From NYU CCPP Wiki
This is a page about the PRIMUS project.
[edit]
Directory structure
The PRIMUS data directory structure is as follows (the most important files are bold-faced):
$PRIMUS_DATA/
design/
[survey]/
targets/
[survey]_all.fits - all possible targets (ie. very inclusive)
[survey]_targets.fits - parallel to all file, with target flags
[survey]_sample.fits - parallel to targets file, with sparse sampling info
masks/
[survey]_bsmask.ply - full bright star mask
[survey]_bsmaskhalf.ply - half radius bright star mask
[survey]_bsmaskquart.ply - quarter radius bright star mask
[run]/
[survey]_slits_[number].fits - file for mask number [number] with assignments (unique!)
[survey].mask-[number].ps - PostScript observing files
[survey]_[run]_stats.txt - Text file with statistics information
[maskname].obs, .dat, .sub - IMACS files for each mask (using 8-digit mask names)
[survey]_assign.fits - assignments of all targets for this run
[survey]_masks.fits - summary file with mask positions
[survey]_masks_stats.fits - summary file with mask design results
[survey]_slits_[number].fits - file for mask number [number] (unique!)
[survey]_masks_poly.ply - mangle style polygon file for each mask
[survey]_masks_window.ply - mangle style polygon file for each mask (with bright star mask removed)
[survey]_mask_conflicts_[number].fits.gz - map of mask (at half resolution), should contain only 0s and 1s
photo/
[survey]/ -individual photometric catalogs for each target survey
templates/
bruzualcharlot/
ssp_[age]_[Z].spec.gz -- single burst model with given age and metallicity
cst_[age]_[Z].spec.gz -- constant star formation model
t[tau]_[age]_[Z].spec.gz -- exponential SFH
CWW/
EFlux.dat -- Elliptical spectral template
IrrFlux.dat -- Irregular spectral template
SbcFlux.dat -- Sbc Spectral template
ScdFlux.dat -- Scd Spectral template
fstarflux/ - F-star fluxing data
skyflux/ - sky fluxing data
flats/ - flats for 8 IMACS CCDs
old_reduction/ - obsolete redux
$PRIMUS_RAWDATA/
ut[date]/
ccd[exp]c[ccd].fits.gz -- raw data from IMACS
$PRIMUS_REDUX/
flats/ -- dome flat redux for various nights
1d/ -- redshift fitting and other one-d results
[rerun]/ -- should be four digits (corresponds to two-d rerun below)
ut[date]/
[mask]-zAll.fits -- redshift results
[rerun]/ -- should be four digits
ut[date]/
[mask].extract.fits.gz -- Extracted and coadded spectra for the mask
ccd[exp].extract.fits.gz -- Extracted spectra for a single exposure
[mask]_aper.fits -- FITS file containing location of each slit
[mask]_coadd.fits.gz -- Total image of all input images
[mask]_halo.fits.gz -- Halo correction image
[mask]_phot.fits.gz -- Photometry targeting data for each slit
[mask]_spec.fits.gz -- Linematched spectro. data for each slit
[mask]_summary.txt -- Summary of extraction on the mask
badpix/
ccd[exp]_badpix[ccd].fits.gs -- bad pixel & CR mask for given exposure
calibration/
ccd[exp]c[ccd]_arc.fits -- Helium arc wavelength scale solution
flat_c[ccd].fits.gz -- Flat field for the night (average of all filters)
flat_[filter]_c[ccd].fits.gz -- Flat field for a single filter
qaplots/
ccd[exp]-qaplot.ps -- Qaplot of the S/N for exposure
ccd[exp]-signalmap.jpg -- Image of the predicted S/N - measured
ccd[exp]-traceqaplot.jpg -- Slope of each trace on focal plane
[mask]-qaplot.ps
[mask]-signalmap.jpg
[mask]_findslit_qa.ps -- Plot showing location of each slit on mask
The "survey" name is meant to be something like COSMOS or COMBO-17 or DEEP2 or whatever. If we retarget the same area twice, we assign a new "survey" name. The spectra should be labeled with the mask number they come from, but we haven't specified anything more than that.
The mask numbers should be globally unique, so that we do not get confused. Note that nothing in the naming convention distinguishes according to position angle; that is, although every position will have multiple position angles, there is not necessarily anything in the numbering that connects them.
The mask names are constrained to be eight digits, and so we generally choose them to be of the form:
xxxx####
where "####" is the four-digit unique mask number, and "xxxx" is shorthand for the survey they are part of. The list of runs tells you what each 4-letter shorthand means.
Currently, $PRIMUS_RAWDATA and $PRIMUS_REDUX point to /saguaro/rcool/primus/rawdata and /saguaro/rcool/primus/redux on saguaro and /global/data/primus/rawdata and /global/data/primus/redux on howdy.
[edit]
Data Products
The two-dimension pipeline performs reductions of the IMACS images, including correcting for relative gains in the IMACS ccds, flat fielding to correct for detector artifacts, masking of bad columns, coadding individual images to create bad pixel masks, correcting for the scattering halo in the IMACS optics, performing nod and shuffle sky subtraction, and finally, extracting the spectra. Here, we discuss the most relavent data products and the data model for each. Some nomenclature: here SHORTSLIT data indicates spectra taken with two separate slits that the object is nodded between, LONGSLIT indicates the object is nodded along a single slit. The early PRIMUS configurations were LONGSLIT while the masks designs for March 2006 and after are SHORTSLIT
ccd[exp].extract.fits.gz - Binary FITS file containing the extraction
- of a single IMACS data frame.
INFILE - Name of SMF file used to generate targeting information
TYPE - Aperture type (either SLIT or HOLE)
NAME - Name of object in SMF file
RA
DEC
XPLANE - Location of object on Baade's focal plane
YPLANE
DX
DY1
DY2
XCORNER - Corners of the box cutout from whole image before extraction
YCORNER
CCDNUM - CCD spectrum was extracted from
FOPT1 - Extracted counts per pixel in position A [SHORTSLIT ONLY!]
FIVAR1 - Inverse Variance of measured counts in position A
SKY1 - Sky spectrum in position A
WAVE1 - Wavelength vector for position A spectrum
CALIB1 - Multiplicative conversion factor to convert extracted
counts to counts per second per micron
MASK1 - Bad Pixel Mask for Position A.
Use PRIMUS_FLAGNAME("SPPIXMASK", value) to interpret
FOPT2 - Extracted spectral information for position B [SHORTSLIT ONLY!]
FIVAR2
SKY2
WAVE2
CALIB2
MASK2
FOPT - Average extracted counts per pixel for the mask.
- For LONGSLIT data, this represents the pure extacted counts,
- while for SHORTSLIT data, this is the average of positions A & B.
- When coadding in SHORTSLIT, the two spectra are align by an interger pixel
- amount, and thus working on FOPT1 and FOPT2 may be more preferable.
FIVAR
WAVE
CALIB
BADEXTRACT - Mask bit describing any problems during extraction. Use
PRIMUS_FLAGNAME('SPEXTRACT', value) to interpret -- SHORTSLIT only
ANDMASK - Final AND pixel mask for the exposure
ORMASK - Final OR pixel mask for the exposure
SKY - Final sky spectrum for the exposure
TRACESLOPE - Slope of the trace measured on the 2d image [SHORTSLIT only]
XCEN - Location of traces used for extraction [SHORTSLIT only]
[mask].extract.fits.gz - Binary FITS file containing the coadded data for a mask.
Similar data model to the ccd[exp].extract.fits.gz files.
All keywords have the same definition as above except for those
listed below.
SCALEVAL -- When coadding, a multiplicative scalar is applied to each spectrum
-- to bring them to the same median level. This array contains
-- those corrections.
[mask]_coadd.fits.gz - Coadded image of all of the two-dimensional
images for a given mask. In order to preserve Poisson statistics, the
coadded image is the total image.
HDU (2*[ccd]-1) -- Total coadded image in units of total counts for CCD #[ccd]
HDU (2*[ccd]) -- Weight map for CCD #[ccd]. Each pixel lists the number of input
images that contribute to it (and thus the number of pixels not
affected by bad pixels or cosmic rays).
[mask]_halo.fits.gz - Halo correction image for a given mask.
HDU 0 - [2084,2060] binned image of the halo correction calculated
HDU 1 - [2048,2060] binned image of the slits used to create halo correction
HDU 2 - [2048,2060] binned pixel mask when calculating halo image
HDU 3 - 8*nexp - Structure with one entry per ccd including the exposure time
median flux level of the image
[mask]_summary.txt - Text file summarizing several of the statistics of each of the
input images and the final coadd. These include the signal-to-noise ratio
for each input image (including position A and B for SHORTSLIT data), a
summary of how many slits were extracted, and the number of slits affected
by various extraction problems (SHORTSLIT only).
"[mask]_zAll.fits.gz" - Output structure from 1d processing.
RA - Astronometry of object
DEC
OBJNAME - Object number (platenumber - obj number on plate)
ZGRID_GAL - [ngalgrid] - Redshifts object is fit at - if all are zero, then object was not fit
CHI2_GAL - [ngalgrid] - Chi2 value at each redshift in ZGRID_GAL
ZMIN_GAL - [nmin] - Lowest nmin minima in chi2 surface (default to 5 values)
ZMINERR_GAL - [nmin] - 1 sigma error estimate of the minimum redshift based on quadratic fit
CHI2MIN_GAL - [nmin] - Chi2 of galaxy fit at each of the ZMIN_GAL
COEFF_GAL - [ngaltemp, nmin] - Coefficients for best fit minima
FMOD_GAL1 - [npix, nmin] - Model prediction for counts in each aperture
FMOD_GAL2
TWEAK_GAL1 - [npix, nmin] - Fluxing flexibility used in fitting objects
TWEAK_GAL2
TYPE_STAR - string name for each type of star fitted
PIXSHIFT_STAR - [nstarshift] - if pixel shifts are tested with the stellar
templates, this lists the shifts used
CHI2_STAR - chi2 for each stellar template
FMOD_STAR1 - [npix, nstar] - model prediction for star counts in each aperture
FMOD_STAR2
TWEAK_STAR1 - [npix, nstar] - fluxing tweak applied in fitting stellar templates
TWEAK_STAR2
POWERLAW_INDEX - [nplindex] - array of power-law indices used to fit for power-law
CHI2_PL - best fit chi2 for each powerlaw index
FMOD_PL1 - [npic, nplindex] - model prediction for power-law counts in each aperture
FMOD_PL2
TWEAK_PL1 - [npix, nplindex] - fluxing tweak applied in fitting powerlaw templates
TWEAK_PL2
ZGRID_AGN - [nagn] - Redshift AGN templates are fit at
CHI2_AGN - [nagn] - Chi2 value for each fitted AGN redshift
ZMIN_AGN - the following parameters are identical to _GAL execpt using AGN templates
ZMINERR_AGN
CHI2MIN_AGN
COEFF_AGN
FMOD_AGN1
FMOD_AGN2
TWEAK_AGN1
TWEAK_AGN2
ZBEST_GAL - best galaxy redshift
ZBESTERR_GAL - error on best galaxy redshift
CHI2BEST_GAL - chi2 for best fitting galaxy
ZBEST_AGN - best AGN redshift
ZBESTERR_AGN - error on best agn redshift
CHI2BEST_AGN - chi2 for best fit agn
ZBEST_STAR - pixel shift for best fitting star
TYPEBEST_STAR - name of best fitting stellar type
CHI2BEST_STAR - chi2 of best fitting star
ZPRIMUS - final primus redshift
ZPRIMUS_ERR - error on final primus redshift
ZPRIMUS_CLASS - class (AGN, GAL, STAR)
ZPRIMUS_TYPE - subtype (for STARS)
ZWARNING - flags asociated with this object
PAIR - array of which apertures were used to fit.
If [1,0] then only A was used, [0,2] only B, [1,2] both
