Kcorrect
From NYU CCPP Wiki
General description
The code and results are described in a paper in the Astronomical Journal.
This IDL code fits very restricted spectral energy distribution models to galaxy photometry or spectra in the restframe UV, optical and near-infrared. The main purpose of the fits are for calculating K-corrections, which the code does as well. The templates it uses for the fits may also be interpreted physically, since they are based on the Bruzual-Charlot stellar evolution synthesis codes. Thus, for each fit galaxy we can provide an estimate of the stellar mass-to-light ratio.
The easiest interface to the code, if it is available to you, is the IDL code. Detailed documentation is available for all of the routines. For users of the IDL version of the code, we have special high-level routines to handle data from a number of commonly used surveys: SDSS, GALEX, 2MASS, DEEP2 and GOODS. However, the lower level IDL routines are written generally enough that one can use those routine to handle data from any survey in the restframe UV, optical, or near-infrared.
The lowest level code compiles into a shared object library callable by C, so that non-IDL users can incorporate the K-correction routines directly into their code. It is possible in principle to link the C libraries into code based on SM, TCL/Tk, or Python, and I would be interested in helping interested parties to do this.
One can imagine using the results of this code to calculate the evolution of the luminosity function, the distribution of galaxy colors, as well as to develop galaxy classification algorithms.
What's new in v4
There are three main things that are new in the latest version:
- The templates have been extended to work in the rest-frame ultraviolet, and still work in the optical and near infrared. In particular, our templates are consistent with data at low redshift from GALEX, SDSS, and 2MASS can be explained by our templates, and with data up to redshifts of 1.5 or 2.0 in DEEP2 and GOODS.
- Because they are based on unaltered stellar evolution synthesis results from Bruzual-Charlot models, the fits have a natural physical meaning and one can make physical conclusions about recent star-formation histories and stellar masses.
- The SDSS AB corrections have changed somewhat.
- I have added a bunch of higher level IDL routines to perform common tasks, like K-correcting SDSS data, GOODS data, GALEX data, DEEP2 data, and 2MASS data, in addition to taking SDSS data and simulating DEEP2 or GOODS observations at higher redshift. When possible, users should stick to these high-level routines.
The new templates were obtained using a clever algorithm called "nonnegative matrix factorization," which finds the best set of N nonnegative templates which can be linearly, nonnegatively combined to explain a set of data. Think of it as "nonnegative PCA." However, it has the advantage over PCA that you can account explicitly for the uncertainties in the data. In our case, we use a basis set for the templates which consists of predictions of stellar evolution synthesis models, but you could simply use a set of discrete delta functions if you liked. This method has a bunch of potential applications in astronomy. The code is NMF_SPARSE in idlutils.
Conditions of use
Conditions of using the code released here are to send your email address to mb144@nyu.edu, to cite Blanton et al. 2007, and to specify the version tag (eg. v4_1_4) of the code used. (You can check this in the IDL code using the k_version command). Note that a previous version of the kcorrect paper existed (Blanton et al. 2003) which you need only cite if you are using v3_2 or earlier.
Obtaining the software
First off, if you are downloading the code, please register it. Do so by sending your email address to mb144@nyu.edu. No other information is required. I want this because it will allow me to contact users in case of any improvements or bug fixes.
Note that the IDL code depends on the idlutils package. The best way to obtain the code is to simply download the latest version from this site. The tagged and released versions, from latest to earliest, are:
The earlier version is only included so that you can reproduce old results if you like. Note that there have been some usage changes. It is also possible to export tagged versions from a public CVS repository, using the CVS "export" feature, as follows (pirated this description from spectro.princeton.edu):
- Make sure your environment uses ssh to access CVS:
CVS_RSH=ssh export CVS_RSH
- The first time you access the repository, create the .cvspass file (run this in your home directory):
cvs -d :pserver:anonymous@spectro.princeton.edu:/usr/local/cvsroot login When the above asks for a password, just hit Enter.
- After this, you can just export any version you want:
cvs -d :pserver:anonymous@spectro.princeton.edu:/usr/local/cvsroot export -r v4_1_4 kcorrect where you can replace v4_1_4 with whatever version you want.
Please export only based on revision tag, so that we can track versions properly.
It is also possible to CVS "checkout" the latest version of the code. This is not recommended unless you want to become involved in developing kcorrect (that is, if you will want to alter the code and check it back in). Please contact me if you want to do this.
Installing the software
The code has been tested, and compiles and runs on a Red Hat 7.1 OS. Little experimentation with other operating systems has been performed. Other Linux versions should port very easily, at least. In order to compile the code follow these directions:
First, set the environmental variables. For example, in bash:
KCORRECT_DIR=/usr/local/kcorrect PATH=$KCORRECT_DIR/bin:$PATH LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$KCORRECT_DIR/lib export KCORRECT_DIR export LD_LIBRARY_PATH IDL_PATH=$IDL_PATH:+$KCORRECT_DIR/pro
or in tcsh:
setenv KCORRECT_DIR /usr/local/kcorrect
setenv PATH $KCORRECT_DIR/bin:$PATH
setenv LD_LIBRARY_PATH ${LD_LIBRARY_PATH}:$KCORRECT_DIR/lib
setenv IDL_PATH ${IDL_PATH}:+$KCORRECT_DIR/pro
These default settings may need to be changed on your system as follows:
- KCORRECT_DIR: Simply substitute whatever directory you have downloaded this product into.
- PATH: Obviously needed to run the binaries created
- LD_LIBRARY_PATH: Required for the stand-alone code, so that the OS can find libkcorrect.so at runtime.
- IDL_PATH: Set only if you are using IDL
You probably want to put these settings in your .bashrc or .tcshrc file. EvilUPS or UPS users can just type "setup -r /usr/local/kcorrect kcorrect"
Second, type "kevilmake -k" while in $KCORRECT_DIR. Then everything should compile At this point, you should be able to open IDL and start using the utilities described below, or run the stand-alone C code.
The IDL code depends on the idlutils package.
Note that the C codes "fit_coeffs.c" and "reconstruct_maggies.c" only depend on the K-correction library; thus, you can use them as templates for incorporating the K-correction code directly into your C code, as long as you include the K-correction library. In fact, you would probably put something like "fit_coeffs.c" into your code but include the k_reconstruct_maggies() call directly inside to get the K-corrections.
There is a known problem with compiling the stand-alone C code on a Mac OS X system. See Known problems for a description.
Basic concept of obtaining K-corrections
I'll describe the basic idea before getting into exactly how to use the code.
If you want to convert apparent magnitudes in band R to absolute magnitudes in band Q, you need to calculate the K-correction, which is defined by the equation:
where mR is the apparent magnitude, MQ is the absolute magnitude, DM(z) is the distance modulus, accounting for the angular diameter distance and cosmological surface-brightness dimming, and KQR(z) is the K-correction. By absolute magnitude we mean: the apparent magnitude in band Q the object would have if it were observed at rest, 10 pc away, using an aperture that contains its total flux. The distance modulus accounts for the difference between an object's actual distance and 10 pc. The K-correction accounts for the fact that you observed a redshifted galaxy in band R but the absolute magnitude requires a rest-frame observation in band Q. Obviously the difference between the fluxes observed in different bandpasses is fully determined by the galaxy SED and the description of the bandpasses. See Hogg et al. (2002) for a full description of what the K-correction is.
In order to get the appropriate SED for a set of galaxy fluxes, kcorrect fits an SED which is a nonnegative linear combination of some small number of templates. The templates have been optimized to minimize the residuals between the actual galaxy fluxes and the galaxy fluxes reconstructed from the galaxy SED fit. The units of the fit spectra created are in ergs cm−2 s−1 A−1.
As an example, consider the figure below. I have taken actual spectra from the SDSS and projected them onto the g, r, and i bandpasses (since these are the ones the spectrum overlaps). Then I have run kcorrect (in this case version v3_2) and tried to reconstruct the spectrum. What kcorrect does here is to fit the nonnegative sum of three (carefully chosen) templates to the g, r, and i fluxes. The black line is the original spectrum, the red is the reconstructed. As you can see, the fit is reasonable.
I designed the code so it could calculate K-corrections from some set of observed AB bands to those same bands shifted by an arbitrary redshift, if so desired. For example, for the SDSS spectroscopic survey, most galaxies are observed near redshift 0.1; it makes sense, therefore, to K-correct to bandpasses shifted by 0.1 (ie. by a factor 1.1). This procedure minimizes errors in the estimated K-corrections, because the galaxies at z=0.1 will then have a K-correction which independent of their SEDs (and therefore their colors) and equal to −2.5log10(1+0.1). In general, for a bandpass shifted by z_0, the K-correction for a galaxy at z_0 to the observed bandpass shifted by z_0 is −2.5log10(1+z_0), independent of galaxy SED. A bandpass b shifted by redshift z is denoted zb (as in 0.1g).
The inputs to the code are used AB maggies. These are the ratio of the source to the AB standard source in each band. To relate these quantities to magnitudes, an object with total flux f in maggies has magnitude
An advantage of the maggie unit system is that it is linear, and thus can when necessary accommodate negative flux estimates. Two notes:
- As discussed below, SDSS catalog numbers are NOT on our best guess for the AB system. A set of offsets must be applied to the magnitudes to achieve our best guess. Our high level IDL code deals with this automatically.
- SDSS catalog magnitudes obtained from the official survey database are luptitudes, which for reasonably bright objects are equivalent to magnitudes. While maggies are simply related to magnitudes by 10−0.4m, the conversion from luptitudes to maggies is a bit more complicated; see the description accompanying the DR2 documentation). Our high level IDL code deals with these conversions automatically.
But you may still be wondering what I mean by an "AB" magnitude. The AB system is designed such that a flat spectrum object with fν = 3631 Jy = 3.631 × 10−20 ergs cm−2 s−1 Hz−1 should have every magnitude equal to zero. The beauty of the AB system is that the uniform definition makes it convenient to synthesize AB magnitudes from theory or models. The tragic flaw is that the quality of the AB-ness of a system is very dependent on precise spectrophotometry of standards and the carefulness of the calibrators, since no objects have a flat spectrum. There is a tension between these two needs --- similar to other tensions throughout astronomy between making precise measurements and making interpretable ones.
Finally, I have also included code to calculate photometric redshifts based on the templates. This procedure is as simple as running the K-correction code at each redshift and finding that redshift which provides the best fit in the χ2 sense.
IDL routines for SDSS, GALEX, 2MASS, DEEP2, and GOODS users
The easiest interface to the code, if it is available to you, is the IDL code. The detailed documentation is available for all of the routines. The IDL code depends on the idlutils package.
The following routines exist:
- sdss_kcorrect: SDSS K-corrections
- sdss2bessell: Take SDSS ugriz fluxes and return Bessell UBVRI fluxes
- sdss_kphotoz: SDSS photometric redshifts
- galex_kcorrect: GALEX+SDSS K-corrections
- twomass_kcorrect: 2MASS+SDSS K-corrections
- gst_kcorrect: GALEX+SDSS+2MASS kcorrections
- deep_kcorrect: DEEP K-corrections
- sdss2deep: Take local SDSS observations and return fluxes which would be observed in DEEP at some given redshift.
- goods_kcorrect: GOODS K-corrections
- sdss2goods: Take local SDSS observations and return fluxes which would be observed in GOODS at some given redshift.
In all cases I have tried to make the inputs as close to the catalogs that the user might be using, and in the form of a structure passed by the user. The code calls routines like the following to handle the transformations:
As noted in the documentation, these routines sometimes perform corrections to the AB system (such as the small corrections for the SDSS and the transformation from the Vega system in 2MASS). Also, they tend to add small uncertainties in quadrature with all of the uncertainties in order not to overfit to possible calibration errors. Sometimes they require the Schlegel, Finkbeiner and Davis dustmaps to be correctly installed on you system. Finally, they try to recognize the usual error flags and set inverse variances to zero where appropriate. The K-correction routines do output the maggies and inverse variances they calculate in the "omaggies" and "oivar" keywords.
In the case of SDSS, I provide both calibObj structure inputs (like those used by the Princeton reductions and the NYU-VAGC) and tsObj inputs (like those used by the Data Archive Servers). I don't ever use the Catalog Archive Server but would be happy to add an input structure for people who do if you let me know what columns to assume is in it. In the meanwhile, basically anything called "Counts" in the tsObj structure corresponds to anything called "Mag" in the CAS. Note the oddity that COUNTS_MODEL corresponds to modelMag.
These K-correction and photometric redshift routines will also return absolute magnitudes (and inverse variances of the same). To do so it assumes Ωm = 0.3 and ΩΛ = 0.7 (though these can be set differently on input). Absolute magnitudes are reported minus 5log10(h) (where Ho = h*100 km/s/Mpc), with h=1 (ie, to use H0=72 km/s/Mpc one has add 5log10(.72) to the output of the code).
You will also note some outputs having to do with mass-to-light ratios and masses.
Lower-level IDL routines for K-corrections and photo-z's of arbitrary data
All the above K-correction and photometric redshift routines use the same basic engines, kcorrect and kphotoz. These routines uses SDSS filters by default but allow any type to be input. To perform K-corrections for other sets of bandpasses (say, the Bessell bandpasses or 2MASS bandpasses), look for them in $KCORRECT_DIR/data/filters and insert them in the filterlist= flag.
As an example of using kcorrect, one might do the following:
kcorrect, [1., 4.78, 10.96, 14.45, 19.05], $
[1100., 28., 7.7, 4.4, 2.5], $
0.03, kcorrect, band_shift=0.1, chi2=chi2
which would take the set of maggies in the first argument, observed for a galaxy at redshift 0.03, and return the K-corrections to the ugriz magnitudes shifted by 0.1. This choice of maggies and uncertainties should return a good chi^2. Note that it is a nonnegative fit, so linear goodness of fit measures don't exactly apply.
Please note that kcorrect uses a lot of overhead (2-3 seconds worth), so if you can, use it on several thousand galaxies simultaneously at least. If you use it on each object individually (as in the example given above), you will be in trouble. If you truly cannot avoid it, then specify the extra arguments rmatrix and zvals. For example, if you run the following:
kcorrect, [1., 4.78, 10.96, 14.45, 19.05], $
[1100., 28., 7.7, 4.4, 2.5], $
0.03, kcorrect, band_shift=0.1, chi2=chi2, $
rmatrix=rmatrix, zvals=zvals
kcorrect, [1., 4.73, 11.26, 14.25, 18.85], $
[1100., 28., 7.7, 4.4, 2.5], $
0.03, kcorrect, band_shift=0.1, chi2=chi2, $
rmatrix=rmatrix, zvals=zvals
You will find that the second call takes far less time than the first, because the temporary data stored in rmatrix and zvals will not have to be regenerated. The high-level routines I discuss above all maintain common blocks which store rmatrix and zvals for you so you don't have to worry about this.
If you want to look at the fit spectrum, then specify vmatrix, lambda, and coeffs. The commands
kcorrect, [1., 4.78, 10.96, 14.45, 19.05], $
[1100., 28., 7.7, 4.4, 2.5], $
0.03, kcorrect, band_shift=0.1, chi2=chi2, $
vmatrix=vmatrix, lambda=lambda, coeffs=coeffs
plot, lambda, vmatrix#coeffs, xra=[2000., 12000.]
will plot the best fit spectrum. You can also just call k_load_vmatrix to get the templates.
If you want to project that spectrum onto a new set of bandpasses, use k_reconstruct_maggies. For example, to project back onto the SDSS bandpasses, the command
k_reconstruct_maggies, coeffs, 0.03, maggies, vmatrix=vmatrix, $
lambda=lambda, filterlist=['sdss_u0.par', 'sdss_g0.par', $
'sdss_r0.par', 'sdss_i0.par', 'sdss_z0.par']
will pass back the reconstruction in the array maggies. You can project onto any bandpasses you want. For example,
k_reconstruct_maggies, coeffs, 0.03, maggies, vmatrix=vmatrix, $
lambda=lambda, filterlist=['bessell_B.par', 'bessell_V.par']
vega2ab=k_vega2ab(filterlist=['bessell_B.par', 'bessell_V.par'],/kurucz)
bessellmags=-2.5*alog10(maggies)-vega2ab
BminusV=bessellmags[0]-bessellmags[1]
will yield the Vega B-V for the best fit.
Fitting photometric redshifts just requires fitting the coefficients for a number of redshifts and minimizing the &chi2 residuals for the fits. I provide the IDL routine kphotoz to do so (similar to the kcorrect routine). For example, one might do the following:
kphotoz, [1., 4.78, 10.96, 14.45, 19.05], $
[1100., 28., 7.7, 4.4, 2.5], $
photoz
Again, there is large overhead, so one-at-a-time is not the way to call kphotoz. As for kcorrect, you can avoid spending this overhead more than once either by feeding it lists of objects or as follows:
kphotoz, [1., 4.78, 10.96, 14.45, 19.05], $
[1100., 28., 7.7, 4.4, 2.5], $
photoz, rmatrix=rmatrix, zvals=zvals
kphotoz, [1., 4.78, 10.96, 14.45, 19.05], $
[1100., 28., 7.7, 4.4, 2.5], $
photoz, rmatrix=rmatrix, zvals=zvals
By default, kphotoz uses a very simple prior (equal to the redshift z) to push the fits away from z=0. You can turn this off with the flag /noprior, or specify your own prior using the inputs zprior (which indicates the redshift grid) and lprior (which indicates the natural log of the prior).
A piece of advice on the type of magnitudes to use for SDSS photometric redshifts. For modern versions of PHOTO (v5_4 and later) it is best to simply use the model magnitudes for all galaxies. (And indeed this is what we use in sdss_kphotoz).
Users of kcorrect v3_2 will hopefully find that I retained backwards compatibility. Please note that the calling procedure has changed since v1_16, in the sense that inputting maggies is now the default.
Other useful utilities in the IDL library
In addition to the K-correction tools, kcorrect has some other useful tools in it:
- k_project_filters() is a standalone IDL routine to project a spectrum onto a filter.
- k_smooth smooths a spectrum with a Gaussian.
- k_solar_magnitudes() calculates Solar absolute magnitudes in any filter.
- k_vega2ab() provides conversions between AB and Vega magnitudes.
- lf_distmod() calculates distance moduli given a redshift and cosmology.
- A number of useful routines (mostly starting with "lf_") exist for calculating luminosity functions.
- There are various pieces of code (not all written by me) for reading in certain types of data files, like Bruzual-Charlot predictions (by John Moustakas), Bruce Draine's PAH spectra, PEGASE2 predictions (by Alex Quintero), Witt-Gordon multiple-scattering dust extinction predictions, and Mappings-III style emission line fluxes.
Stand-alone C code
The C code requires a bit more attention to use. If people say they would use it if it were easier, I will improve it. In this section, I will describe the stand-alone C programs which fit for the coefficients and which calculate the reconstructed maggies. In the next section, I will briefly describe how to use the libraries within your own C code.
There are two stand-alone programs, fit_coeffs and reconstruct_maggies. fit_coeffs uses the maggies in each band to calculate the coefficients. reconstruct_maggies uses the list of coefficients (as output by fit_coeffs to reconstruct the magnitudes.
fit_coeffs has a help flag:
apeiron.[kcorrect].7 % fit_coeffs --help
Usage: cat | fit_coeffs [--vfile vfile --lfile lfile
--ffile ffile ]
As indicated, you would call it like
cat maggies.dat | fit_coeffs
where each line of "maggies.dat" has the form:
redshift umaggies gmaggies rmaggies imaggies zmaggies uinvvar ginvvar rinvvar iinvvar zinvvar
All of the quantities in this file are in maggies, as described above. So you have to convert the luptitudes or magnitudes --- and the inverse variances --- into maggies before calling this. (Note that the conversion to the inverse variances from the maggies and the magnitude errors is (0.4 ln(10) × maggies × magerr)-2).
From the set of maggies and errors, the code calculates a set of four coefficients which correspond to its guess about the nature of the galaxy SED. The coefficients are output to standard out in the form:
redshift coeff0 coeff1 coeff2
where the redshift is included for reasons which will be clear soon.
Once you have the coefficients, you can calculate the reconstructed maggies for a galaxy with the same SED and same total flux, observed at any redshift through a shifted bandpass, using reconstruct_maggies. The help flag for reconstruct_maggies yields
apeiron.[kcorrect].8 % reconstruct_maggies --help
Usage: cat | reconstruct_maggies [--vfile vfile --lfile lfile
--ffile ffile --band-shift band_shift --redshift redshift]
So you can call it like:
cat coeffs.dat | reconstruct_maggies
where coeffs.dat is in the same format as the output to fit_coeffs:
redshift coeff0 coeff1 coeff2
The optional parameter "band_shift" specifies the bandpasses to reconstruct (band_shift=0., the default, indicates to use the observed frame bandpasses). "redshift" specifies the redshift at which to observe the galaxy (by default, the input redshift is used). The output is in the form:
redshift u_rec g_rec r_rec i_rec z_rec
in the original maggy units.
To summarize with an example, if you wanted to calculate the K-corrections from the observed bandpasses to rest-frame bandpasses shifted by 0.1, you could call the code as follows:
cat $KCORRECT_DIR/test/sample.dat | fit_coeffs | reconstruct_maggies >! maggies.dat cat $KCORRECT_DIR/test/sample.dat | fit_coeffs | reconstruct_maggies --band-shift 0.1 --redshift 0. >! maggies.z0.dat
The K-corrections are then defined by −2.5 log10(maggies/maggies.z0).
You can use the ffile option to reconstruct_maggies to specify a file which contains a list of the filters to use (by default, it uses the SDSS filters).
The stand-alone program fit_photoz works in the same way as fit_coeffs:
apeiron.[v3_1].27 % fit_photoz --help
Usage: cat | fit_photoz [--vfile --lfile
--ffile ]
As in:
cat maggies.dat | fit_photoz
where in this case each line of "maggies.dat" has the form:
umaggies gmaggies rmaggies imaggies zmaggies uinvvar ginvvar rinvvar iinvvar zinvvar
The output is again to standard out and is of the form:
photoz coeff0 coeff1 coeff2
The same rules apply here as to the IDL code: call the code for many objects at a time, or else the overhead (about 3 seconds) will kill you.
As I noted above, I have not worked very hard to make the stand-alone C code or the C library very usable, as generally speaking that would have meant replicating features in IDL and C, and I wasn't sure how many people would use the pure C versions. If there is a need to do this, I might be convinced to -- or if YOU do it, we could include your stuff in the library.
kcorrect C library
Why would you bother incorporating the K-correction code into your code at all, rather than just calculating the K-corrections once and reading in the results? Well, perhaps you have to calculate 1/Vmax, in which for each object you have to check its K-correction at (nearly) arbitrary redshifts. It is best to calculate the K-corrections on the fly in these cases, so we provide ways of doing this through a shared object libary.
For example, the stand-alone C code uses the "libkcorrect.so" library (in fact, this is *exactly* the same library which is linked into IDL). If you have this library in your LD_LIBRARY_PATH and you include the header file "kcorrect.h" (in $KCORRECT_DIR/include), your own code can call it. The best way to figure out how to use it is to read the examples in fit_coeffs.c and reconstruct_maggies.c, but I'll try to explain the essentials here.
1.First, the code has to have the basic information about the templates. You will usually be using the "default" templates in $KCORRECT_DIR/data/templates, so you will have to load in the files:
$KCORRECT_DIR/data/templates/vmatrix.default.dat $KCORRECT_DIR/data/templates/lambda.default.dat
These files are in a special ASCII format which can be read into a "float*" variable in C using the routine k_load_ascii_table. For example:
k_load_ascii_table(&vmatrix,&ndim,&sizes,vmatrixfile);
loads in a set of data from vmatrixfile. The pointer to the data is returned in "vmatrix". The number of dimensions in the data is returned in "ndim". "sizes" is an array of size ndim giving the size of each dimension. The information in each of these files is the following:
- lambda.default.dat: The wavelength scale covered by the templates (the *edges*, not centers, of all pixels) [NLAMBDA+1]
- vmatrix.default.dat: The spectra spanning the SED space [NV,NLAMBDA]
2. Second, it is necessary to read in the filters. The list of filters is in the directory:
$KCORRECT_DIR/data/templates/sdss_filters.dat
This list is read by the routine "k_load_filters", which assumes all the filters are in the directory $KCORRECT_DIR/data/filters (or if KCORRECT_DIR is not set, in the current directory). This routine is called as:
k_load_filters(&filter_n,&filter_lambda,&filter_pass,&maxn,&nk,filterfile);
"nk" is the number of filters. "filter_n" is the number of points in each filter curve, "filter_lambda" is the wavelength scale of each filter, and "filter_pass" is the response curve of each filter. "maxn" is the maximum of "filter_n". This sets the indexing of "filter_lambda" and "filter_pass", so that to get the ith wavelength of the kth filter you access "filter_lambda[k*maxn+i]".
3. Third, you have to define the redshift range over which you are going to define a look-up table. I usually use 0. to 1., to be safe, with 1000 steps, using the code:
zvals=(float *) malloc(nz*sizeof(float)); for(i=0;i<nz;i++) zvals[i]=zmin+(zmax-zmin)*((float)i+0.5)/(float)nz;
where nz=1000, zmin=0., and zmax=1.
4. Finally, you have to define the look-up table which tabulates the projection of each basis element in the bmatrix onto each filter, at each redshift in zvals:
rmatrix=(float *) malloc(nz*nv*nk*sizeof(float));
k_projection_table(rmatrix,nk,nv,vmatrix,lambda,nl,zvals,nz,filter_n,
filter_lambda,filter_pass,band_shift,maxn);
One defines "nv" and "nl" based on the "sizes" returned by the k_load_ascii_table calls described above.
Once this overhead has been taken care of (and it is a significant amount of time --- a few seconds --- so only do it once!) you can use the routines k_fit_nonneg and k_reconstruct_maggies.
k_fit_nonneg is called as:
k_fit_nonneg(coeffs, rmatrix, nk, nv, zvals, nz, maggies, maggies_ivar,
redshift, ngalaxy, tolerance, maxiter, niter, chi2,
verbose)
The array coeffs should be of the size (nv*ngalaxy), and the coefficients are returned in it.
k_reconstruct_maggies is called as:
k_reconstruct_maggies(zvals, nz, rmatrix, nk, nv, coeffs, redshift,
reconstruct_maggies, ngalaxy)
where reconstruct_maggies are expressed in maggies.
The photo-z code can also be accessed through the C library libkcorrect.so. Instead of k_fit_coeffs, you will be calling k_fit_photoz, as in:
k_fit_photoz(photoz, coeffs, rmatrix, nk, nv, zprior, lprior, nprior,
zvals, nz, maggies, maggies_ivar, ngalaxy, tolerance,
maxiter, niter, chi2, verbose)
zprior, lprior, and nprior should contain the prior grid to use.
As I noted above, I have not worked very hard to make the stand-alone C code or the C library very usable, as generally speaking that would have meant replicating features in IDL and C, and I wasn't sure how many people would use the pure C versions. If there is a need to do this, I might be convinced to -- or if YOU do it, we could include your stuff in the library.
Filter curves
It is of course necessary to have filter responses to do any of this work. Thus, in kcorrect I have put together a fairly large library of filter curves in a homogeneous format. I have tried as far as possible to provide the provenance of all of these curves, though it has not always been possible to figure out.
The files are in a special SDSS format called FTCL described at the SDSS DR2 site. IDL code (in particular, yanny_readone()) exists in idlutils to read in such files.
It turns out that the SDSS filters do not have the responses they were designed to have, but Mamoru Doi has done the hard work of measuring them. Jim Gunn then took these, ran them through models of the telescope and the atmosphere (using an appropriately scaled model of the Palomar atmospheric model at airmass 1.3). Daniel Eisenstein deserves special mention for the extensive testing he has done on these. We have a repository of these and other curves. For kcorrect we use the SDSS files:
sdss_u0.par sdss_g0.par sdss_r0.par sdss_i0.par sdss_z0.par
The other SDSS files refer to the responses for each camcol.
Absolute calibration and its uncertainty in SDSS magnitudes
Our best estimate (from Eisenstein's work on hot white dwarfs) of the absolute calibration of SDSS data indicates that the magnitudes output by the SDSS pipeline code are not exactly on an AB system. There are offsets Delta m = m_AB - m_SDSS = [-0.036, 0.012, 0.010, 0.028, 0.040] in u, g, r, i, and z bands. Note these numbers have changed since v3_2. We have trained our templates on SDSS magnitudes corrected to this system. For the moment we recommend that the user apply these corrections to the magnitudes they are using.
In addition, we always apply minimum uncertainty of [0.05, 0.02, 0.02, 0.02, 0.03] mag in ugriz (the high level routines SDSS_KCORRECT, etc. already do so). This is because there is always an uncertainty in the calibration of any given galaxy, and we don't want the templates working super-hard to fit (say) some unrealistically small error bar in one band at the expense of the other bands.
Linear relationships between different band systems
Using this code on a number of SDSS galaxies, we have estimated the linear relationships between a number of different bandpass systems. The kcorrect paper has a table with these relationships.
Known problems
1. In v4_1_4 and earlier, Linux 64-bit boxes fail with the message:
ld -shared -o /u/rreyes/IDL/kcorrect/lib/libkcorrect.so idl_k_projection_table.o k_projection_table.o idl_k_fit_spec.o k_fit_spec.o idl_k_fit_nonneg.o k_fit_nonneg.o idl_k_fit_photoz.o k_fit_photoz.o idl_k_reconstruct_maggies.o k_reconstruct_maggies.o k_strparse.o k_yanny_readone.o k_filter_struct.o k_fileopen.o k_load_filters.o k_read_ascii_table.o k_nonneg_solve.o k_interpolate.o idl_k_binspec.o k_binspec.o k_utils.o k_qromo.o k_midpnt.o k_choldc.o k_cholsl.o k_locate.o k_brent.o k_evolve.o k_zbrent.o k_polint.o idl_lf_calc_vmax.o idl_ztoV.o idl_Vtoz.o idl_z2dm.o idl_z2t.o idl_t2z.o idl_lf_eep.o idl_lf_select_eep.o ztransform.o iterate_lf.o philike.o phierrors_lf.o gaussj.o lf_select_eep.o lf_calc_vmax.o lf_eep.o lf_eepfit.o lf_WH_interp.o lf_set_AB.o lf_sum_AB.o ld: k_projection_table.o: relocation R_X86_64_32 against `k_midpnt' can not be used when making a shared object; recompile with -fPIC k_projection_table.o: could not read symbols: Bad value make[1]: *** [/u/rreyes/IDL/kcorrect/lib/libkcorrect.so] Error 1
You want to edit the Linux section of kevilmake so it reads:
"Linux" ) SO_EXT=so X_CFLAGS="-fPIC -g -Wall" X_FFLAGS="-ffixed-line-length-none" SDSS_CFLAGS= X_LD_FLAGS=-shared ;;
Or just use the idlutils "evilmake" utility (if you are using the IDL version of kcorrect you need idlutils installed anyway).
2. There is a known problem with compiling the stand-alone C code on a Mac OS X 10.4 system or earlier, which results in the following error message:
cc -I/Users/blanton/kcorrect/include -I/Users/blanton/kcorrect/src -g -Wall -dy namic -O3 -DCHECK_LEAKS -o /Users/blanton/kcorrect/bin/fit_photoz fit_photoz.c - lm -L/Users/blanton/kcorrect/lib \ -lkcorrect ld: /Users/blanton/kcorrect/lib/libkcorrect.dylib is input for the dynamic link editor, is not relocatable by the static link editor again make: *** [/Users/blanton/kcorrect/bin/fit_photoz] Error 1 make: Target `all' not remade because of errors.
I am honestly not really sure what causes this. The library libkcorrect.dylib is fine for importing into IDL but NOT for importing into the stand-alone C code. If you want the library to be built correctly for importing into C programs, you can uncomment the two lines in the "kevilmake" script:
# Uncomment the following two lines to get .dylib libraries that can # be linked into C programs on a Mac OS X system # LD=libtool # X_LD_FLAGS="-dynamic -L/usr/lib -lc -lcc_dynamic"
In this case, the library will be fine for importing into C but no good for using from IDL. I can't make the Mac do both with a single library file. Sorry! If somebody knows how to do this, PLEASE let me know.
3. In Mac OS X 10.5 (Leopard), there occurred some changes to the gcc libraries, which require the following addition to "kevilmake" at the end of the "Darwin" section:
if [ `uname -r | cut -c 1` == 9 ]; then
X_LD_FLAGS="-bundle -flat_namespace -undefined suppress"
# Uncomment the following two lines to get .dylib libraries that can
# be linked into C programs on a Mac OS X system
# LD=libtool
# X_LD_FLAGS="-dynamic -L/usr/lib -lc "
fi
4. In v3_2 and earlier, fit_coeffs sends output to stdout occasionally of the form:
ERROR: choldc failed (sum==-1.197184e+06)
which screws up the inputs to reconstruct_maggies. If this affects you, change the line in k_choldc.c:
printf("ERROR: choldc failed (sum==%e)\n",sum);
to
fprintf(stderr, "ERROR: choldc failed (sum==%e)\n",sum);
5. In versions v1_10 and earlier, the code returned K-corrections to non-AB systems (oops!). Back in those days, Ivan Baldry pointed out that kcorrect returned non-AB magnitudes when it was requested to output K-corrections to shifted bandpasses. The sense of the error was that the K-correction was missing a term of −2.5 log10 (1+z) for a bandpass shift of z_0. This error occurred independent of bandpass, color, redshift, and anything else. Thus, when interpreting magnitudes returned by kcorrect v1_10 and earlier, please apply a correction of −2.5 log10 (1+z), where z_0 is the shift of the bandpass (eg. zb is the b-band shifted by z). Luckily this only is an error in the absolute measures of magnitude; colors are unchanged, the shapes of LFs are unchanged (just shifted in magnitude), etc. Many thanks to Ivan for pointing out the error.
6. In Mac OS 10.5 and greater, and IDL v7.0.4 and greater, kcorrect only works if "idl32" is used. This is because the compilation doesn't do handle the 64-bit stuff correctly yet. We're working on a fix!
7. The twomass_*.par filter curves are off by a factor of lambda, because apparently the 2MASS web site posted energy-counting filter curves (but called them photon-counting). Will be fixed in next release.
Acknowledgements
This work was funded in part by a GALEX Archival research grant (Program 38).
Sam Roweis provided irreplaceable help in teaching me the algorithms used in this code. David Hogg tracked down many of the bandpass responses given here. Ivan Baldry was critical to debugging the code. Hewett and Warren provided a bandpass response for the bj band. Thanks to Alison Coil, Jeff Newman, and Risa Wechsler for help understand and using the DEEP2 data. Thanks to Lexi Moustakas for help using the GOODS catalog -- and the GOODS team for access to that catalog. Thanks to John Moustakas for his code for reading in Bruzual-Charlot templates.
Funding for the Sloan Digital Sky Survey (SDSS) has been provided by the Alfred P. Sloan Foundation, the Participating Institutions, the National Aeronautics and Space Administration, the National Science Foundation, the U.S. Department of Energy, the Japanese Monbukagakusho, and the Max Planck Society. The SDSS Web site is http://www.sdss.org/.
The SDSS is managed by the Astrophysical Research Consortium (ARC) for the Participating Institutions. The Participating Institutions are The University of Chicago, Fermilab, the Institute for Advanced Study, the Japan Participation Group, The Johns Hopkins University, Los Alamos National Laboratory, the Max-Planck-Institute for Astronomy (MPIA), the Max-Planck-Institute for Astrophysics (MPA), New Mexico State University, University of Pittsburgh, Princeton University, the United States Naval Observatory, and the University of Washington.
This publication makes use of data products from the Two Micron All Sky Survey, which is a joint project of the University of Massachusetts and the Infrared Processing and Analysis Center/California Institute of Technology, funded by the National Aeronautics and Space Administration and the National Science Foundation.
The Galaxy Evolution Explorer (GALEX) is a NASA Small Explorer. The mission was developed in cooperation with the Centre National d'Etudes Spatiales of France and the Korean Ministry of Science and Technology.
DEEP2 is a collaboration between UC Santa Cruz and UC Berkeley. Funding for the DEEP2 survey has been provided by NSF grant AST-0071048 and AST-0071198. (Some of) the data presented herein were obtained at the W.M. Keck Observatory, which is operated as a scientific partnership among the California Institute of Technology, the University of California and the National Aeronautics and Space Administration. The Observatory was made possible by the generous financial support of the W.M. Keck Foundation. The DEEP2 team and Keck Observatory acknowledge the very significant cultural role and reverence that the summit of Mauna Kea has always had within the indigenous Hawaiian community and appreciate the opportunity to conduct observations from this mountain.
Frequently Asked Questions
Why do I get a compilation error in K_LOAD_FILTERS?
filter_str=yanny_readone(curr_path+'/'+filterlist[i],hdr=hdr, $ ^ % Syntax error. At: /Applications/kcorrect/pro/utils/k_load_filters.pro, Line 70 % 1 Compilation error(s) in module K_LOAD_FILTERS.
Because you do not have idlutils installed. Install the idlutils package (you don't need the idlspec2d or anything else described on the above web page).
kcorrect assumes h=1. How do I change that?
For calculating absolute magnitudes, kcorrect uses a Hubble constant H0 = 100h km/s/Mpc with h = 1, whereas in the actual Universe h = 0.7 or so. In order to convert the kcorrect outputs to your desired value of h you need to apply the term:
M(h) = M(h = 1) + 5log10h
In this formula M(h = 1) is the output of kcorrect and M(h) is the absolute magnitude you want. That is, for a fixed flux the actual absolute magnitudes are brighter (i.e. you have to subtract about 0.77 mag) than the kcorrect outputs, because the Hubble constant is smaller, and thus things are further away.
How can I project onto filter XXX?
; First enumerate the filters you want to project onto: fs = ['galex_FUV.par','bessell_U.par','bessell_B.par'] ; Then project the SEDs into the variable maggies k_projection_table, maggies, vm # coeffs, lambda, redshifts, fs, zmin=0,zmax=0,nz=1
lambda, vm and coeffs come from the routine kcorrect.
redshifts & fs are from you. zmin,zmax and nz are set so that you will be given the maggies at the redshifts you specify.--Konidaris 21:57, 15 September 2006 (EDT)
