Data OverviewThe WSA holds processed WFCAM data primarily originating from the five UKIDSS surveys: Large Area Survey, Galactic Plane Survey, Galactic Clusters Survey, Deep Extragalactic Survey and the Ultra Deep Survey (see the UKIDSS pages for survey descriptions). Non-survey data taken by WFCAM for registered open-time programmes are also archived.
The underlying content and schema of the archive are described below.
The WSA consists of a series of database releases. For information on these releases, content and sky coverge see the surveys page. More detailed information can be found in the papers accompanying the releases, see publications.
Users of the WSA data should note the limitations and known issues detailed under the release history.
The following sections discuss the structure and content of the WSA database. Further details can be found using the schema browser.
WSA - Detailed OverviewMultiframes
The primary component of the WSA is the multiframe which is represented by and stored on disk as a multi-extension FITS (MEF) file. The most common multiframe is a pipeline reduced WFCAM observation in a single passband. Typically the associated FITS file will contain a primary extension and four image extensions each representing one of the WFCAM detectors. If the multiframe has associated object catalogue data then these are held in a separate MEF file.
Users wishing to work with the flat-files directly (i.e. the FITS images and catalogues) might find the flat file page helpful.
Multiframes will exist for all observation types e.g. darks, flats, science etc and data products e.g. stacks, mosaics, difference images etc. It should be noted that the FITS image extensions are actually held as RICE compressed binary table extensions and that not all multiframes will have 4 four image extension e.g. mosaics.
Schematic diagram of a multiframe
Multiframes are described in the database by entries in three main tables. The multiframe table primarily holds the meta-data that are applicable to a multiframe as a whole rather than an individual detector e.g. basic observation details. These meta-data originate from the primary header of the MEF file. Every multiframe held in archive is represented by a single row in the multiframe table with the attributes/columns in the row holding the meta-data. The multiframeDetector table contains details applicable to individual detector frames that are part of a multiframe. The third main table, currentAstrometry, contains the current astrometric calibration coefficients (WCS) for each science detector frame that is again part of a multiframe. If a multiframe has four detectors/extensions there will be four corresponding entries (rows) in multiframeDetector and currentAstrometry. A schematic representation of a multiframe is shown above
Each multiframe is assigned a unique ID number multiframeID which is an attribute in many tables held in the archive including each of the three tables discussed above (multiframe, multiframeDetector and currentAstrometry). The multiframeID number is used, often in conjunction with the extension number attribute (extNum) to link the multiframe records held across the tables.
An example: A stacked J-band WFCAM science observation at a single pointing is generated by the pipeline which produces a MEF file that holds the image data and a MEF file holding the catalogue data. This observation constitutes a multiframe and is assigned a multiframeID, say 12345, at ingest into the archive. A single row entry is written into the multiframe table, attributes in the row include the multiframeID, the filename of the image MEF file, the filename of the catalogue MEF file, the number of detectors/extensions (in this case 4) and observational meta-data. For every detector/extension in the multiframe, rows are written into multiframeDetector and currentAstrometry tables. Each of the 4 rows written into these two tables contains the same multiframeID (12345) and the relevant extension number (numbered 2-5). So if we want to find the WCS coefficients for the 4th detector in this multiframe we would need to query the currentAstrometry table looking for a row that has multiframeID=12345 and extNum=5.
Schematic diagram of source merging via a frameSet
Taking the DXS as an example, the detections are held in dxsDetection, the source merging is recorded in dxsMergeLog and the merged sources placed in dxsSource. Imagine that three multiframes, taken in J, H, and K and covering approximately the same part of the sky, are produced by the pipeline. The multiframeIDs are say 12345, 23456 and 34567 respectively. Objects detected on each multiframe are loaded into dxsDetection. The multiframes are run through source merging which produces an entry in dxsMergeLog for each set of paired multiframe extensions. Typically the first extension in J will be paired with the first extensions of H and K. So looking at the pairing of the first extensions dxsMergeLog will contain an entry with a unique frameSetID, say 98765, which has attributes jmfID=12345, jeNum=2, hmfID=23456, heNum=2, kmfID=34567 and keNum=2. The records written into dxsSource by this pairing will all have frameSetID=98765. If we wanted to examine the full H band detection attributes of a given source in dxsSource that had say an hSeqNum=87654 we would first need know the extNum (e.g. 2) and H multiframeID (e.g. 23456), available by querying dxsMergeLog for frameSetID=98765 and then query dxsDetection for the entry corresponding to multiframeID=23456 extNum=2 and seqNum=87654.
The above description attempts to give users an overview of the tables they are most likely to use. The WSA browser should be used to further explore the contents and layout of the archive.
WSA - TablesThe main tables that users are likely to query for a given survey are the corresponding source and/or detection table
Source tables contain merged records for a given object in the passbands/epochs that define the survey. Detection tables contain full details on the individual passband/epoch measurements. The source and detection tables are linked via reference ID numbers held as parameters.
WSA - Source merging and seaming
Individual passband detections that come from WFCAM images and that are stored in the *Detection tables in the WSA are automatically merged into multi-colour, multi-epoch source lists stored in the *Source tables. The set of passbands (either different filters and/or repeat visits of the same filter) defined for each survey programme is predefined, and stored in the table RequiredFilters; that prescription is used to associate individual passbands/epochs into frame sets.
The core of the source merging algorithm uses a simple pairing procedure
between pairs of frames before producing the final merged list. For a given
frame set, frames are considered in pairs (say frames A and
B) moving from short to long wavelengths, and early to late epochs
where appropriate, and pairs are made by looking for the nearest detection
in B for every detection in A out to a maximum pairing
tolerance specified for each survey. The default pairing radius is
usually 2.0 arcsec; for certain programmes - e.g. the UKIDSS GPS, DXS
and UDS, the pairing radius is 1.0 arcsec. The pairing criterion for
each programme is stored in table PROGRAMME in attribute
pairingCriterion; note that this is in units of degrees. To
find out the pairing criteria in arcsec for the programmes that form part of
a released database product, simply use the following SQL:
The pairing is only considered as safe if the detection in B is the nearest to the detection in A, AND if the detection in A is the nearest to the detection in B: i.e. two sets of pointers of the nearest object in B to every object in A, and vice versa, are produced, and only consistent pair pointers are used to associate pairs of objects. If detection Y in B is the nearest as seen from the position of detection X in A, but detection Z in A is the nearest as seen from the position of Y in B, then the association is not made and detection X in A will not be paired to any in B.
Source merging then proceeds when all pair combinations of available passbands/epochs in the frame set have had consistent pointer sets produced. The merging procedure is simply to start with the shortest wavelength and/or earliest epoch detection set as a master set, and merge in the pairs pointed to by each of the slave pointer sets; subsequently, the next shortest wavelength/earliest epoch set is considered as the master, and all detections not already merged in previously are considered as slaves amongst the remaining passband sets. This process continues until only unmerged detections in the final passband set are left, and these are written into the merged source list. Hence, every detection amongst all the images in the frame set will have one, and only one, entry somewhere amongst the set of merged sources.
Note that the pairing radius (2 arcsec for LAS/GPS, 1 arcsec for
GPS, DXS and UDS) is large compared to the typical
astrometric errors (and may be too large for a given science application).
This is in order that moving sources (e.g. high proper
motion objects) and sources with unusually high centroiding errors
(e.g. very faint and/or extended, low surface brightness objects) will be
paired up in the merged source lists. The negative side to this is that
some level of spurious matchings will inevitably
occur. For well exposed, non-moving
sources, some indication of the reliability of the merged set can be
checked at usage (query) time by applying filters to the *Xi and
*Eta attributes in the *Source table. These attributes
provide the offset from master to slave for pairings, in arcsec, and should
not be larger than about 1 arcsec for non-moving, well exposed sources. If
your particular science application requires a more restrictive pairing
tolerance, then simply tune by selecting on *Xi and
*Eta at query time:
Seaming - the final stage in the source table creation is to perform
seaming which flags duplicate objects by populating the priOrSec (primary or secondary) flag.
Matching objects in overlap regions are ranked according to their filter coverage then their
quality error bit flags and finally their proximity to a detector edge. For example say the same source is found
on three overlapping framesets having framesetID = X, Y and Z. Framesets Y and Z have coverage in J,
H and K bands and X only has the J band. The quality error bit information (*ppErrBits) shows that
more confidence can be
placed in the source produced by frameset Y. The conclusion for this source is that the measurement taken
from frameset Y is the primary and those taken from X and Z are secondary. The priOrSec flag for
all three sources is set to framesetID Y. Sources that do not have duplicate measurements have
priOrSEc=0. The SQL where clause to select only primary sources (i.e. purge duplicates) is
WSA - Quality Error bit flaggingQuality error bit attributes in the Detection and Source tables flag the confidence of a given detection or source. The attributes and the meaning of the values assigned during WFAU post-processing are described on ppErrBits. Examples of how the quality error bit flags can be used in queries are given in the SQL cookbook.
WSA - MagnitudesKron and Petrosian magnitudes are underestimated for large galaxies, because the aperture is limited to the largest aperture radius, aperRad13 = 12 arcsec. This is the approximately the same size as the background estimation mesh, so larger apertures risk producing large errors due to poor background subtraction. There are a small number of bright objects which do have larger Kron and Petrosian radii. Most of these are flagged as saturated, but a small number (a few tens) have no errors. Generally these are isolated large, bright galaxies. The values kronRad and petroRad are limited to 15 pixels and 30 pixels for non-interleaved and 2x2 interleaved data respectively (apart from the small number of bright objects discussed above). This corresponds to 6 arcsec, not the 12 arcsec stated above. It should be noted that the value of kronRad is the first moment of the surface brightness profile and the value of petroRad is when the radius at which the surface brightness in an annulus is 0.2 times the mean surface brightness. The apertures in which the flux is calculated is twice this radius in both cases.
Home | Overview | Browser | Access | Login | Cookbook | nonSurvey
Listing | Region | MenuQuery | FreeSQL
Links | Credits
WFAU, Institute for Astronomy, firstname.lastname@example.org