Introduction to Data Access with the Butler

Contact author(s): Alex Drlica-Wagner
Last verified to run: 2024-08-16
LSST Science Pipelines version: Weekly 2024_32
Container Size: large
Targeted learning level: beginner

Description: This notebook uses the Butler to query DP0 images and catalogs.

Skills: Query and retrieve images and catalog data with the Butler.

LSST Data Products: Images (calexp, goodSeeingDiff_differenceExp, deepCoadd) and catalogs (sourceTable, diaSourceTable, objectTable, forcedSourceTable, forcedSourceOnDiaObjectTable).

Packages: lsst.daf.butler

Credit: This tutorial was originally developed by Alex Drlica-Wagner.

Get Support: Find DP0-related documentation and resources at dp0.lsst.io. Questions are welcome as new topics in the Support - Data Preview 0 Category of the Rubin Community Forum. Rubin staff will respond to all questions posted there.

1. Introduction

The Butler is the LSST Science Pipelines interface for managing, reading, and writing datasets. The Butler can be used to explore the contents of the DP0 data repository and access the DP0 data. The current version of the Butler (referred to as "Gen3") is still under development, and this notebook may be modified in the future. Full Butler documentation can be found here.

This notebook demonstrates how to:

1. Create an instance of the Butler
   
2. Use the Butler to retrieve image data
   
3. Use the Butler to retrieve catalog data
   

1.1. Import packages

Import general python packages and several packages from the LSST Science Pipelines, including the Butler package and AFW Display, which will be used to display images. More details and techniques regarding image display with AFW Display can be found in the rubin-dp0 GitHub Organization's tutorial-notebooks repository.

# Generic python packages
import pylab as plt
import gc
import numpy as np

# LSST Science Pipelines (Stack) packages
import lsst.daf.butler as dafButler
import lsst.afw.display as afwDisplay

# Set a standard figure size to use
plt.rcParams['figure.figsize'] = (8.0, 8.0)
afwDisplay.setDefaultBackend('matplotlib')

2. Create an instance of the Butler

To create the Butler, we need to provide it with a configuration for the data set (often referred to as a "data repository"). This configuration can be the path to a yaml file (often named butler.yaml), a directory path (in which case the Butler will look for a butler.yaml file in that directory), or a shorthand repository label (i.e., dp02). If no configuration is passed, the Butler will use a default value (this is not recommended in most cases). For the purposes of accessing the DP0.2 data, we will use the dp02 label.

In addition to the config, the Butler also takes a set of data collections. Collections are lightweight groups of self-consistent datasets such as raw images, calibration files, reference catalogs, and the outputs of a processing run. You can find out more about collections here. For the purposes of accessing the DP0.2 data, we will use the 2.2i/runs/DP0.2 collection. Here, '2.2i' refers to the imSim run that was used to generated the simulated data and 'runs' refers to the fact that it is processed data.

Create an instance of the Butler pointing to the DP0.2 repository.

config = 'dp02'
collections = '2.2i/runs/DP0.2'
butler = dafButler.Butler(config, collections=collections)

Learn more about the Butler by uncommenting the following line and executing the cell.

# help(butler)

3. Butler image access

The Butler can be used to access DP0 image data products. The DP0.2 Data Products Definition Document (DPDD) describes three different types of image data products: processed visit images (PVIs; calexp), difference images (goodSeeingDiff_differenceExp), and coadded images (deepCoadd). We will demonstrate how to access each of these.

In order to access a data product through the Butler, we will need to tell the Butler what data we want to access. This call generally has two components: the datasetType tells the Butler what type of data product we are seeking (e.g., deepCoadd, calexp, objectTable), and the dataId is a dictionary-like identifier for a specific data product (more info on dataIds can be found here).

3.1. Processed Visit Images

Processed visit images can be accessed with the calexp datasetType. These are image data products derived from processing of a single detector in a single visit of the LSST Camera. To access a calexp, the minimal information that we will need to provide is the visit number and the detector number.

datasetType = 'calexp'
dataId = {'visit': 192350, 'detector': 175}
calexp = butler.get(datasetType, dataId=dataId)

To view all parameters that can be passed to the dataId for calexp, we can use the Butler registry.

print(butler.registry.getDatasetType(datasetType))

DatasetType('calexp', {band, instrument, detector, physical_filter, visit_system, visit}, ExposureF)

Plot a calexp.

fig = plt.figure()
display = afwDisplay.Display(frame=fig)
display.scale('asinh', 'zscale')
display.mtv(calexp.image)
plt.show()

Figure 1: The calexp image displayed in grayscale, with scale bar at right and axes labels in pixels.

Clean up.

del calexp
gc.collect()

9

3.2. Difference Images

Difference images can be accessed with the goodSeeingDiff_differenceExp datasetType. These are PVIs which have had a template image subtracted from them. These data products are used to measure time-varying difference in the fluxes of astronomical sources. To access a difference image we require the visit and detector.

datasetType = 'goodSeeingDiff_differenceExp'
dataId = {'visit': 192350, 'detector': 175}
diffexp = butler.get(datasetType, dataId=dataId)

Plot a goodSeeingDiff_differenceExp.

fig = plt.figure()
display = afwDisplay.Display(frame=fig)
display.scale('asinh', 'zscale')
display.mtv(diffexp.image)
plt.show()

Figure 2: The difference image displayed in grayscale, with scale bar at right and axes labels in pixels. Compared to the calexp in Figure 1, only a few point sources are obvious in the difference image.

Clean up.

del diffexp
gc.collect()

378

3.3. Coadded Images

Coadded images combine multiple PVIs to assemble a deeper image; they can be accessed with the deepCoadd datasetType. Coadd images are divided into “tracts” (a spherical convex polygon) and tracts are divided into “patches” (a quadrilateral sub-region, with a size in pixels chosen to fit easily into memory on desktop computers). Coadd patches are roughly the same size as a single-detector calexp image. To access a deepCoadd, we need to specify the tract, patch, and band that we are interested in.

datasetType = 'deepCoadd'
dataId = {'tract': 4431, 'patch': 17, 'band': 'i'}
coadd = butler.get(datasetType, dataId=dataId)

Plot a deepCoadd.

fig = plt.figure()
display = afwDisplay.Display(frame=fig)
display.scale('asinh', 'zscale')
display.mtv(coadd.image)
plt.show()

Figure 3: A deepCoadd image displayed in grayscale, with scale bar at right and axes labels in pixels. Compared to a calexp in Figure 1, many more objects are visible for two reasons: one, this is a deeper image and two, there are more objects to see because the location of a rich cluster was chosen.

Clean up.

del coadd
gc.collect()

859

4. Butler table access

While the preferred technique to access DP0 catalogs is through the table access protocol (TAP) service, the Butler can also provide access to these data products. We will demonstrate how to access several different catalog products described in the DPDD. The catalogs are returned by the Butler as pandas DataFrame objects, which can be further manipulated by the user. The full descriptions of the column schema from the DP0.2 tables can be found here.

4.1. Processed Visit Sources

The sourceTable provides astrometric and photometric measurements for sources detected in the individual PVIs (calexp). Thus, to access the sourceTable for a specific PVI, we require similar information as was used to access the calexp. More information on the columns of the sourceTable can be found here.

datasetType = 'sourceTable'
dataId = {'visit': 192350, 'detector': 175}
src = butler.get(datasetType, dataId=dataId)
print(f"Retrieved catalog of {len(src)} sources.")

Retrieved catalog of 2548 sources.

Display the table.

src

	coord_ra	coord_dec	ccdVisitId	parentSourceId	x	y	xErr	yErr	ra	decl	...	hsmShapeRegauss_flag_not_contained	hsmShapeRegauss_flag_parent_source	sky_source	detect_isPrimary	band	instrument	detector	physical_filter	visit_system	visit
sourceId
103267213875609601	52.982242	-33.905601	192350175	0	302.000000	7.000000	NaN	NaN	52.982242	-33.905601	...	False	False	False	True	i	LSSTCam-imSim	175	i_sim_1.4	1	192350
103267213875609602	53.024289	-33.917699	192350175	0	967.000000	5.000000	NaN	NaN	53.024289	-33.917699	...	False	False	False	True	i	LSSTCam-imSim	175	i_sim_1.4	1	192350
103267213875609603	53.112693	-33.943320	192350175	0	2366.000000	5.000000	NaN	NaN	53.112693	-33.943320	...	False	False	False	True	i	LSSTCam-imSim	175	i_sim_1.4	1	192350
103267213875609604	53.152837	-33.954927	192350175	0	3001.000000	5.000000	NaN	NaN	53.152837	-33.954927	...	False	False	False	True	i	LSSTCam-imSim	175	i_sim_1.4	1	192350
103267213875609605	53.110035	-33.942728	192350175	0	2325.000000	8.000000	NaN	NaN	53.110035	-33.942728	...	False	False	False	True	i	LSSTCam-imSim	175	i_sim_1.4	1	192350
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
103267213875612144	53.089672	-34.165696	192350175	103267213875611492	3361.291218	3904.733326	0.002749	0.002765	53.089672	-34.165696	...	False	False	False	True	i	LSSTCam-imSim	175	i_sim_1.4	1	192350
103267213875612145	53.090007	-34.166623	192350175	103267213875611492	3371.512003	3918.880046	0.153121	0.141199	53.090007	-34.166623	...	False	False	False	True	i	LSSTCam-imSim	175	i_sim_1.4	1	192350
103267213875612146	53.089140	-34.164681	192350175	103267213875611492	3347.792883	3890.073242	0.428516	0.395548	53.089140	-34.164681	...	False	False	False	True	i	LSSTCam-imSim	175	i_sim_1.4	1	192350
103267213875612147	52.887434	-34.105846	192350175	103267213875611493	159.495789	3883.742944	0.203543	0.283224	52.887434	-34.105846	...	False	False	False	True	i	LSSTCam-imSim	175	i_sim_1.4	1	192350
103267213875612148	52.886719	-34.106332	192350175	103267213875611493	152.334559	3895.556815	0.354580	0.409233	52.886719	-34.106332	...	False	False	False	True	i	LSSTCam-imSim	175	i_sim_1.4	1	192350

2548 rows × 143 columns

Clean up.

del src
gc.collect()

15

4.2. Difference Image Sources

The diaSourceTable provides astrometric and photometric measurements for sources detected in the difference images. To access the diaSourceTable for a specific difference image, we require similar information as was used to access the difference image itself. However, the diaSourceTable groups together all sources detected in a single visit, and thus the detector number is not used. More information on the columns of the diaSourceTable can be found here.

datasetType = 'diaSourceTable'
dataId = {'visit': 192350}
dia_src = butler.get(datasetType, dataId=dataId)
print(f"Retrieved catalog of {len(dia_src)} DIA sources.")

Retrieved catalog of 11076 DIA sources.

Display the table.

dia_src

	diaSourceId	ccdVisitId	filterName	diaObjectId	ssObjectId	parentDiaSourceId	midPointTai	pixelId	bboxSize	ra	...	psfFlux_flag_edge	forced_PsfFlux_flag	forced_PsfFlux_flag_noGoodPixels	forced_PsfFlux_flag_edge	shape_flag	shape_flag_unweightedBad	shape_flag_unweighted	shape_flag_shift	shape_flag_maxIter	shape_flag_psf
0	103267119923200091	192350000	i	0	0	0	59837.380094	9874940695446	28	53.290976	...	True	True	False	True	True	True	False	False	False	False
1	103267119923200092	192350000	i	0	0	0	59837.380094	9874940716674	47	53.295917	...	False	False	False	False	True	True	False	False	False	False
2	103267119923200093	192350000	i	0	0	0	59837.380094	9874940720895	39	53.298530	...	False	False	False	False	False	False	False	False	False	False
3	103267119923200094	192350000	i	0	0	0	59837.380094	9874994883937	37	53.115455	...	False	False	False	False	False	False	False	False	False	False
4	103267119923200095	192350000	i	0	0	0	59837.380094	9874994879068	25	53.125167	...	False	False	False	False	True	True	False	False	False	False
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
83	103267220854931671	192350188	i	0	0	0	59837.380094	9873466265914	21	54.106765	...	False	False	False	False	True	True	False	True	False	False
84	103267220854931672	192350188	i	0	0	0	59837.380094	9872330137867	23	53.937256	...	False	False	False	False	True	False	False	True	False	False
85	103267220854931673	192350188	i	0	0	0	59837.380094	9872330452124	23	53.896168	...	False	False	False	False	True	True	False	True	False	False
86	103267220854931674	192350188	i	0	0	0	59837.380094	9872330419483	36	53.892027	...	True	True	False	True	True	True	False	False	False	False
87	103267220854931675	192350188	i	0	0	0	59837.380094	9873465306633	58	54.010445	...	True	True	False	True	True	False	True	False	False	False

11076 rows × 65 columns

To retrieve sources from a specific detector, we can select a subset of the sources returned by our query to the diaSourceTable. In particular, we note that the ccdVisitId column contains a value that combines the visit and detector IDs. We could build this value with some simple Python string formatting; however, the more robust way is to ask the Butler what this value should be.

dataId = {'visit': 192350, 'detector': 5}
det_string = str(dataId['detector'])
ccd_visit_id = np.longlong(str(dataId['visit'])+det_string.rjust(3, '0'))
selection = dia_src['ccdVisitId'] == ccd_visit_id
dia_src_ccd = dia_src[selection]
print(f"Found catalog of {len(dia_src_ccd)} DIA sources associated with {dataId}.")

Found catalog of 57 DIA sources associated with {'visit': 192350, 'detector': 5}.

Display the table selection.

dia_src_ccd

	diaSourceId	ccdVisitId	filterName	diaObjectId	ssObjectId	parentDiaSourceId	midPointTai	pixelId	bboxSize	ra	...	psfFlux_flag_edge	forced_PsfFlux_flag	forced_PsfFlux_flag_noGoodPixels	forced_PsfFlux_flag_edge	shape_flag	shape_flag_unweightedBad	shape_flag_unweighted	shape_flag_shift	shape_flag_maxIter	shape_flag_psf
0	103267122607554649	192350005	i	0	0	0	59837.380094	9874526984351	24	53.705277	...	True	True	False	True	False	False	False	False	False	False
1	103267122607554650	192350005	i	0	0	0	59837.380094	9874524670578	79	53.559254	...	False	False	False	False	True	False	True	False	False	False
2	103267122607554651	192350005	i	0	0	0	59837.380094	9874527507841	23	53.625081	...	False	False	False	False	True	True	False	False	False	False
3	103267122607554652	192350005	i	0	0	0	59837.380094	9874527515401	36	53.622959	...	False	False	False	False	True	True	False	False	False	False
4	103267122607554653	192350005	i	0	0	0	59837.380094	9874524682210	17	53.561687	...	False	False	False	False	True	True	False	False	False	False
5	103267122607554654	192350005	i	0	0	0	59837.380094	9874526776705	41	53.741219	...	False	False	False	False	True	True	False	False	False	False
6	103267122607554655	192350005	i	0	0	0	59837.380094	9874524939667	29	53.555735	...	False	False	False	False	True	True	False	False	False	False
7	103267122607554656	192350005	i	0	0	0	59837.380094	9874524941315	13	53.555215	...	False	False	False	False	True	True	False	False	False	False
8	103267122607554657	192350005	i	0	0	0	59837.380094	9874524221629	36	53.538756	...	False	False	False	False	True	False	True	False	False	False
9	103267122607554658	192350005	i	0	0	0	59837.380094	9874524975215	36	53.559786	...	False	False	False	False	True	False	True	False	False	False
10	103267122607554659	192350005	i	0	0	0	59837.380094	9874528156170	27	53.666651	...	False	False	False	False	False	False	False	False	False	False
11	103267122607554660	192350005	i	0	0	0	59837.380094	9874527303574	21	53.677040	...	False	False	False	False	True	True	False	False	False	False
12	103267122607554661	192350005	i	0	0	0	59837.380094	9874528204083	23	53.653415	...	False	False	False	False	True	False	True	True	False	False
13	103267122607554662	192350005	i	0	0	0	59837.380094	9874525117079	17	53.556548	...	False	False	False	False	True	True	False	False	False	False
14	103267122607554663	192350005	i	0	0	0	59837.380094	9874524916687	17	53.573818	...	False	False	False	False	True	True	False	False	False	False
15	103267122607554664	192350005	i	0	0	0	59837.380094	9874528767943	29	53.757296	...	False	False	False	False	False	False	False	False	False	False
16	103267122607554665	192350005	i	0	0	0	59837.380094	9874528010765	46	53.601382	...	False	False	False	False	True	False	True	True	False	False
17	103267122607554666	192350005	i	0	0	0	59837.380094	9874527461374	23	53.682423	...	False	False	False	False	False	False	False	False	False	False
18	103267122607554667	192350005	i	0	0	0	59837.380094	9874526706345	69	53.722268	...	False	False	False	False	True	True	False	False	False	False
19	103267122607554668	192350005	i	0	0	0	59837.380094	9874528595965	13	53.738082	...	False	False	False	False	True	True	False	False	False	False
20	103267122607554669	192350005	i	0	0	0	59837.380094	9874527246751	77	53.691537	...	False	False	False	False	True	True	False	False	False	False
21	103267122607554670	192350005	i	0	0	0	59837.380094	9874524444625	38	53.581936	...	False	False	False	False	True	True	False	True	False	False
22	103267122607554671	192350005	i	0	0	0	59837.380094	9874524543230	26	53.554380	...	False	False	False	False	True	True	False	False	False	False
23	103267122607554672	192350005	i	0	0	0	59837.380094	9874525405055	13	53.711015	...	False	False	False	False	True	True	False	False	False	False
24	103267122607554673	192350005	i	0	0	0	59837.380094	9874524593178	27	53.563518	...	False	False	False	False	True	True	False	False	False	False
25	103267122607554674	192350005	i	0	0	0	59837.380094	9874527857583	49	53.615334	...	False	False	False	False	True	False	True	False	False	False
26	103267122607554675	192350005	i	0	0	0	59837.380094	9874491137133	13	53.516487	...	False	False	False	False	True	True	False	False	False	False
27	103267122607554676	192350005	i	0	0	0	59837.380094	9874525809094	21	53.619741	...	False	False	False	False	True	True	False	False	False	False
28	103267122607554677	192350005	i	0	0	0	59837.380094	9874524347782	21	53.569086	...	False	False	False	False	True	False	True	False	False	False
29	103267122607554678	192350005	i	0	0	0	59837.380094	9874491288265	37	53.532155	...	False	False	False	False	True	True	False	False	False	False
30	103267122607554679	192350005	i	0	0	0	59837.380094	9874525938018	22	53.633760	...	False	False	False	False	True	True	False	False	False	False
31	103267122607554680	192350005	i	0	0	0	59837.380094	9874525943026	16	53.631645	...	False	False	False	False	True	True	False	False	False	False
32	103267122607554681	192350005	i	0	0	0	59837.380094	9874525747007	13	53.619370	...	False	False	False	False	True	True	False	False	False	False
33	103267122607554682	192350005	i	0	0	0	59837.380094	9874525753056	43	53.592022	...	False	False	False	False	True	True	False	False	False	False
34	103267122607554683	192350005	i	0	0	0	59837.380094	9874493971201	19	53.552785	...	False	False	False	False	True	True	False	False	False	False
35	103267122607554684	192350005	i	0	0	0	59837.380094	9874494010509	20	53.498763	...	False	False	False	False	True	True	False	False	False	False
36	103267122607554685	192350005	i	0	0	0	59837.380094	9874494148810	15	53.538073	...	False	False	False	False	True	False	True	False	False	False
37	103267122607554686	192350005	i	0	0	0	59837.380094	9874525220777	30	53.663914	...	False	False	False	False	False	False	False	False	False	False
38	103267122607554687	192350005	i	0	0	0	59837.380094	9874529438090	15	53.709646	...	False	False	False	False	True	True	False	False	False	False
39	103267122607554688	192350005	i	0	0	0	59837.380094	9874529354556	15	53.672492	...	False	False	False	False	True	True	False	False	False	False
40	103267122607554689	192350005	i	0	0	0	59837.380094	9874493007165	32	53.552607	...	False	False	False	False	False	False	False	False	False	False
41	103267122607554690	192350005	i	0	0	0	59837.380094	9874493554923	22	53.573046	...	False	False	False	False	False	False	False	False	False	False
42	103267122607554691	192350005	i	0	0	0	59837.380094	9874494674074	25	53.496340	...	False	False	False	False	True	True	False	False	False	False
43	103267122607554692	192350005	i	0	0	0	59837.380094	9874493560452	14	53.584402	...	False	False	False	False	True	True	False	False	False	False
44	103267122607554693	192350005	i	0	0	0	59837.380094	9874493455282	15	53.562149	...	False	False	False	False	False	False	False	False	False	False
45	103267122607554694	192350005	i	0	0	0	59837.380094	9874529345152	17	53.683894	...	False	False	False	False	True	True	False	False	False	False
46	103267122607554695	192350005	i	0	0	0	59837.380094	9874529423041	13	53.699500	...	False	False	False	False	True	True	False	False	False	False
47	103267122607554696	192350005	i	0	0	0	59837.380094	9874493604092	20	53.593499	...	False	False	False	False	False	False	False	False	False	False
48	103267122607554697	192350005	i	0	0	0	59837.380094	9874494543697	29	53.481376	...	False	False	False	False	False	False	False	False	False	False
49	103267122607554698	192350005	i	0	0	0	59837.380094	9874493336664	26	53.531567	...	False	False	False	False	False	False	False	False	False	False
50	103267122607554699	192350005	i	0	0	0	59837.380094	9874492662893	16	53.623050	...	False	False	False	False	True	True	False	False	False	False
51	103267122607554700	192350005	i	0	0	0	59837.380094	9874494733667	28	53.645560	...	False	False	False	False	True	True	False	False	False	False
52	103267122607554701	192350005	i	0	0	0	59837.380094	9874494743504	15	53.634130	...	False	False	False	False	False	False	False	False	False	False
53	103267122607554702	192350005	i	0	0	0	59837.380094	9874493532196	21	53.577616	...	False	False	False	False	True	True	False	False	False	False
54	103267122607554703	192350005	i	0	0	0	59837.380094	9874493689575	26	53.512649	...	False	False	False	False	False	False	False	False	False	False
55	103267122607554704	192350005	i	0	0	0	59837.380094	9874494840765	20	53.599586	...	False	False	False	False	True	True	False	False	False	False
56	103267122607554705	192350005	i	0	0	0	59837.380094	9874495112552	17	53.572470	...	False	False	False	False	True	True	False	False	False	False

57 rows × 65 columns

Clean up.

del dia_src, selection, dia_src_ccd
gc.collect()

69

4.3. Coadd Objects

The objectTable provides astrometric and photometric measurements for objects detected in coadded images. These objects are assembled across the set of coadd images in all bands, and thus contain multi-band photometry. For this reason, we do not specify the band in the dataId. More information on the columns of the objectTable can be found here.

datasetType = 'objectTable'
dataId = {'tract': 4431, 'patch': 17}
obj = butler.get(datasetType, dataId=dataId)
print(f"Retrieved catalog of {len(obj)} objects.")

Retrieved catalog of 37026 objects.

obj

column	coord_dec	detect_isDeblendedSource	footprintArea	shape_xx	merge_peak_sky	refExtendedness	sky_object	deblend_nChild	detect_fromBlend	x	...	u_kronFlux_flag	u_kronFlux_flag_bad_radius	u_kronFlux_flag_bad_shape	u_kronFlux_flag_bad_shape_no_psf	u_kronFlux_flag_edge	u_kronFlux_flag_no_fallback_radius	u_kronFlux_flag_no_minimum_radius	u_kronFlux_flag_small_radius	u_kronFlux_flag_used_minimum_radius	u_kronFlux_flag_used_psf_radius
objectId
1909948454470156289	-32.321901	True	85	11.079702	False	NaN	False	1	False	12022.000000	...	True	False	True	False	True	False	False	False	False	False
1909948454470156290	-32.317513	False	46337	10.072662	False	1.0	False	71	False	12709.547949	...	False	False	False	False	False	False	False	False	False	False
1909948454470156292	-32.321009	False	26388	7.936708	False	NaN	False	34	False	13088.000000	...	True	False	True	False	False	False	False	False	False	False
1909948454470156293	-32.321966	True	170	9.769333	False	NaN	False	1	False	13725.000000	...	True	False	True	False	True	False	False	False	False	False
1909948454470156294	-32.321188	False	471	3.956755	False	NaN	False	2	False	13741.000000	...	True	False	True	False	False	False	False	False	False	False
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
1909948454470206951	-32.192869	False	132	5.962306	False	1.0	False	0	False	13972.564211	...	False	False	False	False	False	False	False	False	False	False
1909948454470206952	-32.151663	False	144	5.779773	False	1.0	False	0	False	14718.496074	...	False	False	False	False	False	False	False	False	False	False
1909948454470206953	-32.134073	False	132	5.545843	False	1.0	False	0	False	12685.637631	...	False	True	False	False	False	False	False	False	False	True
1909948454470206954	-32.128051	False	132	3.886748	False	1.0	False	0	False	15173.540565	...	False	False	False	False	False	False	False	False	False	False
1909948454470206955	-32.108361	False	121	4.020427	False	0.0	False	0	False	12332.266190	...	False	False	False	False	False	False	False	False	False	False

37026 rows × 990 columns

del obj
gc.collect()

717

4.4. Difference Image Objects

The diaObjectTable contains derived summary parameters for DIA sources associated by sky location and includes lightcurve statistics (e.g., flux chi2, Stetson J). The DIA objects can be accessed from the diaObjectTable_tract table. As implied by the table name, it is intended to be queried by coadd tract. More information on the diaObjectTable_tract can be found here.

datasetType = 'diaObjectTable_tract'
dataId = {'tract': 4431}
dia_obj = butler.get(datasetType, dataId=dataId)
print(f"Retrieved catalog of {len(dia_obj)} DIA objects.")

Retrieved catalog of 282493 DIA objects.

dia_obj

	ra	decl	nDiaSources	radecTai	pixelId	uPSFluxLinearSlope	uPSFluxLinearIntercept	uPSFluxMAD	uPSFluxMaxSlope	uPSFluxErrMean	...	yPSFluxPercentile05	yPSFluxPercentile25	yPSFluxPercentile50	yPSFluxPercentile75	yPSFluxPercentile95	yPSFluxSigma	yTOTFluxSigma	yPSFluxSkew	yPSFluxChi2	yPSFluxStetsonJ
diaObjectId
1909798920888778753	56.367088	-32.569499	45	61371.287075	9.871410e+12	NaN	NaN	0.000000	NaN	823.215806	...	-422402.479529	-202970.392838	71319.715526	345609.823890	565041.910582	775809.582547	849924.603804	NaN	9.613949e+04	263.001684
1909798920888778754	56.338145	-32.602724	301	61404.195949	9.871402e+12	-0.295242	2.100552e+04	2721.957679	1.684604e+07	1514.241454	...	-16163.967943	-8203.173356	-3057.039271	1406.995815	7967.834623	8945.527745	46264.248565	-0.251146	1.039003e+02	2.629522
1909798920888778755	56.398382	-32.610126	138	61391.080217	9.871410e+12	16.969156	-1.019156e+06	6211.722714	6.124528e+06	2090.914702	...	-65848.213696	-39087.659394	-34028.558271	-26721.370037	-23367.402022	14972.417082	409058.536739	-0.761950	1.833847e+02	1.409964
1909798920888778756	56.313494	-32.650287	1	59583.123688	9.871402e+12	NaN	NaN	0.000000	NaN	200.572532	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
1909798920888778757	56.448778	-32.715438	17	61404.195949	9.871401e+12	-4.963484	2.986091e+05	2417.566412	-4.963484e+00	470.548476	...	-11313.129159	-11313.129159	-11313.129159	-11313.129159	-11313.129159	NaN	NaN	NaN	6.744220e-31	NaN
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
1910221133353851309	54.784432	-31.424229	1	61394.180318	9.878122e+12	NaN	NaN	NaN	NaN	NaN	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
1910221133353851310	54.786305	-31.424365	1	61394.180318	9.878122e+12	NaN	NaN	NaN	NaN	NaN	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
1910221133353851311	54.856538	-31.374838	1	61394.180318	9.878113e+12	NaN	NaN	NaN	NaN	NaN	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
1910221133353851312	54.861137	-31.407285	1	61394.180318	9.878113e+12	NaN	NaN	NaN	NaN	NaN	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
1910221133353851313	54.925328	-31.344368	1	61394.180318	9.878112e+12	NaN	NaN	NaN	NaN	NaN	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN

282493 rows × 137 columns

del dia_obj
gc.collect()

561

4.5. Forced Photometry Sources

Forced photometry refers to the process of fitting for a source at a specific location in an image regardless of whether the source has been detected in that image. This is useful when objects are not detected in all bands (e.g., drop outs) or observations (e.g., transient or variable objects). The forcedSourceTable contains forced photometry performed on the individual PVIs at the locations of all detected objects and linked to the objectTable. In contrast, the forcedSourceOnDiaObjectTable contains forced photometry on the individual PVIs at the locations of all objects in the diaObjectTable. Note that the tables returned by these butler queries are quite large and can fill up the memory available to your notebook.

More information on the columns of the forcedSourceTable can be found here, and the columns of the forcedSourceOnDiaObjectTable are similar.

Warning: forcedSourceTable takes dataIds comprised of tract and patch only, which returns too many sources for a Jupyter Notebook with a small or medium container.

Instead of forcedSourceTable, it is recommended to use forcedSource (which takes dataIds comprised of e.g., band, instrument, detector, and visit) and to apply spatial and temporal constraints whenever applicable. This is considered intermediate-level use of the Butler, and so is demonstrated in DP0.2 tutorial notebook 04b.

Only uncomment and execute the three cells below if you are using a large container.

# datasetType = 'forcedSourceTable'
# dataId = {'tract': 4431, 'patch': 16, 'band':'i'}
# forced_src = butler.get(datasetType, dataId=dataId)
# print(f"Retrieved catalog of {len(forced_src)} forced sources.")

# forced_src

# del forced_src
# gc.collect()

Query the forcedSourceOnDiaObjectTable. This will execute with a medium container.

datasetType = 'forcedSourceOnDiaObjectTable'
dataId = {'tract': 4431, 'patch': 16}
dia_forced_src = butler.get(datasetType, dataId=dataId)
print(f"Retrieved catalog of {len(dia_forced_src)} DIA forced sources.")

Retrieved catalog of 2097144 DIA forced sources.

dia_forced_src

	diaObjectId	parentObjectId	coord_ra	coord_dec	ccdVisitId	band	psfFlux	psfFluxErr	psfFlux_flag	psfDiffFlux	...	pixelFlags_cr	pixelFlags_bad	pixelFlags_suspect	pixelFlags_interpolatedCenter	pixelFlags_saturatedCenter	pixelFlags_crCenter	pixelFlags_suspectCenter	skymap	tract	patch
forcedSourceOnDiaObjectId
340230178712071314	1909939658377134081	0	55.790364	-32.270040	633728092	g	NaN	NaN	False	NaN	...	False	False	False	False	False	False	False	DC2	4431	16
340230178712071315	1909939658377134082	0	55.792233	-32.292355	633728092	g	NaN	NaN	False	NaN	...	False	False	False	True	True	False	False	DC2	4431	16
340230178712071316	1909939658377134083	0	55.809149	-32.316398	633728092	g	NaN	NaN	False	NaN	...	False	False	False	True	True	False	False	DC2	4431	16
340230178712071317	1909939658377134108	0	55.791854	-32.291598	633728092	g	NaN	NaN	False	NaN	...	False	False	False	False	False	False	False	DC2	4431	16
340230178712071318	1909939658377134119	0	55.809368	-32.316248	633728092	g	NaN	NaN	False	NaN	...	False	False	False	True	True	False	False	DC2	4431	16
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
348223638446538939	1909939658377138596	0	55.828977	-32.113390	648617071	z	NaN	NaN	False	NaN	...	False	False	False	False	False	False	False	DC2	4431	16
348223638446538940	1909939658377138602	0	55.786140	-32.095704	648617071	z	NaN	NaN	False	NaN	...	False	False	False	False	False	False	False	DC2	4431	16
348223638446538941	1909939658377138673	0	55.830825	-32.123568	648617071	z	NaN	NaN	False	NaN	...	False	False	False	False	False	False	False	DC2	4431	16
348223638446538942	1909939658377138683	0	55.817763	-32.095889	648617071	z	NaN	NaN	False	NaN	...	False	False	False	False	False	False	False	DC2	4431	16
348223638446538943	1909939658377138715	0	55.793946	-32.123654	648617071	z	NaN	NaN	False	NaN	...	False	False	False	False	False	False	False	DC2	4431	16

2097144 rows × 35 columns

del dia_forced_src
gc.collect()

366

5. Summary

In this notebook we demonstrated Butler access for a set of image and catalog data products described in the DPDD for DP0.2. However, we have not demonstrated the powerful capability of the Butler to query the holdings of a data repository. The full power of the Butler can be found in the Butler documentation, or in DP0.2 tutorial notebook 04b "Intermediate Butler Queries".