Introduction to the RSP API Aspect

On the the main landing page at data.lsst.cloud there is an “APIs” panel with more information on the API Aspect. API stands for Application Programming Interface. An API is, essentially, code that permits two other programs (applications) to communicate with each other. If it helps, think of it as a request-and-response (or a client-server) situation: one program (a client) requests something (data) and the other (a server) responds.

The API’s services for DP0.2 include TAP, ObsTAP, SODA (image cutouts and mosaics), and HiPS. Currently, Rubin Community Science Team (CST) support for the RSP API services beyond TAP is limited, but more support will become available soon. For the time being, this page will focus on the API’s TAP services.

Longer term, Rubin Observatory will support SCS for simple catalog searches, SIAv2 for image searches, and VOSpace (in addition to WebDAV) for access to user files.

As described below, the API aspect permits Rubin data access both from inside the RSP environment (e.g., via the RSP notebook aspect) and from outside the RSP environment (e.g., via TOPCAT or from a pyvo-enabled environment like NOIRLab’s Astro Data Lab).

Currently, there are no additional database constraints imposed on external environments. As for the internal RSP enviroment, the database server will generally be unable to process queries that have an internal intermediate form that is larger than roughly 5GB, and the TAP service will have difficulties with final results that are over roughly 2GB in size. That said, this is by no means the final word on what limits might be in operations.

Important

The API Aspect has a lot of new features for DP0.2, which will eventually be added to this page. Check back soon for new information!

Use of TAP via the RSP Notebook Aspect

The primary means of accessing TAP is via the RSP’s Portal and Notebook aspects.

Use of the TAP service to query catalogs via the Portal is described in Introduction to the RSP Portal Aspect.

In the Notebook Aspect, a TAP service is instantiated in a python notebook and used to execute an ADQL query and return a result set. Within the RSP’s Notebook Aspect, a set of utilities are provided to get a TAP service instance.

from lsst.rsp import get_tap_service, retrieve_query
service = get_tap_service()
query = "SELECT TOP 100 * FROM dp02_dc2_catalogs.Object"
results = service.search(query)
results.to_table().show_in_notebook()

Several of the DP0 Notebook tutorials demonstrate how to use the TAP service programmatically from a python notebook.

Use of TOPCAT with the RSP TAP service

One popular and useful non-RSP utility that supports the TAP API is TOPCAT. With TOPCAT, it is possible to run ADQL queries on the Rubin DP0.2 data set, explore tables, and make a variety of 2D and 3D plots via an interactive graphical users interface (GUI). (For a broader view of TOPCAT capabilities, please see the TOPCAT webpage, which includes documentation, examples, and tutorials.)

To access DP0.2 from a non-RSP TAP utility like TOPCAT, an RSP access token needs to be generated. How to generate and use an RSP access token is described by the Rubin Science Platform APIs webpage and by the Science Platform Tokens webpage.

Important

Note that tokens should be treated like passwords: they should not be shared with others. Take precautions to keep tokens secure. Never store tokens in git-tracked files.

Get started with TOPCAT

This section provides a basic guide to get TOPCAT set up to explore the DP0.2 tables.

1. Create an RSP access token. See the Creating user tokens webpage for a step-by-step guide for creating an RSP access token. It is recommended that the token you create has the following propoerties: a name that includes “TOPCAT” as a substring, a scope of read:tap, and no expiration date.) The token will only be visible once. You must cut-and-paste the token to a secure file for future use. A new token is not needed for every TOPCAT session; the token can be reused, but it must be kept secure.

2. Start up TOPCAT on your own computer. See TOPCAT homepage for download and install instructions.

3. Click on “Table Access Protocol (TAP) Query” under the “VO” menu. This will open up a separate Table Access Protocol (TAP) Query window.

The main TOPCAT window, with the “VO” menu pulled down and the “Table Access Protocol (TAP) Query” highlighted.

4. Fill in the relevant in the “TAP URL” window and click the “Use Service” button in the Table Access Protocol (TAP) Query window. For DP0.2, use https://data.lsst.cloud/api/tap. If you wish to access DP0.3 – which is a database of solar system objects that supplements the main DP0.2 database – use https://data.lsst.cloud/api/ssotap instead.

The Table Access Protocol (TAP) Query window with the location of the “Selected TAP Service” panel indicated by a blue oval.

5. Populate the Authentication window that pops up. Fill in x-oauth-basic for the “User” and your security token forthe “Password” and click “OK”.

The Authentication window with the values for the user and the password filled in.

6. Note that the RSP TAP service is now accessible from your instance of TOPCAT. An indicator that the service is now accessible is that a list of DP0.1 and DP0.2 tables available has appeared in the Metadata panel of the TAP Query window.

The Table Access Protocol (TAP) Query window; a list of DP0.1 and DP0.2 tables available for query can be be seen in the Metadata panel.

7. Explore. At this stage, the Rubin DP0.2 data set can be explored via TOPCAT. For an example, see the 01. API TOPCAT Bright Stars Color-Magnitude Diagram Tutorial (beginner).

Use of pyvo with the RSP TAP service

Another way to access the Rubin data from outside the RSP environment is via the pyvo python module, an affiliated astropy package for providing access to remote data and services of the Virtual Observatory using python. By this method, if pyvo is installed, one can access the RSP TAP service directly from one’s own laptop. If not, one can access the RSP TAP service from other freely accessible services that have pyvo pre-installed (like, e.g., NOIRLab’s Astro Data Lab Jupyter Notebook server).

Important

Recall that tokens should be treated like passwords: they should not be shared with others. Take precautions to keep tokens secure. Never store tokens in git-tracked files.

Get started with pyvo

This section provides a basic guide to provide access to the DP0.2 TAP service via python code on your own computer or on an online service like NOIRLab’s Astro Data Lab Jupyter Notebook server.

1. Copy an RSP access token into a file in your home directory. As with the TOPCAT example above, one needs an RSP access token. Either generate one as described above in Use of TOPCAT with the RSP TAP service, or just use a previously generated (but unexpired) RSP access token. Ideally, copy the RSP access token into a file in your home directory that is only read/write accessible by the file owner and that is accessible to the python session that will be accessed in the steps below. Specifically, in a UNIX/MacOS/Linux environment, the following commands can be performed in order to create this file in a secure manner that avoids exposing the RSP token to outside resources for even a short period of time.

• Open a terminal window (not a Jupyter notebook) on your computer or in your non-RSP user environoment.

• Change directory to the home directory.

cd ~

• In the home directory, create an empty file that will eventually contain the RSP token. One can do this via the touch command. Note that using a ‘hidden’ file – one with a name that starts with a . – improves security.

touch .rsp-tap.token

• Change the permissions on this file to remove world and group read/write access. The chmod 600 command will do this while maintaining read/write access for the file owner.

chmod 600 .rsp-tap.token

• Insert the RSP token into this file securely. The following command permits this by requesting the file owner to enter the RSP token at the first prompt.

cat <<EOF > .rsp-tap.token

• Close this file by issuing an “end of file” command at the second prompt. After entering this command, the file will be ready for use for as long as the RSP token is unexpired.

EOF

2. Start up a python session. This could be a standalone python session running on (say) a laptop, or a Jupyter notebook running elsewhere but displayed on one’s own browser.

3. Import relevant python modules. At the minimum, import the pyvo and os python modules.

import pyvo
import os

4. Define the relevant TAP server URL and read in your security token. For DP0.2, the proper TAP server URL is https://data.lsst.cloud/api/tap, as is shown below. (For DP0.3, use https://data.lsst.cloud/api/ssotap instead.) The os.path.expanduser('~') command is a cross-platform method for identifying the home directory without hardwiring its path into the code. (As a side benefit, it works in both the UNIX/MacOS/Linux and Windows environments.)

RSP_TAP_SERVICE = 'https://data.lsst.cloud/api/tap'
homedir = os.path.expanduser('~')
token_file = os.path.join(homedir,'.rsp-tap.token')
with open(token_file, 'r') as f:
    token_str = f.readline()

5. Set up appropriate authorization to access the RSP TAP server. In line 1 of the following code block, a pyvo CredentialStore is instantiated. In line 2, the TAP user ("x-oauth-basic") and the RSP token (token_str) is passed to the CredentialStore. Line 3 establishes that the RSP TAP service conforms to the interface requirements of the International Virtual Observatory (IVOA) for HTTP basic authentication; hence the ivo://ivoa.net/sso#BasicAA security method is designated. Finally, in line 4, a request session to the RSP TAP service is established.

cred = pyvo.auth.CredentialStore()
cred.set_password("x-oauth-basic", token_str)
credential = cred.get("ivo://ivoa.net/sso#BasicAA")
rsp_tap = pyvo.dal.TAPService(RSP_TAP_SERVICE, credential)

6. Run a query. For example, in the following case, the query requests a list of the catalogs that are available from the RSP TAP service. More examples of useful DP0.2 queries can be found in the DP0.2 Notebook tutorials and particularly in DP0.2 Tutorial Notebook 2: Catalog Queries with TAP.

query = "SELECT * FROM tap_schema.schemas"
results = rsp_tap.run_sync(query)
results.to_table()