Introduction to the RSP API Aspect

On the the main landing page at data.lsst.cloud there is an “APIs” panel with more information on the API Aspect.

An API is an Application Programming Interface. It is a piece of code that permits two other programs to communicate with each other.

The API’s services for DP0.2 include TAP, ObsTAP, SODA (image cutouts and mosaics), and HiPS. Currently, Rubin Community Science Team (CST) support for the RSP API services beyond TAP is limited, but more support will become available soon. For the time being, this page will focus on the API’s TAP services.

Longer term, Rubin Observatory will support SCS for simple catalog searches, SIAv2 for image searches, and VOSpace (in addition to WebDAV) for access to user files.

Important

The API Aspect has a lot of new features for DP0.2, which will eventually be added to this page. Check back soon for new information!

Use of TAP via the RSP Notebook Aspect

The primary means of accessing TAP is via the RSP’s Portal and Notebook aspects.

Use of the TAP service to query catalogs via the Portal is described in Introduction to the RSP Portal Aspect.

In the Notebook Aspect, a TAP service is instantiated in a python notebook and used to execute an ADQL query and return a result set. Within the RSP’s Notebook Aspect, a set of utilities are provided to get a TAP service instance.

from lsst.rsp import get_tap_service, retrieve_query
service = get_tap_service()
query = "SELECT TOP 100 * FROM dp02_dc2_catalogs.Object"
results = service.search(query)
results.to_table().show_in_notebook()

Several of the DP0 Notebook tutorials demonstrate how to use the TAP service programmatically from a python notebook.

Use of TOPCAT with the RSP TAP service

One popular and useful non-RSP utility that supports the TAP API is TOPCAT. With TOPCAT, it is possible to run ADQL queries on the Rubin DP0.2 data set, explore tables, and make a variety of 2D and 3D plots via an interactive graphical users interface (GUI). (For a broader view of TOPCAT capabilities, please see the TOPCAT webpage, which includes documentation, examples, and tutorials.)

To access DP0.2 from a non-RSP TAP utility like TOPCAT, an RSP access token needs to be generated. How to generate and use an RSP access token is described by the Rubin Science Platform APIs webpage and by the Science Platform Tokens webpage.

Important

Note that tokens should be treated like passwords: they should not be shared with others. Take precautions to keep tokens secure. Never store tokens in git-tracked files.

Get started with TOPCAT

This section provides a basic guide to get TOPCAT set up to explore the DP0.2 tables.

1. Create an RSP access token. See the Creating user tokens webpage for a step-by-step guide for creating an RSP access token. It is recommended that the token you create has the following propoerties: a name that includes “TOPCAT” as a substring, a scope of read:tap, and no expiration date.) It is highly recommended to cut-and-paste the token somewhere secure for future reference.

2. Start up TOPCAT on your own computer. See TOPCAT homepage for download and install instructions.

3. Click on “Table Access Protocol (TAP) Query” under the “VO” menu. This will open up a separate Table Access Protocol (TAP) Query window.

A screenshot of the main TOPCAT window with the Table Access Protocol item highlighted by the cursor under the VO drop-down menu.

The main TOPCAT window, with the “VO” menu pulled down and the “Table Access Protocol (TAP) Query” highlighted.

4. Fill in the relevant in the “TAP URL” window and click the “Use Service” button in the Table Access Protocol (TAP) Query window. For DP0.2, use https://data.lsst.cloud/api/tap. If you wish to access DP0.3 – which is a database of solar system objects that supplements the main DP0.2 database – use https://data.lsst.cloud/api/ssotap instead.

A screenshot of the the Table Access Protocol (TAP) Query window in which the value for the TAP URL has been filled in with the URL https://data.lsst.cloud/api/tap . A blue oval indicates the location of the Selected TAP Service panel in the window.

The Table Access Protocol (TAP) Query window with the location of the “Selected TAP Service” panel indicated by a blue oval.

5. Populate the Authentication window that pops up. Fill in x-oauth-basic for the “User” and your security token forthe “Password” and click “OK”.

A screenshot of the Authentication window. The user has been filled in with a value of x-oauth-basic, and the password is shown (for security purposes) as a series of filled black circles.

The Authentication window with the values for the user and the password filled in.

6. Note that the RSP TAP service is now accessible from your instance of TOPCAT. An indicator that the service is now accessible is that a list of DP0.1 and DP0.2 tables available has appeared in the Metadata panel of the TAP Query window.

A screenshot of the Table Access Protocol (TAP) Query window. The Table Access Protocol (TAP) Query window now shows three panels, stacked vertically. The top panel is the Metadata panel, and it shows a list of DP0.1 and DP0.2 schemas and tables that are available to query. The middle panel is the Service Capabilities panel, and it shows that the available Query Language is ADQL-2.0. The bottom panel is the ADQL Text panel, and it indicates the current Mode is Synchronous; the bottom panels text box is currently empty.

The Table Access Protocol (TAP) Query window; a list of DP0.1 and DP0.2 tables available for query can be be seen in the Metadata panel.

7. Explore. At this stage, the Rubin DP0.2 data set can be explored via TOPCAT. For an example, see the 01. API TOPCAT Bright Stars Color-Magnitude Diagram Tutorial (beginner).

Use of pyvo with the RSP TAP service

Another way to access the Rubin data from outside the RSP environment is via the pyvo python module, an affiliated astropy package for providing access to remote data and services of the Virtual Observatory using python. By this method, if pyvo is installed, one can access the RSP TAP service directly from one’s own laptop. If not, one can access the RSP TAP service from other freely accessible services that have pyvo pre-installed (like, e.g., NOIRLab’s Astro Data Lab Jupyter Notebook server).

Important

Recall that tokens should be treated like passwords: they should not be shared with others. Take precautions to keep tokens secure. Never store tokens in git-tracked files.

Get started with pyvo

This section provides a basic guide to provide access to the DP0.2 TAP service via python code on your own computer or on an online service like NOIRLab’s Astro Data Lab Jupyter Notebook server.

1. Copy an RSP access token into a file in your home directory. As with the TOPCAT example above, one needs an RSP access token. Either generate one as described above in Use of TOPCAT with the RSP TAP service, or just use a previously generated (but unexpired) RSP access token. Ideally, copy the RSP access token into a file in your home directory that is only read/write by the file owner and that is accessible to the python session that will be accessed in the steps below. Specifically, in a UNIX/MacOS/Linux environment, the following commands can be performed.

  • Open a terminal window (not a Jupyter notebook) on your computer or in your non-RSP user environoment.

  • Change directory to the home directory.

cd ~

  • Create a file in the home directory containing the RSP token. One can do this via the echo command. In the following, <token> is to be replaced by the the actual RSP token string. Note that using a ‘hidden’ file – one with a name that starts with a . – aids security.

echo '<token>' > .rsp-tap.token

  • Change the permissions on the file containing the RSP token to remove world and group read/write access. The chmod 600 command will do this while maintaining read/write access for the file owner.

chmod 600 .rsp-tap.token

2. Start up a python session. This could be a standalone python session running on (say) a laptop, or a Jupyter notebook running elsewhere but displayed on one’s own browser.

3. Import relevant python modules. At the minimum, import the pyvo and os python modules.

import pyvo
import os

4. Define the relevant TAP server URL and read in your security token. For DP0.2, the proper TAP server URL is https://data.lsst.cloud/api/tap, as is shown below. (For DP0.3, use https://data.lsst.cloud/api/ssotap instead.) The os.path.expanduser('~') command is a cross-platform method for identifying the home directory without hardwiring its path into the code. (As a side benefit, it works in both the UNIX/MacOS/Linux and Windows environments.)

RSP_TAP_SERVICE = 'https://data.lsst.cloud/api/tap'
homedir = os.path.expanduser('~')
token_file = os.path.join(homedir,'.rsp-tap.token')
with open(token_file, 'r') as f:
    token_str = f.readline()

5. Set up appropriate authorization to access the RSP TAP server. In line 1 of the following code block, a pyvo CredentialStore is instantiated. In line 2, the TAP user ("x-oauth-basic") and the RSP token (token_str) is passed to the CredentialStore. Line 3 establishes that the RSP TAP service conforms to the interface requirements of the International Virtual Observatory (IVOA) for HTTP basic authentication; hence the ivo://ivoa.net/sso#BasicAA security method is designated. Finally, in line 4, a request session to the RSP TAP service is established.

cred = pyvo.auth.CredentialStore()
cred.set_password("x-oauth-basic", token_str)
credential = cred.get("ivo://ivoa.net/sso#BasicAA")
rsp_tap = pyvo.dal.TAPService(RSP_TAP_SERVICE, credential)

6. Run a query. For example, in the following case, the query requests a list of the catalogs that are available from the RSP TAP service. More examples of useful DP0.2 queries can be found in the DP0.2 Notebook tutorials and particularly in DP0.2 Tutorial Notebook 2: Catalog Queries with TAP.

query = "SELECT * FROM tap_schema.schemas"
results = rsp_tap.run_sync(query)
results.to_table()