Integrating Single-Target Data Service Queries

The base TOM Toolkit comes with ATLAS, PanSTARRS, and (coming soon) ZTF query services. These services query a specific catalog to return data for an individual target and are optional, requiring additional configuration in order to integrate into your TOM.

Additional services can be added by extending the BaseSingleTargetDataService implementation (see below).

Integrating existing Single-Target Data Services

You must add certain configuration to your TOM’s settings.py to setup the existing single-target data services. This configuration will go in the SINGLE_TARGET_DATA_SERVICES section shown below:

SINGLE_TARGET_DATA_SERVICES = {
    'ATLAS': {
        'class': 'tom_dataproducts.single_target_data_service.atlas.AtlasForcedPhotometryService',
        'url': "https://fallingstar-data.com/forcedphot",
        'api_key': os.getenv('ATLAS_FORCED_PHOTOMETRY_API_KEY', 'your atlas account api token')
    },
    'PANSTARRS': {
        'class': 'tom_dataproducts.single_target_data_service.panstarrs_service.panstarrs.PanstarrsSingleTargetDataService',
        'url': 'https://catalogs.mast.stsci.edu/api/v0.1/panstarrs',  # MAST Base URL
        # MAST_API_TOKEN is not required for public data
        'api_key': os.getenv('MAST_API_TOKEN', 'MAST_API_TOKEN not set')
    },
    # TODO: coming soon...
    #     # 'ZTF': {
    # }
}

DATA_PRODUCT_TYPES = {
    ...
    'atlas_photometry': ('atlas_photometry', 'Atlas Photometry'),
    'panstarrs_photometry': ('panstarrs_photometry', 'PanSTARRS Photometry'),
    ...
}

DATA_PROCESSORS = {
    ...
    'atlas_photometry': 'tom_dataproducts.processors.atlas_processor.AtlasProcessor',
    'panstarrs_photometry': 'tom_dataproducts.processors.panstarrs_processor.PanstarrsProcessor',
    ...
}

As you can see in the SINGLE_TARGET_DATA_SERVICES configuration dictionary above, some services require an API key. Information on how to obtain an API key is available for both for ATLAS and for PanSTARRS. (PanSTARRS Photometry is accessed via Catalogs.MAST).

Configuring your TOM to serve tasks asynchronously:

Several of the services are best suited to be queried asynchronously, especially if you plan to make large queries that would take a long time. The TOM Toolkit can be setup to use django-tasks as an asynchronous task manager. To use django-tasks you can enable it by adding the following to your settings.py file:

INSTALLED_APPS = [
    ...
    'django_tasks',
    'django_tasks.backends.database',
    ...
]

TASKS = {
    "default": {
         "BACKEND": "django_tasks.backends.database.DatabaseBackend"
        # "BACKEND": "django_tasks.backends.immediate.ImmediateBackend"
    }
}

After adding the django_tasks installed app, you will need to run ./manage.py migrate once to setup its DB tables. If this configuration is set in your TOM, the existing services which support asynchronous queries, Atlas and ZTF, should start querying asynchronously. (Note: You must also start the task workers: ./manage.py db_worker. If you do not add these settings, those services will still function but will fall back to synchronous queries.

Adding a new Single-Target Data Service

The Single-Target Data services fulfill an interface defined in BaseSingleTargetDataService. To implement your own single-target data service, you need to do three things:

1. Subclass BaseSingleTargetDataService

2. Subclass BaseSingleTargetDataServiceQueryForm

3. Subclass DataProcessor

Once those subclasses are implemented, don’t forget to update your settings for SINGLE_TARGET_DATA_SERVICES, DATA_PRODUCT_TYPES, and DATA_PROCESSORS for your new service and its associated data product type.

Subclass BaseSingleTargetDataService:

The most important method here is the query_service method which is where you put your service’s business logic for making the query, given the form parameters and target. This method is expected to create a DataProduct in the database at the end of the query, storing the result file or files. If queries to your service are expected to take a long time and you would like to make them asynchronously (not blocking the UI while calling), then follow the example in the atlas implementation and place your actual asynchronous query method in your module’s tasks.py file so it can be found by django-tasks. Like in the atlas implementation, your code should check to see if the current task backend is asynchronous or immediate and handle the result appropriately.

The get_data_product_type method should return the name of your new data product type you are going to define a DataProcessor for. This must match the name you add to DATA_PROCESSORS and DATA_PRODUCT_TYPES in your settings.py. You will also need to define a DataProcessor for this data type.

Subclass BaseSingleTargetDataServiceQueryForm:

This class defines the form users will need to fill out to query the service. It uses django-crispy-forms to define the layout programmatically. You first will add whatever form fields you need to the base of your subclass, and then just fill in the layout() method with a django-crispy-forms layout for your fields, and optionally the clean() method if you want to perform any field validation. The values of the fields from this form will be available to you in your service class in the query_service method.

Subclass DataProcessor:

You must create a custom DataProcessor that knows how to convert data returned from your service into a series of either photometry or spectroscopy datums. Without defining this step, your queries will still result in a DataProduct file being stored from the service’s query_service method, but those files will not be parsed into photometry or spectroscopy datums. You can read more about how to implement a custom DataProcessor here.