Data Processing

# TODO: Fix this

The TOM is configured with built-in data processing on the upload of Data Products. This processing is broken up into three components.

  • tom_dataproducts.data_processor.run_data_processor

  • tom_dataproducts.data_processor.DataProcessor

  • tom_dataproducts.data_serializers

The DataProductUploadView calls run_data_processor upon saving the DataProduct. run_data_processor instantiates the DATA_PROCESSOR_CLASS, which can be specified in settings.py. run_data_processor then processes the uploaded DataProduct based on the tag value, which can either be Photometry or Spectroscopy.

In the case of spectra, the default behavior is that DataProduct is converted into a specutils.Spectrum1D from either a CSV or a FITS file. The spectrum is multiplied by the flux constant of the facility it was taken at, or not multiplied. The Spectrum1D is then serialized into JSON for database via the SpectrumSerializer.

For photometry, the default behavior is simply to read the CSV and convert it to JSON for database storage.

Data Serializers

Data Processors

tom_dataproducts.data_processor.run_data_processor(dp)

Reads the data_product_type from the dp parameter and imports the corresponding DataProcessor specified in settings.py, then runs process_data and inserts the returned values into the database.

Parameters

dp (DataProduct) – DataProduct which will be processed into a list

class tom_dataproducts.data_processor.DataProcessor
process_data(data_product)

Routes a photometry processing call to a method specific to a file-format. This method is expected to be implemented by any subclasses.

Parameters

data_product (DataProduct) – DataProduct which will be processed into a list

Returns

python list of 2-tuples, each with a timestamp and corresponding data

Return type

list of 2-tuples