Publish and Subscribe to a Kafka Stream¶
Publishing data to a stream and subscribing to a stream are handled independently and we describe each below.
Publish Data to a Kafka Topic¶
TOM Toolkit supports publishing data to a Kafka stream such as Hermes (an interface to HOPSKOTCH) and GCNClassicOverKafka.
When sharing photometry data via Hermes, the TOM publishes the data to be shared to a topic on the HOPSKOTCH Kafka stream. At this time, only photometry data is supported by TOM Toolkit. To submit via the Hermes API, you will need to copy your Hermes API Key from your Hermes profile page. When hermes sharing is configured, you will also see buttons to open your data in hermes with the form pre-filled - this is a good option if you want to make slight changes to your message or data before sharing.
Configuring your TOM to Publish Data to a stream:¶
You will need to add a DATA_SHARING
configuration dictionary to your settings.py
that gives the credentials
for the various streams with which you wish to share data.
# Define the valid data sharing destinations for your TOM.
DATA_SHARING = {
'hermes': {
'DISPLAY_NAME': os.getenv('HERMES_DISPLAY_NAME', 'Hermes'),
'BASE_URL': os.getenv('HERMES_BASE_URL', 'https://hermes.lco.global/'),
'HERMES_API_KEY': os.getenv('HERMES_API_KEY', 'set HERMES_API_KEY value in environment'),
'DEFAULT_AUTHORS': os.getenv('HERMES_DEFAULT_AUTHORS', 'set your default authors here'),
'USER_TOPICS': ['hermes.test', 'tomtoolkit.test'] # You must have write permissions on these topics
},
}
Subscribe to a Kafka Topic¶
TOM Toolkit allows a TOM to subscribe to a topic on a Kafka stream, ingesting messages from that topic and handling the data. This could involve simply logging the message or extracting the data from the message and saving it if it is properly formatted.
Configuring your TOM to subscribe to a stream:¶
First you will need to add tom_alertstreams
to your list of INSTALLED_APPS
in your settings.py
.
INSTALLED_APPS = [
...
'tom_alertstreams',
]
Then you will need to add an ALERT_STREAMS
configuration dictionary to your settings.py
. This gives the credentials
for the various streams to which you wish to subscribe. Additionally, the TOPIC_HANDLERS
section of the stream OPTIONS
will include a list of handlers for each topic.
Some alert handlers are included as examples. Below we demonstrate how to connect to a Hermes Topic. You’ll want to check
out the tom-alertstreams
README for more details.
ALERT_STREAMS = [
{
'ACTIVE': True,
'NAME': 'tom_alertstreams.alertstreams.hopskotch.HopskotchAlertStream',
'OPTIONS': {
'URL': 'kafka://kafka.scimma.org/',
'USERNAME': os.getenv('SCIMMA_CREDENTIAL_USERNAME', 'set SCIMMA_CREDENTIAL_USERNAME value in environment'),
'PASSWORD': os.getenv('SCIMMA_CREDENTIAL_PASSWORD', 'set SCIMMA_CREDENTIAL_USERNAME value in environment'),
'TOPIC_HANDLERS': {
'tomtoolkit.test': 'tom_dataproducts.alertstreams.hermes.hermes_alert_handler',
},
},
},
]