Cadence and Observing Strategies¶
The TOM has a couple of unique concepts that may be unfamiliar to some at first, that will be describe here before going into detail.
The first concept is that of an observing strategy. An observing strategy is something of a template. If an observer is consistently submitting observations with a lot of similar parameters, it may be useful to save those as a kind of template, which can just be loaded later. The TOM Toolkit offers an interface that allows facilities to define a strategy form, that will be saved as an observing strategy. The strategy can then be applied to an observation, with the remaining parameters filled in or changed. An observing strategy can also be creating from a past observation, with a button to do so that’s available on any ObservationRecord detail page.
The second concept referred to is a cadence strategy. A cadence is as it sounds–a series of observations that are performed at regular intervals. However, most observatories don’t have built-in support for cadences, and, if they do, they may be limited to a predetermined cadence. The TOM Toolkit, on the other hand, allows for a reactive cadence. Because data is collected programmatically, and observations are submitted programmatically, a user can write their own cadence to submit observations depending on the success of a prior observation or the data collected from a prior observation.
Writing a custom cadence strategy¶
Many of the TOM modules leverage a plugin architecture that enables you to write your own implementation, and the cadence strategy plugin is no different. If you’re familiar with the other modules, you’ve already seen examples of this in the Writing an alert broker, Writing an observation module, and Customizing data processing tutorials.
Create a cadence strategy file¶
First, you’ll need a file where you’ll put your custom cadence strategy. If you have a fresh TOM installation, you’ll have a directory structure that looks something like this:
├── data ├── db.sqlite3 ├── manage.py ├── mytom │ ├── __init__.py │ ├── settings.py │ ├── urls.py │ └── wsgi.py ├── static ├── templates └── tmp
We’ll create a new file called mycadence.py
and place it next to settings.py
. To get started, we’ll just put a
small skeleton into our new file, so to begin with, it should look like this:
```python from tom_observations.cadence import CadenceStrategy
- class MyCadenceStrategy(CadenceStrategy):
pass
We also need to add the cadence strategy to settings.py
so that our TOM knows that it exists:
```python TOM_CADENCE_STRATEGIES = [
‘tom_observations.cadence.RetryFailedObservationsStrategy’, ‘tom_observations.cadence.ResumeCadenceAfterFailureStrategy’, ‘mytom.mycadence.MyCadenceStrategy’
]¶
Add logic to the new cadence strategy¶
You may have noticed that our MyCadenceStrategy
class inherits from CadenceStrategy
. The CadenceStrategy
interface only has one method, which is run()
. All of the logic for a CadenceStrategy
lives in the run()
method. Rather than demonstrating the implementation of a new cadence strategy, this tutorial is going to walk through
the business logic of a built-in cadence strategy. We’re going to review the ResumeCadenceAfterFailureStrategy
.
It should also be worth mentioning at this point that the CadenceStrategy
constructor takes an observation group
.
The observation_group
is the set of observations that make up the cadence, and is created in the ObservationCreateView
when the first observation of a cadence is submitted.
The ResumeCadenceAfterFailureStrategy
is designed to ensure that, even after an observation fails, the cadence remains
consistent. If, for example, you submit an observation with a cadence of three days, and the observation fails, the cadence
should attempt to get the observation as soon as possible, and then resume observing once every three days.
Let’s look at the strategy piece by piece.
`python
last_obs = self.observation_group.observation_records.order_by('-created').first()
facility = get_service_class(last_obs.facility)()
facility.update_observation_status(last_obs.observation_id)
last_obs.refresh_from_db()
`
The first thing this strategy does is get a couple of pieces of information. First, from the observation group that the
cadence consists of, the most recent observation is selected. The facility class for the facility that the cadence is
submitting observations to is also instantiated. With these values, the status of the most recent cadence observation is
updated, and the ObservationRecord
object is refreshed.
`python
start_keyword, end_keyword = facility.get_start_end_keywords()
observation_payload = last_obs.parameters_as_dict
new_observations = []
`
These lines are, again, just more setup. Each facility has its own unique keywords representing the start and the end of the observation window, so we get those from the facility class. Then, we get the original observation parameters that were submitted to the facility, and we initialize a list for any new observations that will be submitted when the cadence is updated.
```python if not last_obs.terminal:
return
- elif last_obs.failed:
# Submit next observation to be taken as soon as possible window_length = parse(observation_payload[end_keyword]) - parse(observation_payload[start_keyword]) observation_payload[start_keyword] = datetime.now().isoformat() observation_payload[end_keyword] = (parse(observation_payload[start_keyword]) + window_length).isoformat()
- else:
# Advance window normally according to cadence parameters observation_payload = self.advance_window(
observation_payload, start_keyword=start_keyword, end_keyword=end_keyword
)
Here we have some logic for the three cases–either the most recent observation hasn’t happened yet, it failed, or it succeeded.
If it hasn’t happened, then there’s nothing to do–we’ll check again later. If if failed, we want to submit it again to be taken
immediately, so we get the original length of the observation window, and set our new observation payload to start immediately,
and end such that the new window length is the same. Finally, if our observation succeeded, we update our new observation
parameters to start 72 hours after the last observation, using a utility method that’s part of the
ResumeCadenceAfterFailureStrategy
called advance_window
.
```python obs_type = last_obs.parameters_as_dict.get(‘observation_type’) form = facility.get_form(obs_type)(observation_payload) form.is_valid() observation_ids = facility.submit_observation(form.observation_payload())
- for observation_id in observation_ids:
# Create Observation record record = ObservationRecord.objects.create(
target=last_obs.target, facility=facility.name, parameters=json.dumps(observation_payload), observation_id=observation_id
) self.observation_group.observation_records.add(record) self.observation_group.save() new_observations.append(record)
- for obsr in new_observations:
facility = get_service_class(obsr.facility)() facility.update_observation_status(obsr.observation_id)
return new_observations
The last part of our strategy is when we submit our new observations. Regardless of how we modified the observing window, we initialize our observation form, validate it, and submit the observation to our facility. The rest of the code is saving any resulting observations to the database, getting their new status from the facility, and returning them.
Just to review, here is the strategy’s run()
in its entirety:
last_obs = self.observation_group.observation_records.order_by(‘-created’).first() facility = get_service_class(last_obs.facility)() facility.update_observation_status(last_obs.observation_id) last_obs.refresh_from_db() start_keyword, end_keyword = facility.get_start_end_keywords() observation_payload = last_obs.parameters_as_dict new_observations = [] if not last_obs.terminal:
return
- elif last_obs.failed:
# Submit next observation to be taken as soon as possible window_length = parse(observation_payload[end_keyword]) - parse(observation_payload[start_keyword]) observation_payload[start_keyword] = datetime.now().isoformat() observation_payload[end_keyword] = (parse(observation_payload[start_keyword]) + window_length).isoformat()
- else:
# Advance window normally according to cadence parameters observation_payload = self.advance_window(
observation_payload, start_keyword=start_keyword, end_keyword=end_keyword
)
obs_type = last_obs.parameters_as_dict.get(‘observation_type’) form = facility.get_form(obs_type)(observation_payload) form.is_valid() observation_ids = facility.submit_observation(form.observation_payload())
- for observation_id in observation_ids:
# Create Observation record record = ObservationRecord.objects.create(
target=last_obs.target, facility=facility.name, parameters=json.dumps(observation_payload), observation_id=observation_id
) self.observation_group.observation_records.add(record) self.observation_group.save() new_observations.append(record)
- for obsr in new_observations:
facility = get_service_class(obsr.facility)() facility.update_observation_status(obsr.observation_id)
return new_observations
Configuring the cadence strategy to run automatically¶
As you may have noticed, the cadence strategies act on updates to the status of an ObservationRecord
. Ideally, we want
the cadence strategies to run as soon as an observation status changes–so, we need to automate that and have it run
periodically.
Fortunately, the TOM Toolkit comes with a built-in management command to update all cadences in the TOM. If you’ve perused
the TOM Toolkit documentation previously, you may have noticed a section about automation of tasks, and, more specifically,
a subsection about Using cron with a management command. You can simply apply the
instructions here, but use the management command runcadencestrategies.py
in place of the example. If you set your cron
to run every few minutes or so, you’ll ensure that your cadences are kept up to date!