Control Loops Outside the Lab · 02
Keeping a Threshold from Flapping
A freshness alert keeps opening and closing while the pipeline barely changes. Separate recovery thresholds and timers can make each transition useful.
A data-freshness alert opens because a pipeline is ten minutes behind. The next batch commits, lag drops to nine minutes, and the alert closes. A minute later it opens again.
The pipeline is hovering near a boundary. Each batch briefly brings it back under the limit, and each interval between batches takes it over again. If every crossing sends a notification, the alert can consume attention without revealing a new operational problem.
The first question is what a change of state should mean. Does one reading beyond a line justify doing something, or should it take evidence of a more sustained change?
When a boolean creates work
A single threshold is an appealing implementation. Compare a value with a constant and return a boolean. For a utilization signal near 70 percent, it might receive:
69, 71, 68, 72, 70, 69, 71The measurements barely move, but the boolean keeps changing. If that boolean merely colors a chart, the result is distracting. If it pauses ingestion or starts a worker, the rule creates real work of its own.
The first simulation turns a noisy signal into an on/off state. The signal uses a generic 0–100 scale; it could represent utilization or a normalized lag measurement. Watch the state timeline when the threshold passes through the busy part of the trace.
- Signal
- Threshold (drag it)
- Switch on
Increase Noise amplitude while leaving the threshold in place. The transition count can grow even though the underlying pattern remains similar. Moving the threshold beyond the signal’s usual range quiets it down, but may also stop it detecting anything useful.
That is the limitation of solving an alerting problem by adjusting one number. There may be no single boundary that both catches the condition early and closes it reliably.
Give recovery its own threshold
For the freshness alert, opening at ten minutes and closing below five minutes gives recovery a separate boundary. A reading of nine minutes then means “keep the current state.” It cannot repeatedly open and close the alert by itself.
if (!active && lagMinutes >= 10) active = true;
else if (active && lagMinutes < 5) active = false;This is hysteresis: the outcome depends on the previous state as well as the current reading. At nine minutes of lag, an open alert stays open and a closed alert stays closed. The same reading produces different outcomes because the two states have different exit conditions.
The gap gives recovery some substance. Closing the alert says the pipeline has caught up enough to leave room for ordinary variation. It does not merely say the latest sample happened to fall on the healthy side.
A dead zone has a different purpose: it typically suppresses an output inside a range. Hysteresis retains an existing state there. That difference matters when implementing the rule.
Distinguish a spike from a sustained condition
Separate thresholds still allow a large, brief spike to trigger an alert. A batch arriving all at once can cross the upper boundary and disappear before anyone has time to investigate.
Persistence adds a time requirement. For example, lag might need to remain above ten minutes for 30 seconds before the alert opens. A sample below that boundary resets the timer. Recovery can have its own timer, so a momentary improvement does not immediately close the incident.
The three timelines below compare a single threshold, separate entry and exit thresholds, and thresholds with persistence. All receive exactly the same signal. This makes it possible to see which events each policy accepts, delays, or ignores.
- Signal
- High / low thresholds
- Single threshold
- State on
Compare Noisy and Drifting. With noise, persistence can discard short excursions. With drift, watch how long the more cautious policy takes to recognize a lasting change. A quieter timeline is useful only if the remaining transitions occur when they are needed.
The signal is sampled ten times per second to keep the comparison visible on screen. Production settings must use the real sampling interval: three samples ten seconds apart and three samples a minute apart imply very different delays. Missing samples also need a policy; treating a missing observation as proof of recovery can hide the failure being monitored.
The displayed cost is a fixed price per transition, in arbitrary units. It helps show the work caused by switching, but leaves out missed incidents and delayed detection. Those costs need to be measured separately.
Where the cost of switching becomes physical
A cooling system has a similar reason to distinguish activation from recovery. Starting a compressor whenever temperature rises above one boundary and stopping it whenever temperature falls below that same boundary can produce frequent restarts. Small fluctuations in temperature become repeated changes in equipment state.
Separate thresholds can start cooling above 24 °C and stop it below 22 °C. At 23 °C, the current state is retained. The Simulink relay model expresses this hysteresis rule directly.
Some equipment also needs explicit minimum on/off times. Those constrain how soon a state may change again, whereas the persistence policy above requires an input condition to last before changing state. The distinction matters in software too: a cooldown after scaling and a requirement for sustained high utilization solve different timing problems.
Choose the failure that is tolerable
For a freshness warning, waiting another 30 seconds may be acceptable. For a process about to exhaust memory, it may be a poor bargain. The entry threshold, recovery threshold, and timers should follow the consequence of a wrong decision.
Entry and recovery also need not be symmetric. An alert can open quickly and demand a longer healthy interval before closing. A severe condition can have an immediate path while ordinary warnings use persistence.
Before changing a production rule, replay both routine traffic and known incidents through it. Count false activations, missed incidents, detection delay, and recovery delay. A policy that never opens an alert will score perfectly on notification volume, which is a good reason to keep that metric in its place.
For a batch pipeline, replaying the normal commit cycle is particularly revealing. If every successful batch closes the alert and every interval between batches opens it, the rule is detecting the schedule as much as the health of the service. Giving it a recovery boundary and a little memory can turn that stream of crossings into a state an operator can actually use.