Control Loops Outside the Lab · 01
Delayed Feedback and Overcorrection
Why autoscalers and retries can overreact while earlier work is still in progress. Follow the decision, startup, and measurement delays before tuning the response.
An ingestion service is falling behind. Its autoscaler requests more workers, but the workers take a minute to start. Thirty seconds later, the backlog is larger. The autoscaler interprets that as insufficient capacity and raises the target again.
Both decisions use accurate measurements. The second one is still missing something important: the first capacity request has not taken effect yet.
This becomes a recurring problem when the decision interval is shorter than the system’s response time. It shows up in worker pools, adaptive concurrency limits, and clients that retry before the server finishes the original request. To understand the resulting behavior, it helps to follow one change from the decision through to its observable effect.
What happens during the startup interval
Suppose ten workers each process 100 records per second. Arrivals rise to 1,500 records per second, so the queue grows by 500 records per second. Twenty workers would provide enough capacity to catch up.
The controller requests those twenty workers, but startup takes sixty seconds. During that minute, another 30,000 records accumulate. The request may be entirely adequate; the queue still gets worse while it is being carried out.
At the next thirty-second check, a controller that considers only the current backlog can ask for an even larger pool. Eventually the earlier requests arrive, capacity exceeds demand, and the queue drains. By then, newer decisions may already be scheduled to reduce the pool. Those reductions will arrive late too.
The result can be a repeating cycle of excess capacity and growing backlog. In a reporting pipeline, that means both an uneven infrastructure bill and uneven data freshness. Increasing controller strength can make the cycle larger because each decision responds more aggressively to a condition that will change before the decision takes effect.
Separate active capacity from requested capacity
A useful scaling dashboard needs at least three pieces of state: queued work, workers doing useful work now, and capacity changes still in progress. Requested replicas alone can look reassuring while the new processes are pulling images or waiting for an assignment.
The simulation below puts backlog above the active and requested worker counts. It aims for a nonzero backlog, representing a deliberate allowance for processing delay rather than enough spare capacity to absorb every burst immediately.
- Backlog
- Target
- Active workers
- Pending capacity target
Start with Stable and inject a spike. Follow the sequence: backlog rises, a larger worker count is requested, and that count takes effect later. Then increase Provisioning delay while leaving the other controls fixed. Watch how far the queue moves before the earlier decisions catch up.
These requests are absolute worker-count targets. Asking for thirty twice does not create sixty workers. Oscillation is still possible because each target is chosen from the backlog at decision time and applied to a different backlog later.
The shaded gap compares active workers with the last target scheduled to take effect. Real orchestration systems may coalesce requests or cancel obsolete ones; this simplified actuator applies each target after its delay. The simulation uses its own scaled timings and workload, rather than the exact numbers in the example above.
Backlog size also needs interpretation. A million small records and a million expensive records imply different drain times. Processing rate and oldest-record age help establish whether a burst is clearing fast enough to meet the freshness requirement.
The measurement can be late as well
Startup is only one source of delay. A controller might use a one-minute utilization average, request new capacity, and then wait for another metric window to show its effect. By that point, the input rate may have changed again.
Polling that same average more often does not necessarily provide newer evidence. The sampling interval, metric window, and startup time all belong in the description of the loop. Control models with explicit delays make it possible to reason about where the lag enters.
There are several ways to respond. Smaller corrections reduce the consequences of acting on an old measurement. Accounting for pending capacity changes makes the controller’s view more complete. Shorter startup times allow decisions to take effect sooner. Each changes a different part of the loop, so it is worth measuring those intervals before choosing a remedy.
Retrying while a query is still running
A reporting UI can create a related failure without an autoscaler. Suppose a query takes 900 ms under load, but its client gives up after 500 ms. If the warehouse continues executing it, the retry adds work while the original is still consuming resources.
The user is still waiting for one result. The system may now be computing it twice, and the extra work can make subsequent requests slower.
A timeout establishes that the client stopped waiting. The server may still finish the operation, and a write may already have committed. Backoff and retry budgets limit added load; end-to-end deadlines can let the server discard work that is no longer useful. Idempotency addresses repeated effects, although processing a repeated request can still cost resources.
The shared issue is outstanding work. A scaling request is not yet active capacity, and a timed-out request is not necessarily a finished operation. Treating either one as absent can lead to an unnecessary second action.
The same sequence at a shower valve
The delayed effect is not specific to software. Water travelling from a mixing valve to a showerhead creates a physical interval between an adjustment and its result, much like the startup interval between requesting a worker and receiving its output.
Turn the valve toward hot and the water at the showerhead stays cold for a few seconds. Another adjustment during that interval is based on water that left the valve before the first change. When the warmer water arrives, both adjustments have already been made.
The controller below tries to reach 38 °C and adjusts the valve every two seconds. Long pipe introduces a six-second transport delay, leaving time for several corrections before the first can be felt.
- What you dialed
- What you feel
- Comfortable
Compare Long pipe with Long pipe, gentle hand. Watch the valve temperature as well as the temperature at the showerhead. The valve can already be heading toward cold while hot water is still arriving. Smaller adjustments leave less correction travelling through the pipe at once.
This controller adds a correction on each tick. For that kind of rule, halving the correction while doubling the adjustment rate can leave roughly the same total correction accumulating during the delay. The autoscaler above instead requests absolute targets, so repeated identical requests have different semantics. Both still need to allow for effects that have not arrived.
Diagnose the delay before changing the gain
When a scaling graph starts oscillating, record when each request was issued, acknowledged, and completed. Put pending capacity beside active capacity. After a small adjustment, observe the response through a full startup interval before deciding it was too small.
Waiting is appropriate while a useful change is in progress. It stops being useful if that change has failed. Completion timestamps and failure acknowledgments give a controller the evidence to distinguish those cases, instead of repeatedly reacting to the same unfinished work.