Machine-learning alerts

When to use a machine-learning alert

A static threshold fires when a number crosses a line. A moving-average rule fires when a number leaves its own recent band. A machine-learning alert fires when a reading leaves the band around what a model predicted for that moment, given everything else the site and the sky were doing. It is the rule to reach for when the normal value of a variable depends on other things: conductivity that rises with every storm, a depth that tracks rainfall upstream, a temperature that follows air temperature and cloud. A fixed bound either pages on every storm or sleeps through the excursion; a model that expects the storm does neither.

Prefer a static or moving-average rule when the condition really is a number crossing a line, when you need the rule to fire from the first reading, or when the site has little history — a model needs weeks of readings to learn from and is trained overnight, so it is never instant. It is also the only rule kind whose answer you should periodically check for quality; the score history is where you do that.

Recipients, pause and resume, the quiet period and the alert log work as they do for every other kind and are described on the alerts overview. Rules live under Machine-learning alerts (the page calls them that; the alerts hub calls them advanced).

How it works

1. When you save a rule, a model is trained on the site's recent history (a hyperparameter search over the estimator you chose, plus a search over which of your selected variables actually help). The newest slice of history is held back and the finished model is scored on it.
2. Every evaluation cycle the stored model predicts the current value of the target from the latest readings (and the weather forecast, if you gave it a location), and a band is drawn around that prediction from the two thresholds.
3. A reading outside the band is a breach; Min consecutive breaches breaches in a row is a firing, subject to the quiet period and the rain settings.
4. On its schedule the model is reinforced with the data that has arrived since, and optionally rebuilt from scratch.

Creating a rule

+ Create a New Alert on the Machine-learning alerts page opens the form. Above it, Visualize rule previews the thresholds on example data — useful for seeing why a storm the model expects does not fire while an excursion it did not expect does. The fields, top to bottom:

Site, location and model

Training window and schedule

What it predicts, and from what

Hyperparameters

A collapsed Hyperparameters section sits below the excluded periods. Almost nobody needs to open it: the recommended parameters for your estimator are pre-ticked and everything else is pinned to a stated value. Inside it, Search trials is how many candidate models the search tries before picking one (10–200, default 50); more trials explore more and cost proportionally more on every retrain. Each parameter row has a checkbox (Hyperparameters to tune — ticked means searched over the range shown, unticked means fixed at the pinned value), an editable pinned value, and an editable sweep range, all clamped to the allowed span printed beside them. Fewer ticked boxes usually means a more reproducible model and, on a fixed trial budget, a better one, because the budget is shared with the variable-selection search. Only the group matching your Model type is used; Auto-select Best Model uses them all.

Two switches in the same section decide what a nightly retrain costs, and the search is about nine tenths of it. Warm-start from the last model (on by default) begins the search at the settings that won last time instead of at a random draw — it cannot make the result worse, because the search still explores the same ranges and still keeps whichever model scores best, and it usually gets somewhere good in far fewer trials. Turn it off if you want every retrain to search independently, for instance while comparing two nights' models. Measure a tree-count curve (off by default) is the answer to "how many trees does this rule actually need": switch it on and the next retrain also scores the model at 50, 100, 150, 200 and 300 trees on held-out readings from your own sensor, with the spread of those readings under each number, and the curve appears on the rule's model page under Trees against error. Two sizes whose ranges overlap are two sizes the measurement cannot tell apart, so you can drop Number of trees with something behind the decision. It costs about half an extra fit per retrain, so the usual pattern is to switch it on, read the curve once, and switch it off.

Excluded periods

Stretches the model must not learn from: a sonde out for calibration, known works upstream, a planned release. + Add period adds a row with two UTC date-times and a note; Pick on a chart opens the sensor's own readings so you can drag over the flat line where it came out of the water. When the rule borrows features from other sensors, the chart draws their ticked variables too, each panel labelled with its sensor, so a period is marked against everything the model will see. Editing the list retrains the model. Past periods can be left in place.

The band and when it fires

Rain & weather, quiet period, recipients

The Rain & weather settings button gathers the location, the weather features and the rain fields: Pre rain pause and Post rain pause (hours), Rain behavior — Skip alerts during rain or Use alternate thresholds during rain — and, for the second, Rain upper bound / Rain lower bound, which replace the deviation bounds while the rain window is active (a blank one keeps the normal bound on that side). They behave exactly as on the threshold kinds; see rain behaviour there. A model that has been given rainfall as a feature often needs no rain pause at all, because it expects the storm response — try the rule without one first.

Where the weather comes from

The weather is modelled, not measured. It comes from Open-Meteo, which is not a gauge network but an aggregator of numerical weather models and reanalyses. A rainfall figure here is a model's value for a grid cell — measured from this deployment on 2026-08-29, between 1.5 km and 13.5 km from the point you name, depending on the site and on which endpoint answered. That is fine for deciding whether a storm passed. It is not a substitute for a rain gauge, and it should not be reported as one.

A rule's weather is also spliced from two different systems. Anything older than about six days comes from the historical archive; anything newer comes from a forecast model's own recent past and its forecast. Those disagree: over 1–20 December 2025 at one Californian point, the archive's default gave 17.4 mm and the ERA5 reanalysis gave 11.2 mm for the same twenty days. The Rain & weather settings panel names both halves, and the rain chart on the period picker draws the join as a dashed line rather than hiding it.

Weather source chooses which model answers, and applies to both halves, so naming one makes a rule's history and its forecast come from the same place.

Leave it on Automatic unless you know your catchment. Models disagree by more than people expect — the same hour at the same point, on 2026-08-29, came back as 23.3 °C from the automatic choice and 27.9 °C from global GFS. Automatic is also the only option whose archive reaches back to 1940, which matters for a rule that trains on months of history: pick a model whose archive starts after your training window and the older part of that window falls back to Automatic anyway, which the panel tells you it has done.

Changing the source changes what the rule decides on, so it is part of the rule's version stamp: a firing recorded before the change and one recorded after are not comparable, and the history says which was which.

Hours between alerts is the quiet period after a firing (default 1; 0 notifies on every breaching cycle), and the recipient fields and No recipients — show in the app only work as on every kind — see the overview.

After you save: the model's own status

Saving a new rule queues its first training immediately and the page says so ("Training this rule's model now — it will start evaluating once the first run finishes"). Editing an existing rule retrains only when you changed something the model is made of (model type, variables, features, look-ahead, excluded periods, training days, the split, the aggregation); changing thresholds, streak, rain or recipients does not, and saving a rule with nothing changed does nothing. To ask for a run without editing anything, use the two buttons on the rule's card: ↻ Retrain now rebuilds the model from its full training window, and ⟳ Reinforce now (shown when the rule has a reinforcement frequency) updates the stored model with recent data, which is cheaper. One run per rule is in flight at a time and the model chip follows it. Otherwise the scheduled training job — one task per rule, run one at a time on its own queue with a one-hour limit each — handles reinforcement (every 3 days by default) and walk-forward retrains, and rebuilds any model whose settings no longer match. A model is also rebuilt after a deploy that changes how features are computed (the bundle's feature version no longer matches the code), so after such a deploy every ML rule shows Retrain due until its turn comes. Under thin data — the look-back window less than half full after an outage — the evaluator reports a thin verdict rather than firing, unless the rule opts in to firing on a wide breach anyway.

Each card on the list carries two chips. The first is the usual alert state (Normal, Alerting, and so on). The second is the model chip, which updates live while training runs:

Spikes, losses and the band

Every tree estimator now offers a spike-resistant alternative to its default loss — huber or absolute_error for Gradient Boosting, huber or regression_l1 for LightGBM, and an absolute_error split criterion for the two forests. A fouled or stranded sonde writes spikes, and a squared-error model chases them: it spends its capacity fitting the fault instead of the creek. On test data with 2% spiked training rows, error on the clean data fell from 1.69 to 0.75 with huber and to 0.36 with absolute error.

There is a catch, and it is worth understanding before you tick one. A robust model stops bending to fit the outliers, so the outliers stay in its residuals at full size and the measured σ goes up — in that same run, from 2.11 to 3.84. A standard-deviation band widens with it, so a better model would end up alerting less. Two ways through: turn on Measure σ robustly, which estimates σ from the median deviation and is unmoved by the spikes, or use the predicted interval band, which never reads σ at all. The absolute-error split criterion on a forest is also substantially slower than the default, because every candidate split has to sort rather than accumulate.

Reinforcement versus retraining

A walk-forward retrain discards the model and re-runs the full search on the rolling window. It is the right tool when the rule's definition moved and an expensive one when only the calendar did, which is why its schedule defaults to off.

A reinforcement pass takes the stored model and the readings that arrived since it last learned (at least 20 new rows, otherwise it waits for the next night), and does two things in order. First it measures: the outgoing model is scored on the new rows, which it has never seen, and that score joins the history marked as a reinforcement measurement. Then it updates in place, in the way each estimator allows:

• Multi-layer Perceptron — the network takes extra gradient steps on the new rows (partial fit).
• Random Forest and Extra Trees — new trees are fitted to the new rows and appended; past 500 trees the oldest are dropped, so the forest slides forward through time instead of growing without bound.
• LightGBM — further boosting rounds are fitted against what the existing trees still get wrong on the new rows, and appended to the booster. Trees cannot be removed, so growth stops at 1,000.
• Gradient Boosting and Hist Gradient Boosting — extra boosting stages are fitted against the new rows' residuals. Stages cannot be removed, so growth stops at 1,000; the model still measures and stamps.
• Ridge Regression, K-Nearest Neighbors, Support Vector Regressor — these have no incremental mode, so the same winning configuration (estimator, hyperparameters, selected features) is refitted over the whole window with no search. That is a cheap walk-forward rather than true reinforcement, and because it really is a fit from scratch it stamps the trained clock, not the reinforced one — so a rule on one of these reads as freshly trained, which it is.

The scaler and imputer are frozen on the incremental paths, so the new trees or steps see the same transformed inputs the old ones were built on.

Checking the model: scores, residuals, detail page

On each card, scores opens a Model accuracy popup with two tabs, and visualize rule draws the sensor's readings against the predicted band.

Score history

One row per training or reinforcement run, newest first: Measured (when, with in use on the row that produced the current model and optimistic on a run that had too little history to hold back an untouched split, so its score is an upper bound), Model, Typical error (RMSE), Change against the run before, Beats doing nothing by (skill against simply reusing the last reading; n/a when the series barely moved), Spread (residual σ), Readings tested on and a plain-English Verdict. Lower error is better. The popup also charts the RMSE over time against the band width, and the residual distribution per run, so you can see the error move between retrains.

Residuals

The Residuals tab re-runs the model over a recent window (24 hours to 30 days) and plots, per reading, what the model expected, what the sensor reported, and the gap, with the rule's band drawn with the same arithmetic the evaluator uses — so a residual leaving the band is a firing, subject to the streak. An RMSE of 0.14 might be 0.14 everywhere or perfect all week and 3.0 out every afternoon; this is how you tell.

The detail page

Linked from inside the scores popup, the rule's detail page has a Model panel (estimator, when trained, the retrain and reinforcement cadence, how many features, the variables kept and the ones dropped as unhelpful, typical error against reusing the previous reading, how many readings it was measured on, an Accuracy verdict, an Alert band note on how the band compares with the error, and a Drift line when recent errors have widened relative to training — a prompt to retrain) and the full Score history table. A rule with no model says so; a model built by older feature code is reported as stale and rebuilt on the next run.

Tips and pitfalls

• Give it time and data. A model trained on a fortnight of one season will complain all winter. Check Readings tested on and the verdict before trusting a rule.
• Percent first, σ later. A stddev band is the better statement of "unusual for this model" but needs a measured model; start in percent, read the band note on the detail page, then switch.
• Set a look-ahead for slow drifts. At 0 a model that sees the previous reading will happily follow a creeping sensor fault. A few hours of look-ahead is the setting that catches it.
• Excluded periods do not silence the rule. They shape training only; pause for field work.
• Watch the model chip. A rule can read Normal for a week while Retrain failed ×4 sits beside it; the first chip is about the creek, the second about the thing judging it.
• A worse score after a change is information. The Change column compares each run with the one before; a retrain that got worse usually means the site changed or a chosen variable stopped helping.
• Anyone with read access can open the form to see a rule's configuration and use the chart tools; only the save needs edit rights.