Where machine learning did help: forecasting market volatility

Guessing whether a stock goes up next month turned out to be close to impossible in our tests. Guessing how bumpy the market will be is a different problem, and an easier one. Why volatility is more predictable Calm periods tend to follow calm periods, and stormy ones follow stormy ones (volatility clustering). That persistence is real structure a model can learn. Direction has very little of it. The setup Target: the variance of S&P 500 (SPY) daily moves over the next 20 trading days. Baseline: HAR, the standard textbook model. It's a regression on recent volatility over the last day, week and month. Measure: QLIKE, a loss function suited to variance forecasts (lower is better). Test: a 2025–26 holdout, with the Diebold-Mariano test to check whether the difference is more than noise. Results Model QLIKE (lower is better) vs HAR HAR (baseline) 1.047 — LightGBM on SPY features (pre-registered) 0.816 p = 0.047 GARCH(1,1), Student-t (exploratory) 0.385 p = 0.044 Chronos-Bolt, zero-shot (exploratory) 0.594 p = 0.065, not significant The LightGBM model was the one we registered before the test, and it beat HAR, just inside the usual 5% line. The exploratory runs came afterwards, so they count as hints, not findings. Still, it's notable that the decades-old GARCH model did best of all, and a pretrained time-series foundation model, with no training on this data, landed in between. Keeping it honest One window, one market, p-values close to 0.05. That's a small win. So the model keeps forecasting live, every night, and the forecasts are logged before the outcome is known. If the edge was luck, the live record will show it. The bigger lesson It matches the research paper the project is built on: machine learning is much more useful for measuring risk than for picking winners. From Quantooo Lab, the makers of quantooo.com, a free, open and educational quant lab: every formula, every test and every failure is public. See it here: https://quantooo.com/research Educational, not investment advice.