Detecting the regime.

01Foundations: what a market regime is

A regime is a persistent state of the market in which returns share stable statistical properties — and persists long enough that conditioning your strategy on it pays for the lag in detecting it. Foundational

Four regime axes matter for a discretionary trader's rules. This handbook focuses on the first two, because they decide the trend-versus-reversion question directly; the other two are overlays you bolt on later.

The autocorrelation duality

Strip the indicators away and the entire trend/reversion question is one number: the autocorrelation of returns.

The null hypothesis is the random walk: ρ(k) = 0 for all k, no exploitable structure. Trend-following bets you can reject the null upward; mean-reversion bets you can reject it downward. Almost every detector here — variance ratio, Hurst, ADX, HMM — is a different lens on this same quantity, trading off responsiveness, robustness, and lag.

The equity-vs-FX asymmetry

The single most important framing for your two instruments: a CFD equity index and an FX major are not the same kind of time series, and they do not get the same regime toolkit.

This is why a long-only S&P 500 hypothesis is rational — you're harvesting a real risk premium — while a symmetric mean-reversion-short on the same index fights structural drift. On FX majors there is no drift to harvest, so the entire game is classifying trend versus range and deploying the matching engine. Internalize this and most of Part 6 writes itself.

Stylized facts that break naive models

Returns violate textbook assumptions in consistent ways. A regime model that ignores these is fitting noise.

02Data & preprocessing

Regime detection is exquisitely sensitive to data quality and to look-ahead leaking in through preprocessing. Get this layer wrong and every downstream model is measuring artefacts. Foundational → Intermediate

What your data actually is

S&P 500 CFD mechanics

A cash-index CFD is not the index. Three mechanics directly affect a long-only hold:

• Overnight financing. Holding a CFD long incurs a daily funding charge ≈ (rate + markup) × notional / 360. In a positive-rate environment the long pays to carry. A filter that flattens during bear/high-vol stretches saves financing and avoids drawdown — a hidden second benefit.
• Dividend adjustments. On index ex-dividend days, long CFD holders receive an adjustment (shorts pay). It partly offsets financing; your backtest must credit it or you'll understate long returns.
• Sessions & gaps. The CFD trades nearly 24/5 with a maintenance break and thin off-hours liquidity; spreads widen outside US cash hours. Model spread by session, not as a constant.

For GBPUSD/EURUSD: no dividends, but swap/rollover applies overnight, the spread is tight in the London/NY overlap and wide in the Asian session, and there are weekend gaps. A volatility estimator that ignores the gap will misread Monday.

Bars: time is the worst sampling clock

Spot FX has no true volume, so dollar/volume bars are approximate (tick count proxies activity). For a first build, time bars are acceptable — just know they inflate quiet-period sample counts and bias any test that assumes i.i.d. returns.

Returns, stationarity, and the memory trade-off

• Log returns (r_t = ln(P_t/P_t-1)) over simple: additive, symmetric, better-behaved. Model on these; convert to simple for P&L.
• Stationarity. Test with ADF (null: unit root) and KPSS (null: stationary) — complementary, run both. Returns are usually stationary; prices are not.
• The differencing dilemma. Differencing prices to returns makes them stationary but destroys memory. Fractional differentiation differences by a fractional order d ∈ (0,1), the minimum needed to pass ADF while keeping maximum memory. Worth knowing when a model needs both; overkill for a first build.

Volatility estimators

Vol is the input to vol-regimes, sizing, and stop distance. Pick by what data you have and whether gaps matter.

Features for regime detection

Keep the set small and interpretable:

• Returns (log) and a rolling mean (drift proxy).
• Realized vol (Yang-Zhang) and vol-of-vol.
• Trend strength — ADX, efficiency ratio, R² of a rolling linear regression on price.
• Autocorrelation features — rolling lag-1 ρ, variance ratio.
• Distance from a slow MA (z-scored) — stretch / reversion pressure.

03Classical regime detection: statistical tests

The "is this series trending or reverting?" test battery — model-light, interpretable, and the right first answer before reaching for HMMs. Intermediate

Variance Ratio test (Lo–MacKinlay)

The canonical random-walk test. If returns are i.i.d., the variance of a q-period return is exactly q × the variance of a 1-period return.

Use the heteroskedasticity-robust statistic (returns aren't homoskedastic). Sweep q (2, 4, 8, 16, 32) and plot — the shape tells you the horizon at which structure appears.

Hurst exponent

Summarizes persistence in one scalar via how the range of the series scales with horizon.

Two estimators: classic R/S analysis (rescaled range) and DFA (detrended fluctuation analysis), which is more robust on non-stationary series.

Stationarity / mean-reversion: ADF

The Augmented Dickey-Fuller test doubles as a mean-reversion detector: rejecting the unit-root null on a price (or a spread) says the series is stationary, i.e. mean-reverting. This is the engine behind cointegration testing below.

Ornstein–Uhlenbeck: how fast does it revert?

If a series is mean-reverting, the actionable question is the half-life — how long to revert halfway to the mean. Model it as an OU process:

A half-life of 3 bars → fast reversion, scalp it. A half-life of 200 bars → barely reverting, don't. This number sizes your mean-reversion lookbacks and tells you whether a pair is tradeably reverting at all.

Cointegration (for pairs / stat-arb regimes)

A single FX major rarely mean-reverts cleanly, but a combination of correlated instruments can. Cointegration means a linear combination of non-stationary series is stationary.

04Indicator-based regime filters

The practitioner toolkit: fast, transparent, no model-fitting, and what a discretionary trader actually puts in a rule. These are filters — they gate a strategy on/off — more than predictors. Foundational → Intermediate

The two worth implementing first

ADX is the trader-native trend-strength gate, built from directional movement (DI+, DI−): DX = 100·|DI+ − DI−|/(DI+ + DI−), then Wilder-smoothed into ADX. Above ~25 the market trends with strength (direction from DI+/DI−); below ~20 it ranges. Its weakness is lag — ADX confirms a trend already underway.

Kaufman's Efficiency Ratio is underrated for regime gating and computationally trivial:

Volatility-regime filters

A high/low vol classifier from realized-vol (Yang-Zhang) percentile, ATR percentile, or a Bollinger/Keltner squeeze (Bollinger bands inside Keltner channels = compression). On equities the low-vol regime overlaps the uptrend, so a vol filter often is a usable bull/bear filter. On FX the link is weaker — vol regime and trend regime are more independent, so you need both.

05Probabilistic & ML regime models

Where you move from "estimate the property" to "infer a hidden state with a probability and a transition model." More power, more ways to fool yourself. Advanced

Markov regime-switching (Hamilton)

The econometric workhorse. An autoregressive model whose parameters (mean, variance) switch according to an unobserved Markov chain, estimated by maximum likelihood (Hamilton filter / EM). Output: the filtered/smoothed probability of each regime at each time, plus a transition matrix (persistence). A 2-state switching-variance model on S&P returns separates "quiet drift" from "stressed" almost out of the box.

Hidden Markov Models (HMM)

The same idea in HMM language, usually fit on features (returns + realized vol). Gaussian emissions per state; Baum-Welch (EM) to fit; Viterbi for the single most-likely state path; forward-backward for per-bar state probabilities.

Gaussian Mixture Models

A "static" cousin of the HMM — soft-clusters return/vol observations into regimes without a transition model. Cheaper, no temporal structure; good for labelling the current observation, weak on persistence.

GARCH family — volatility regimes specifically

The right tool when the regime you care about is volatility.

For equities use an asymmetric variant (EGARCH, GJR-GARCH) to capture the leverage effect — vol rises more on down days. Markov-switching GARCH combines discrete vol regimes with GARCH dynamics. Library: arch (Kevin Sheppard) — mature and correct.

Unsupervised clustering

K-means, hierarchical, or DBSCAN on engineered features (return, vol, trend strength) to discover regimes you didn't pre-specify. Useful for exploration and hypothesis generation; weak for live switching because clusters lack a transition model and are unstable to refits. A research lens, not a production switch.

Change-point detection

Instead of classifying every bar, detect when the regime breaks.

BOCPD is the one to know for live use — it updates a posterior over "bars since last change" each new bar, fully causal.

Kalman filters & trend filtering

Extract a smooth, time-varying trend and slope from noisy price — the slope's sign and magnitude is a continuous regime read.

• Kalman / local-linear-trend DLM — estimates level + slope online; the slope is a causal trend signal. Also gives a dynamic hedge ratio for cointegrated pairs. Libraries: pykalman (works, lightly maintained), filterpy.
• L1 trend filtering (Kim–Koh–Boyd) — fits a piecewise-linear trend by penalizing the second difference; the kinks are regime changes. Offline.
• HP filter (Hodrick-Prescott) — smooth trend, but two-sided → look-ahead. Fine for describing history, unsafe as a live signal unless one-sided.

Supervised ML for regime

Turn regime into a labelled prediction problem.

1. Label. The hard part. Triple-barrier (López de Prado): label each observation by which barrier — profit-take, stop, or time — price hits first. Meta-labeling: a secondary model predicts whether to act on a primary signal, sizing rather than directing.
2. Model. Gradient-boosted trees (LightGBM/XGBoost) on the Part 2 feature set predict regime or next-period return sign. Inspect feature importance.

Deep learning — know it, default against it

LSTM/Temporal-CNN/Transformer classifiers and autoencoders (reconstruction error spikes flag new regimes) can model regimes. For FX/single-index regime detection they are usually the wrong call: data-hungry, opaque, prone to overfitting on the few hundred regime transitions a decade contains, and rarely beating a well-built HMM or vol-target. Reach for them only with strong out-of-sample evidence and a reason simpler models failed.

Method selection

06From detection to strategy

Detection is worthless until it changes a position. Three ways to wire a regime signal into a strategy. Intermediate → Advanced

Worked example A — S&P 500 CFD, long-only, bull-regime filter

Hypothesis: the S&P 500 carries a persistent positive drift (equity risk premium), so a long-only exposure that steps aside during bear/high-vol regimes should keep most of the upside with far less drawdown.

Step 1 — validate the drift before building anything.

Step 2 — candidate regime filters (simplest first):

Step 3 — the CFD economics tilt the verdict. Because a long CFD pays overnight financing, a filter that flattens during the worst (bear/high-vol) stretches saves carry and avoids drawdown — two benefits. But every flip costs spread, and off-hours spread is wide, so don't evaluate on signals only: subtract financing, credit dividend adjustments, and charge realistic session-dependent spread.

Step 4 — what "success" looks like. On a drifting index, a good regime filter usually delivers similar CAGR with materially lower max drawdown → a higher Calmar/Sortino, not a higher raw return. If your filtered version also beats buy-and-hold on CAGR, be suspicious of look-ahead before you celebrate.

Worked example B — GBPUSD / EURUSD, trend-vs-reversion switch

Step 1 — classify the regime with an ensemble (they fail differently):

• ADX > 25 and ER > 0.35 → trending.
• Choppiness > 61.8 / ADX < 20 / Hurst < 0.5 → ranging.
• Optionally arbitrate with a 2-state HMM on (returns, vol) and require agreement before switching.

Step 2 — deploy per regime:

Step 3 — handle the switch. Don't flip instantly on a one-bar threshold cross — that's whipsaw. Require N-bar persistence or an HMM probability > τ, and scale down the outgoing engine while scaling up the incoming one rather than a hard cut. Sizing by regime confidence beats a binary switch here too.

The lag/whipsaw problem (applies to both)

Every detector lags; switching costs spread. Mitigations: a persistence requirement before acting; continuous sizing over binary flips; ensemble agreement to cut false switches; and explicitly budgeting the switching cost in the backtest — if turnover eats the regime edge, the filter is net-negative no matter how clever the detector.

07Backtesting & validation

Regime strategies are unusually easy to overfit — the regime label is itself fitted, and the number of true regime transitions in any sample is small. This is the discipline that keeps you honest. Advanced

The cardinal sin: look-ahead in the regime label

The label must be computable using only past data at every point.

• Expanding/rolling refit, not full-sample fit. Fitting an HMM/Markov model on the entire history then "backtesting" over the same history uses future data to set the present regime. At bar t, the model may only have seen data up to t.
• Filtered, not smoothed, probabilities. Smoothed probabilities at t use observations after t. Causal use requires filtered probabilities.
• Indicator warmup & no peeking. ADX/ER/MA all need warmup; the regime at t uses values computed at t with no centred windows.

Out-of-sample protocol

Did the edge survive multiple testing?

You tried many thresholds, lookbacks, and state counts. Some "worked" by chance.

• Deflated Sharpe Ratio (Bailey–López de Prado) — discounts the observed Sharpe for the number of trials, non-normal returns, and sample length. A Sharpe of 1.5 after 200 configurations is not a Sharpe of 1.5.
• Probability of Backtest Overfitting (PBO) — via combinatorially symmetric cross-validation, estimates the chance your "best" config is in-sample luck.

Metrics — measure per regime, not just overall

Costs are not optional

Charge spread (session-dependent), slippage, CFD overnight financing and dividend adjustments (S&P) or swap (FX). Then charge the switching turnover explicitly — the cost of every regime flip. A regime filter that's profitable gross and unprofitable net of switching costs is a negative-edge strategy with extra steps.

Robustness

• Parameter sensitivity — vary thresholds/lookbacks ±20%; a real edge degrades gracefully, an overfit one falls off a cliff.
• Regime stability over time — does the detector identify the same kind of regime across decades, or has its meaning drifted?
• Structural breaks — check performance across known macro breaks (2008, 2020, 2022) separately.

08Tools, libraries & stack

Python-first, matched to a typical quant backend. Maturity/maintenance is current to draft time — re-check at publish, since several of these move. Reference

09Pitfalls & anti-patterns

The recurring ways regime work goes wrong, each with its fix. All levels

1. Look-ahead in the regime fit. The #1 killer. Full-sample HMM fits, smoothed probabilities, centred indicator windows. → Expanding-window refit, filtered probabilities, causal indicators.
2. Overfitting regime count / thresholds. Adding states or tuning a threshold until the backtest sings. → BIC/AIC for state count; deflated Sharpe and PBO for thresholds; out-of-sample everything.
3. HMM label switching. "State 0" silently changing meaning across refits. → Map states to characteristics (sort by mean/variance) every fit.
4. Regimes that exist only in-sample. A beautiful structure that vanishes out-of-sample. → CPCV; check the regime means the same thing across decades.
5. Ignoring switching cost. A filter that's gross-profitable and net-negative once spread/turnover is charged. → Budget switching cost explicitly; prefer continuous sizing to binary flips.
6. Confusing a vol filter for a trend filter. They coincide on equities, not on FX. → Test the directional regime separately from the vol regime on FX majors.
7. Trusting a single Hurst/VR number. Noisy estimators read as precise switches. → Treat them as slow, soft indicators; cross-check estimators; ensemble.
8. Fancy beats simple — usually backwards. A deep net that underperforms a 200-SMA filter. → Earn complexity with out-of-sample evidence; a simple ADX/vol filter is a serious baseline, not a strawman.
9. The regime-is-itself-a-prediction trap. Treating the detected regime as ground truth rather than a lagging, uncertain estimate. → Size by confidence; never bet the account on a coin-flip probability.
10. Non-stationary regimes. Assuming the regimes themselves are stable. → Re-validate periodically; markets change, and so do their states.

10References & further reading

The foundational sources behind every method in this handbook. Reference

Foundational texts

• Hamilton, Time Series Analysis — regime-switching, the canonical treatment.
• Lo & MacKinlay, A Non-Random Walk Down Wall Street — variance ratio, the random-walk tests.
• López de Prado, Advances in Financial Machine Learning — fractional differentiation, triple-barrier / meta-labeling, purged CV, deflated Sharpe, PBO.
• Chan, Algorithmic Trading: Winning Strategies and Their Rationale — mean reversion, cointegration, half-life in practice.
• Kaufman, Trading Systems and Methods — efficiency ratio, adaptive indicators, the practitioner toolkit.
• Ang, Asset Management — economic interpretation of regimes.

Key methods (search terms)

Hamilton (1989) Markov regime-switching · Adams & MacKay (2007) Bayesian online change-point · Kim, Koh & Boyd (2009) L1 trend filtering · Killick et al. PELT · Bailey & López de Prado deflated Sharpe / PBO · Yang & Zhang volatility estimator.

Glossary

• Autocorrelation — correlation of a return with its own lagged value; its sign decides trend vs reversion.
• Variance ratio — variance of a q-period return ÷ q × the 1-period variance; > 1 trend, < 1 reversion.
• Hurst exponent — persistence scalar; < 0.5 reverting, > 0.5 trending.
• Half-life — bars for a mean-reverting series to close half the gap to its mean.
• Filtered vs smoothed probability — filtered uses data up to t (causal); smoothed uses the whole sample (look-ahead).
• Calmar ratio — CAGR ÷ max drawdown; the headline metric for a drawdown-focused filter.
• Deflated Sharpe — Sharpe adjusted for multiple testing, non-normality, and sample length.
• Walk-forward — repeated train-then-test marching through time, mirroring live refits.