Reign Edge Library · Handbook

Where LLMs Belong in a Trading System

Language models are currently the most misapplied technology in retail trading. This handbook draws the line precisely: what an LLM actually is, the jobs it does better than any tool before it — clarifying, structuring, and critiquing trading rules — and the jobs it must never hold: predicting prices, generating live signals, sizing positions, enforcing risk. One principle organises all of it: the LLM works where a human collaborator works, before deployment; deterministic code works where money moves.

LLM System Design Backtesting Determinism Risk Evals

01Why This Handbook Exists

Every trading feed now carries the same two performances: a chatbot screenshot "analysing" a chart, and a demo of an agentic trading bot promising hands-free execution. Most of this content applies a language model in exactly the place it is worst suited to — and most of the backlash concludes, just as wrongly, that language models have no place in trading at all. This handbook exists to draw the line precisely, because the line is real, it is sharp, and almost nobody is drawing it.

The claim this handbook defends is narrow and strong. A large language model is simultaneously the best rule-elicitation and critique tool ever put in front of a trader and one of the worst live decision engines imaginable — and both facts follow from the same underlying properties. Once you understand what the model actually is (Section 02), where it belongs stops being a matter of taste and becomes a matter of engineering.

The hype position

"Let the model watch the market and trade for you." Puts a probabilistic text generator directly on the money path — non-deterministic, un-backtestable, unaccountable. Section 04 catalogues how each variant of this fails.

The dismissal position

"LLMs hallucinate, so keep them away from trading entirely." Throws away the one tool that can interrogate a discretionary trader's rules at expert level, before any capital is exposed. Section 05 catalogues what this position forfeits.

This is vendor-neutral reference documentation, not a product pitch. It assumes you already trade and already think in systems — the mechanics of backtesting, walk-forward validation, and overfitting are covered in the Trading Systems Handbook and are referenced here rather than restated. What this handbook adds is the layer that document deliberately leaves open: when a language model enters the picture, which jobs it should hold, which it must never hold, and the engineering discipline that keeps the two apart.

The thesis An LLM belongs in a trading system exactly where a trusted human expert collaborator belongs: in the design room, before deployment, with a human making the final call. It never belongs where money moves. Everything that follows is the working-out of that one sentence.

02What an LLM Is, for Trading Purposes

Strip away the anthropomorphism and the marketing, and what remains is an engineering component with a spec sheet. You would never wire an unfamiliar component into a live system without reading its spec sheet. Read this one first, because every placement decision in this handbook follows from it.

The spec sheet, in plain terms

A large language model is a next-token predictor over text. Trained on an enormous corpus, it produces, for any prompt, a continuation that is statistically plausible given everything it has read. That single sentence explains both its power and its limits:

  • It has no internal market model. It does not track an order book, a position, or a data feed. When asked where a currency pair is going, it composes the most plausible text about that question — plausibility learned from prose, not from prices.
  • It is non-deterministic by default. Outputs are sampled. Ask the same question twice and you can receive two different answers, each fluently argued. Even at temperature 0, an updated model behind the same API name silently changes every answer (Section 09).
  • It is agreeable. Models are tuned to be helpful, and helpfulness bleeds into sycophancy: a prompt that telegraphs the answer you want tends to receive that answer. A trader looking for permission to enter will usually find it.
  • It is confidently wrong at random. Hallucination is not an occasional glitch; it is the same fluent generation process, applied where the corpus runs thin. The wrong answer arrives with the same polish as the right one.
  • Its knowledge stops at a cutoff. The model is a frozen snapshot of its training data. It does not know the current regime, this week's calendar, or anything after its cutoff — except what you paste into the prompt.

Measured against the job

Set that spec sheet against what a live trading decision engine actually requires, property by property:

PropertyWhat a decision engine requiresWhat an LLM provides
DeterminismSame inputs → same decision, every time, so the system can be backtested and auditedSampled output that varies run to run and shifts silently across model updates
CalibrationProbabilities you can size positions against, grounded in measured frequenciesFluent confidence uncorrelated with correctness — polish, not probability
Point-in-time knowledgeExactly the information available at decision time, nothing after itA frozen corpus with hindsight about the past baked in, and a blind spot after the cutoff
Latency & marginal costMilliseconds, at near-zero cost per evaluation, across every instrument and barSeconds per call, metered, through a third-party dependency
AuditabilityA rule you can point to and a log that explains every decisionAn answer you cannot decompose and could not regenerate
Willingness to disagreeA rule holds its ground no matter how much you want the tradeAn agreeable counterparty that mirrors the conviction in your prompt
The category error Asking a language model where price goes next is not a weak strategy — it is a category error, like asking a librarian to forecast the weather because the library contains meteorology books. The model returns the most plausible continuation of text about markets. Plausible text about markets is what commentary is made of, and commentary has never been an edge.

None of this makes the component useless. Breadth of knowledge, fluency in language, tireless willingness to ask and answer questions — the very properties that disqualify an LLM as a price oracle are the properties you want in a design-room collaborator. The rest of this handbook is about keeping those two roles from contaminating each other.

03The Dividing Line: Design Time vs Runtime

One principle carries this entire handbook, and every later section is an application of it. Split a trading system's life into two zones. Design time is where rules are drafted, formalised, tested, and criticised — before capital is exposed. Runtime is where live data becomes live decisions — where money moves. LLMs belong in the first zone. Only deterministic code belongs in the second.

The reason is an asymmetry in the cost of error. At design time, a wrong suggestion is cheap: it is caught by validation, contradicted by a backtest, or vetoed by the trader reviewing it. The zone is built out of safety nets — nothing that happens there touches an account. At runtime there is no net: an error is position-sized and charged to the account at market speed. So the two zones impose opposite requirements. Design time can tolerate slowness, cost, and non-determinism in exchange for judgement and breadth — which is why a human expert is useful there, and why an LLM is too. Runtime tolerates none of these, which is why neither a human of variable mood nor a probabilistic text generator has any business in it.

DESIGN TIME LLM welcome · human in the loop RUNTIME deterministic · money moves elicit & clarify rules translate into structure critique · gaps · conflicts trader reviews every step strategy spec reviewed & frozen market data deterministic rules engine orders · alerts · kill switch no LLM anywhere on this path errors here are caught by review errors here are position-sized
The dividing line. In the design-time zone, the LLM and the trader iterate freely — errors are caught by review, validation, and backtests. The only artefact that crosses the boundary is a frozen, versioned strategy spec. The runtime zone executes that spec deterministically, with no language model anywhere on the path between market data and orders.

The placement test

One question decides every placement
  • Before an LLM call goes anywhere in your system, ask: if this call returns something wrong, what stands between the output and an order?
  • If the answer is "a human review, a validator, and a backtest" — the call is at design time. Proceed.
  • If the answer is "nothing", or "another LLM" — the call is at runtime. Replace it with deterministic code, or delete it.

Two placements look like exceptions and are not. A sealed LLM-backed input (Section 07) is consumed at runtime but was computed at design time — by the time the system reads it, it is frozen data, not a live model call. And an LLM that explains a report or narrates a filled order at runtime is read-only: it sits after the decision, not before it, and nothing it says feeds back into an order. The line is not "no LLM anywhere near live trading". It is: no LLM upstream of an order.

04The Failure-Mode Catalogue

The same handful of misplacements accounts for nearly every LLM-trading product, tutorial, and viral demo in circulation. Each row below is something real people are building right now. Each fails for reasons that are structural — properties of the component, not defects of any particular model — which is why the fix is never a better prompt, and always the same relocation.

What people buildWhy it failsWhat to do instead
The price oracle — "where is this pair going this week?" No internal market model; the answer is plausible commentary, uncalibrated and unaccountable. Ask twice, get two directions, both confidently argued. Deterministic signal rules you can backtest. Use the LLM to help formalise them (Section 05), never to replace them.
The per-bar signal generator — call the model on every new candle for a verdict Non-deterministic, so no backtest of it can be reproduced (Section 09); seconds of latency and metered cost on every bar (Section 12); a third-party outage becomes a trading outage. An LLM-free runtime scanner executing a frozen spec (Section 14). The model helped write the spec; it does not run it.
The "should I buy?" chat — pasting a chart or a position into a chatbot before entering Sycophancy: the prompt telegraphs the desired answer and receives it. The chat validates conviction; it does not test it. No record, no accountability, no sample to evaluate. A written rule set that decides entries — and an LLM interrogating the rules at design time, where its scepticism is useful and its agreeableness is harmless.
The LLM risk manager — "watch my exposure and cut positions if it gets dangerous" A probabilistic component can only enforce a limit probabilistically, and a limit enforced probabilistically is not a limit (Section 11). Fails exactly in the tail events it exists for. Hard-coded position sizing, exposure caps, drawdown stops, and kill switches — deterministic code with guarantees.
The codegen bot — the model writes free-form execution code from a description Arbitrary generated code on the money path: a security surface, subtle logic errors that compile cleanly, and a different program on every generation — which one did you backtest? Constrained composition of vetted, deterministic building blocks, machine-validated before anything runs (Section 06).
The self-grading loop — the model proposes a strategy, backtests it, and pronounces it robust Agreement bias meets no statistical grounding: the proposer adjudicating its own proposal, with hindsight contamination on top (Section 08). "The model judged it robust" is not evidence. An out-of-sample statistical gate plus human sign-off between every proposal and deployment (Section 10).

Read the middle column again and notice that every failure is one of five recurring reasons — non-determinism, un-backtestability, latency and cost, hallucination, no accountability — and that all five trace back to the spec sheet in Section 02. These are not bugs that the next model release fixes. A better model writes more convincing commentary, agrees more persuasively, and generates subtler wrong code. Capability improves; the category error remains.

A better prompt does not fix a placement error Every row above puts a probabilistic text generator upstream of an order. Prompt engineering changes what the component says; it cannot change what the component is. The fix is always the same: move the LLM to design time, and put deterministic code where it was standing.

05What LLMs Are Genuinely Good At

The dismissal position forfeits real capability. Discretionary traders hold rules in their heads that they have never fully articulated — and the hard part of systematising a discretionary edge was never the code, it was the articulation. That is a language problem, and language models are the best instrument ever built for it. Every job in this table sits at design time, feeds a deterministic artefact, and leaves the final call with a human.

The jobWhy an LLM fitsThe honest caveat
Rule elicitation Asks the clarifying questions a systems engineer would — "which timeframe defines the trend? close basis or intrabar? what happens at session boundaries?" — tirelessly, systematically, without ego. It will happily elicit a bad idea in perfect detail. Elicitation surfaces your rules; it says nothing about whether they carry an edge. The backtest decides that.
Contradiction & gap detection Reads a full rule set at once and spots what a tired human misses: two rules claiming the same bar, an exit that can never trigger, an unhandled state when entry and stop conditions fire together. It finds the gaps it recognises. An empty findings list is not a proof of completeness — validation and testing still carry that weight.
Translation into formal structure Maps "I buy pullbacks in an uptrend, but not into news" into a formal, machine-checkable structure (Section 06) — the bridge from language to something a backtest engine can execute. A fluent wrong translation reads exactly like a right one. Every translation must round-trip back to the trader for confirmation before it is trusted (Section 13).
Explaining backtest results Turns a wall of metrics into plain language — what the drawdown profile means, where the expectancy concentrates, which question to ask next. Only when grounded in the actual numbers. Left unconstrained, the model smooths over what the data does not support. The explanation layer must quote the report, not improvise on it.
Trade-journal coaching Reads months of journal entries and surfaces behavioural patterns — widened stops after losing days, size creep after winning streaks — that no trader spots in their own writing. These are observations, not verdicts: journal samples are small and the model does not compute significance unless the surrounding system forces it to.
Research assistance Summarises methods and literature, explains an unfamiliar technique, drafts the survey you would otherwise spend a weekend on. Hallucinated citations and misremembered details are a known failure mode. Check primary sources before you build on any claim.
What every good job has in common All six jobs share one shape: they sit upstream of capital, their output is reviewed by a human, and what finally reaches the market is a deterministic artefact the LLM helped write but does not run. The model contributes judgement where judgement is cheap to check — never where it is charged to an account.

06The Constrained-Output Principle

Granting the LLM its design-time seat raises the next question: what exactly does it write down? The seductive answer — let the model write the strategy as free-form code — reintroduces at design time most of the dangers just evicted from runtime. The disciplined answer is a principle: the LLM fills a constrained, validated structure; it never authors free-form executable code.

Why free-form generation fails

  • Security surface. Generated code that touches execution is arbitrary code on the money path — including whatever the model absorbed from the text it was shown. The attack and accident surface is the entire language runtime.
  • Determinism. Two generations of "the same" strategy differ in structure, edge cases, and bugs. Which one did you backtest? Which one is live? A strategy that cannot be regenerated identically cannot be trusted to be what was tested.
  • Testability. Every generated program is a novel artefact demanding full review from zero. Nothing accumulates: no component earns trust, because no component survives from one generation to the next.
  • Generation reliability. Models are dramatically more reliable filling a constrained structure than sustaining correctness across hundreds of lines of open-ended code. Constraint plays to the component's strengths instead of its weaknesses.

The alternative: compose, don't author

Give the model a fixed vocabulary of vetted, deterministic building blocks — each one written by humans, tested once, trusted thereafter — and have it express the trader's rules as a composition of those blocks, emitted into a schema that a machine validates before the result exists as a strategy at all. Malformed output is rejected at validation time, loudly and cheaply — not discovered at runtime, expensively. The change turns "did the model write correct code?" (unanswerable in general) into "does this document conform, and is this composition well-formed?" (mechanically checkable, every time).

Free-form generation

The model authors novel executable code. Every output is unique, unaudited, and load-bearing. Review burden is total and recurring; failure is discovered downstream, sometimes by the account.

Constrained composition

The model arranges pre-vetted components inside a validated structure. Creativity is spent on expressing the trader's rules, not on reinventing execution. Failure is caught by the validator, upstream of everything.

The pharmacy analogy A pharmacist compounds a prescription from approved ingredients under a checkable formula; they do not synthesise novel molecules per customer and skip the trials. Constrained output is the same posture: composition within a certified vocabulary, so that safety is a property of the system, not a hope about each individual generation. It is also why "the model wrote my trading bot in ten minutes" demos should read as a warning, not a capability.

07Sealed LLM-Backed Inputs

Some genuinely useful trading inputs are linguistic. The tone of news flow around a currency. Whether a central-bank statement reads hawkish or dovish against expectations. Deterministic code cannot read; a language model can. Does that break the dividing line? Not if the judgment is sealed — reduced to data before the system ever sees it.

The sealing contract

Five conditions, all mandatory
  • Reduce. The LLM's output is a small deterministic value — a category, a bounded score — produced under a fixed rubric. Free text never enters the decision path.
  • Pin. Model version and prompt version are fixed and recorded; generation runs at temperature 0. The judgment is a function, not a conversation.
  • Precompute. Values are computed offline, in batch, off the live path. The runtime reads a cache; it never calls a model.
  • Cache immutably. Each value is keyed by instrument, timestamp, model version, and prompt version — computed once, stored forever. Backtest and live read the same cache.
  • Compose deterministically. The sealed value enters the rules like any other input — "if tone is below the chosen level, stand aside" — ordinary deterministic logic around a frozen number.
sentiment(instrument, t) = f(model vX, prompt vY, documents dated ≤ t)
// same key → same value, forever; the runtime reads the cache, never the model

Under this contract the LLM ran once, in the past, under pinned conditions — and what remains is data. From the system's point of view a sealed input is indistinguishable from an indicator: deterministic, replayable, composable. The dividing line of Section 03 holds, because the model call happened at design time even though its output is consumed at runtime.

Upgrades are versioned events, never drift

Swap the model, or edit the prompt, and you have created a new input series — the judgments will differ, somewhere, in ways that matter. The disciplined path: recompute the series under the new versions, re-run the backtest against it, compare, and promote deliberately, recording the change. What is never acceptable is silent drift — an aliased endpoint upgrading beneath you, so that the strategy quietly becomes one that nobody tested and nobody chose.

Sealed means sealed The moment a live model call sneaks back into the loop — "just re-score today's headlines on the fly" — every guarantee in this section evaporates: determinism, replayability, backtest–live parity, all of it. The cache is not an optimisation. The cache is the design.

08The Leakage Problem Unique to LLMs

Classic look-ahead bias — the backtest accidentally reading tomorrow's data — is a plumbing error, and plumbing can be fixed (the Trading Systems Handbook catalogues those defences). LLMs add a new leak that no plumbing removes: the model itself has read the future. Any model trained on recent history "knows", inside its weights, what happened after any earlier date in its corpus.

Ask a current model to judge, "as of" some past date, the news flow around a currency shock, a rate-cycle turn, a liquidity crisis — and its judgment is contaminated by knowing how the story ended. It has read the post-mortems. A backtest that replays live LLM judgment across historical events is therefore structurally optimistic: the component being tested holds hindsight the deployed component will not have. This is not misuse; it is what a trained model is. You cannot prompt away the weights.

model training corpus (in the weights) backtest window judged with hindsight baked in knowledge cutoff post-cutoff: genuinely unseen time →
The leakage unique to LLMs. A backtest window that falls inside the model's training corpus is inside the model's known past — every "as of" judgment there is made by a component that has read what happened next. Only post-cutoff data tests the model the way live trading will.

Discipline, and its limits

DisciplineWhat it fixesWhat it cannot fix
Point-in-time inputs — prompts contain only documents dated at or before the decision timeStops you from literally feeding tomorrow's headlines into yesterday's judgment; preserves basic backtest hygieneThe outcome knowledge already inside the weights
Frozen caches + pinned versions (Section 07)Makes the contamination at least stable and reproducible — the same judgment every replay, not a moving targetThe contamination itself
Time-gated evaluation — judge the LLM component only on data dated after the model's cutoffProduces genuinely clean evidence of live skill — the model demonstrably could not have known outcomesSample size: post-cutoff history is short and accumulates at the speed of the calendar
Disclosure — the backtest report states which portion overlaps the training windowKeeps the report honest and the reader properly scepticalNothing. It is a caveat, not a repair — which is exactly why it must always be present
The irreducible caveat Any backtest in which LLM judgment is exercised over dates inside the model's training window carries an unremovable optimistic bias, and an honest report says so in print. Weight the sealed component's contribution accordingly, and let post-cutoff performance — not in-corpus performance — carry the evidential load. A vendor backtest of an LLM-driven strategy that does not carry this caveat is telling you something important about the vendor.

09Determinism & Reproducibility

A backtest you cannot re-run bit-for-bit is not evidence — it is an anecdote with a chart. Reproducibility is what separates the two: same spec, same data, same engine, same trade list, down to the last fill. The Trading Systems Handbook treats this as table stakes for any system; LLMs threaten it through four specific doors, each of which can be closed.

The four doors, and how to close them

  • Sampling. Default generation is stochastic. Close it: temperature 0 for any output that feeds the system — greedy decoding, the same continuation every time for a fixed model build.
  • Silent model updates. "Fixed model build" is doing the real work: an aliased endpoint that re-points to a newer model changes every output with no change in your code. Close it: pin named model snapshots, never floating aliases.
  • Untracked prompts. A prompt edit is a logic change. Close it: prompts are versioned artefacts recorded in the spec, not strings someone tweaks in place.
  • Live calls in the loop. The unclosable door — so it stays shut entirely. Any live LLM call inside a backtest makes the run unrepeatable by construction. Close it: cached outputs (Section 07), so re-runs replay, never re-ask.

Note the layering: temperature 0 and pinned versions make outputs deterministic in principle; caching makes them deterministic in fact, surviving provider deprecations, outages, and pricing changes. The cache is the only airtight guarantee — call once, store, replay forever.

Re-running the backtestDeterministic system (sealed inputs included)Live LLM calls in the loop
Same trade list?Bit-for-bit identical, every runDifferent — slightly or wildly, you cannot know in advance
A changed result meansYou changed something: diff the spec, the data, or the enginePossibly nothing — noise from resampling, or a model update you never chose
Debugging one weird tradeReplay the exact state that produced itArchaeology on a call that will never return the same answer again
Verifiable by a third party?Yes — same artefacts, same result, anywhere, any timeNo — the "same" experiment is unrepeatable even by its author
Status as evidenceEvidenceAn anecdote that cost API credits
The one-line standard If re-running your backtest can change its result, you do not have a backtest — you have a slot machine that prints research. Determinism is not a preference or a style; it is the precondition for a backtest meaning anything at all.

10LLM Proposes, Statistics Dispose

Give a language model a strategy and a backtest report and it will propose improvements — fluently, plausibly, endlessly. This is genuinely useful, and genuinely dangerous, for the same reason: articulate justification feels like analysis. It is not. A model can argue either side of any parameter change with identical polish, and the polish carries zero information about whether the edge is real.

The governing rule is a separation of powers. The LLM may propose; only out-of-sample statistics — and the human reading them — may dispose. Every proposal, whether it came from the trader or the model, faces the same gate:

  • A sample large enough to mean something. A handful of trades distinguishes nothing from nothing.
  • Out-of-sample evaluation with walk-forward separation — performance on data the proposal was not shaped by (mechanics in the Trading Systems Handbook).
  • Multiple-testing discipline. Every additional variant you try raises the bar for all of them — search hard enough and something always looks good by luck.
  • Pre-registered criteria. Decide what would count as "better" before the results exist. Moving the goalposts afterwards is curve-fitting with extra steps.
Multiplicity: the productivity trap An LLM makes proposals nearly free, and free proposals multiply. Fifty cheap variants are fifty draws from the luck distribution — the best of them will look impressive, edge or no edge. The more prolific the proposer, the more sceptical the gate must become. A tireless idea generator wired to a lenient gate is an overfitting machine with excellent manners.

The promotion path

Every proposal walks the same one-way path
  • 1 · Propose. The LLM (or the trader) suggests a change, with its reasoning on the record.
  • 2 · Formalise. The change is expressed in the validated structure of Section 06 — no informal "just tweak it".
  • 3 · Backtest. Deterministic re-run, costs included, sealed inputs replayed from cache.
  • 4 · Gate. Out-of-sample, walk-forward, multiplicity-adjusted evaluation. Most proposals should die here. That is the gate working.
  • 5 · Human sign-off. A person reviews, understands, and accepts the change — or declines it regardless of the numbers.
  • 6 · Version & deploy. The new spec is versioned; the old one remains, diffable, in history.

The step that must never exist: the model adjudicating its own proposal. "I analysed the backtest and the improvement is robust" is the proposer grading its own homework — agreement bias in a lab coat. The LLM may explain the gate's verdict afterwards, in plain language, and that explanation is often where the next good question comes from. But the verdict itself belongs to the statistics and the human, always, with no exception for confidence, eloquence, or how much you like the proposal.

11Risk Management Is Never an LLM

Every other section of this handbook weighs trade-offs. This one does not, because risk control is the one place where "usually works" and "does not work" are the same thing. Position sizing, exposure limits, drawdown stops, kill switches — these are deterministic code with hard guarantees, and nothing else is fit for the job.

A risk limit is only a limit if it fires every time, in bounded time, under load, on malformed inputs, during the provider outage, in the flash move — precisely the conditions under which it earns its existence. A probabilistic text system can promise none of that. It can be persuaded, confused, rate-limited, or simply wrong, and any one of those, once, in the wrong hour, is the account. A limit that holds probably is not a limit; it is a mood.

An LLM may

Explain a risk report in plain language. Draft a risk policy for human review. Interrogate your limits at design time — "what happens to exposure if these three positions correlate in a crisis?" — and surface the scenario you had not considered.

An LLM must never

Size a position. Decide whether a limit has been breached. Choose whether to flatten. Hold the kill switch, or stand anywhere in the code path that does. Not as a fallback, not "with human oversight", not temporarily. Never.

The absolute A risk limit that can be argued with is not a risk limit. An LLM in the enforcement path is exactly that — a limit that can be talked out of its job by the market, by a cleverly framed input, or by its own trained agreeableness. Risk enforcement is deterministic code, with guarantees, full stop.

12Cost, Latency, Observability

Suppose you ignored every argument so far and wired a model into the live loop anyway. The economics and the operations fail on their own, independently of everything else. This section is the boring, decisive arithmetic — and the operational discipline any LLM component you do keep must meet.

The arithmetic of per-bar calls

Illustrative arithmetic — hypothetical numbers Twenty instruments on 5-minute bars is roughly 288 bars per instrument per day — about 5,760 model calls a day, some two million a year, every one metered, every one taking seconds against a bar that lasts five minutes. And a backtest over a few years of history re-buys the entire bill on every re-run — which is why nobody who builds this ever re-runs honestly. A deterministic scanner covers the same ground in milliseconds at effectively zero marginal cost.

The design-time architecture inverts the cost curve. LLM spend concentrates where the value is: a bounded number of design-room interactions per strategy — elicitation, translation, critique — plus batch precomputation of any sealed inputs (Section 07). Runtime cost is data and arithmetic. The bill scales with how much designing you do, not with bars × markets × forever.

Operational propertyPer-bar LLM in the loopDesign-time LLM + sealed inputs
Call volumeGrows with bars × instruments, foreverBounded per strategy; batch jobs for sealed series
Latency on the live pathSeconds per decision, provider-dependent, variableNone — cache reads and arithmetic
Marginal cost of a backtest re-runThe full API bill, again, every runZero — deterministic replay
Runtime failure surfaceProvider outages, rate limits, timeouts, silent model swapsNone — no live model dependency exists
Drift detectionPractically impossible — no fixed baseline to drift fromEval set re-run on any change; drift is a diff, not a feeling

Observability for the LLM components you keep

Design-time and sealed components still deserve production discipline. The pattern, independent of any particular stack:

  • Log every call — inputs, outputs, model version, prompt version, timestamp. An unlogged LLM call is an unauditable decision influence.
  • Version everything — prompts, rubrics, and model pins live in the spec, so "what produced this value?" always has an exact answer.
  • Keep an evaluation set — a fixed battery of representative cases with known-good outputs, re-run whenever anything changes: a model upgrade, a prompt edit, a rubric revision. Drift becomes a measured diff against a baseline, caught before promotion — never a hunch about the system "feeling different" discovered in production.
The outage test A model provider's outage should be incapable of touching your live trading — the runtime holds no connection to pull. If a provider status page matters to your open positions, an availability dependency has crept onto the money path, and Section 03 has been violated somewhere. Find it.

13Honest Limits

Not everything a discretionary trader does can be formalised — and any tool that pretends otherwise is lying in a way that will eventually cost its user money. Some rules resist language because the trader cannot articulate them; some resist because there is no stable rule underneath, only pattern recognition that may or may not be real. What separates an honest system from a dangerous one is not coverage. It is what happens at the wall.

There are exactly two behaviours at the wall. The honest one: flag the rule as untranslatable, show the trader precisely what could not be captured, and let them decide — adopt an explicit named proxy they approve, keep that element discretionary, or drop it. The dishonest one: silent approximation — quietly substituting something formalisable and plausible, and never saying so. The second behaviour is the trust-destroying failure mode of this entire product category, because the backtest then tests a strategy the trader does not actually trade. Live results diverge from the backtest, the divergence has no visible cause, and the trader is left auditing a system that misrepresented itself on day one.

The phraseWhy it resists formalisationThe honest handling
"Feels overextended"Could mean distance from a mean in volatility units, a run of one-sided closes, or something the trader cannot name — often it is genuinely unclear to the traderElicit candidate definitions, present each as an explicit, named proxy, and backtest only what the trader confirms as a fair translation
"Smart money is accumulating"A narrative about unobservable participants; a dozen incompatible operationalisations circulate, none agreedEither adopt one explicit, named definition the trader signs off on — or flag it untranslatable and leave it discretionary
"I just don't like this setup"An unarticulated veto — possibly encoding real, unverbalisable pattern recognition; possibly encoding nothingKeep the human veto outside the formal system: rules propose, the trader may still decline. Log every veto and let the journal reveal, over time, whether the veto carries information
Fidelity beats coverage A faithful formalisation of 80% of your rules, with the untranslatable remainder named and visible, is worth incomparably more than a "complete" formalisation that silently invented the last 20%. The first is a system you can trust and improve. The second is a stranger wearing your strategy's name — and you will meet the difference live, at the worst possible moment, with no way to diagnose it.

This is also the deeper reason the round-trip in Section 05 is non-negotiable: every translation goes back to the trader for confirmation before it is trusted, precisely because a fluent approximation is indistinguishable, on the surface, from a faithful one. Honesty at the wall is not a UX nicety. It is the difference between a formalisation tool and a fiction generator.

14Reference Architecture & Checklist

Assembled, the doctrine of this handbook is one picture. An LLM works the design room: clarifying, structuring, critiquing — with the trader reviewing every artefact. What leaves the room is a versioned, deterministic spec. A deterministic engine backtests it, replaying any sealed inputs from cache. A statistical gate and a human approval stand between the backtest and live capital. The runtime that finally watches the market contains no language model at all — and its risk layer is hard-coded. Monitoring and the journal feed the next design cycle, closing the loop through the human, never around them.

DESIGN TIME trader's rules LLM: clarify · structure · critique versioned spec backtest engine · deterministic replay statistical gate + human approval out-of-sample · walk-forward · multiplicity rejected → redesign approved RUNTIME — LLM-FREE risk limits & kill switch hard-coded market data scanner runs the frozen spec orders · hard risk limits monitoring & journal feed the next design cycle
The reference architecture. The LLM's entire footprint is the design-time layer (plus batch precomputation of sealed inputs, not shown). Between the spec and live capital stand a deterministic backtest and the statistical gate with human approval — the only door to runtime, and it swings one way. The runtime is LLM-free, and its risk layer is hard-coded.

The checklist

Before you put an LLM anywhere near a trading decision, ask
  • 1. If this LLM call returns something wrong, what stands between its output and an order? (Section 03)
  • 2. Can I re-run my backtest and get the identical trade list, bit for bit? (Section 09)
  • 3. Is every model output on the decision path reduced to a validated, deterministic value — or is free text steering money somewhere? (Sections 06–07)
  • 4. Are model and prompt versions pinned, and is upgrading either one a deliberate, re-tested, recorded event? (Sections 07, 09)
  • 5. Does any backtest window overlap the model's training corpus — and does the report disclose it? (Section 08)
  • 6. Who judges the model's proposals — out-of-sample statistics and a human, or the model's own confidence? (Section 10)
  • 7. Can every risk limit fire with no LLM anywhere in its code path? (Section 11)
  • 8. If the model provider goes down at 2 a.m., does my live system even notice? (Section 12)
  • 9. When a rule cannot be faithfully formalised, does the system say so — or silently approximate? (Section 13)
The line, one last time The question was never whether language models belong in trading. They do — as the sharpest design-room collaborator a discretionary trader has ever had. The question is where. LLMs where a human expert belongs: before deployment, under review, upstream of the gate. Deterministic code where money moves. Hold that line and the model makes your system better. Cross it and the model is your system — and nobody, including the model, knows what that system will do next.