testing money ideas

a backtest is a simple promise: take a trading idea, replay it against history, and see what would have happened. it sounds like measurement. most of the time it is closer to flattery.

i built a small research stack to do this properly. rust for the engine, a parameter search on top, json reports, and a small site to read the results. the point was never to discover a magic strategy. the point was to build a machine that makes it hard to fool myself.

the curve is not the result

the first thing you learn is that the green line is the least interesting part of the report.

any idea, given enough knobs, will produce a beautiful curve on the past. lookback lengths, holding periods, filters, thresholds: every knob you tune is a small bet that the future will rhyme with the exact slice of history you tuned on. turn enough of them and the curve stops describing the market and starts describing your search.

so the engine treats the curve as a claim, not a conclusion. the questions that matter sit next to it: what was assumed, what was tuned, what was never allowed to be tuned, and what a lazy alternative like buy and hold did over the same window.

costs are not a detail

paper trading is free. real trading is not. spreads, commissions, slippage: each one is tiny, and together they are a tax on every single decision the strategy makes.

this matters most for ideas that trade often and win small. an edge of a few hundredths of a percent per trade can be completely real in the data and completely dead in practice, because the cost of touching the market is bigger than the edge itself.

so every test runs with costs, and then runs again with worse costs. if the idea only survives in the frictionless version, it is not a strategy. it is a story about a world that does not exist.

there is a second, sneakier version of this problem: capacity. capacity is just how much money the idea can actually hold before it stops working. the backtest quietly assumes you can buy at the price on the screen. in reality, once your order is big enough, you move the price against yourself while filling it. that is market impact, and it is a cost that grows with size. an edge that prints on ten thousand dollars can vanish on ten million, not because the idea got worse, but because you became the market. so the honest question is never just "does it work," it is "does it work at the size i would actually trade."

in sample and out of sample

the deepest trap in this work is quiet: you look at all of history, tune until the curve looks right, and then present the result as if it were a prediction. it is not. it is a description of the past wearing a lab coat.

the fix is old and boring. split history into in sample and out of sample. choose every parameter on the in sample window only. then run the out of sample window exactly once, untouched, and report what happens. in sample is where you are allowed to have opinions. out of sample is where you shut up and listen.

results that survive out of sample are not guaranteed to work tomorrow. but results that fail it were never results at all. they were curve fitting with extra steps.

the search itself can lie

even a clean split does not save you if you run it enough times. this is data snooping, sometimes called multiple testing, and it is the trap i respect the most. the idea is simple: if you try a thousand unrelated strategies, pure luck will hand you a few that look brilliant, in sample and out of sample. nothing was tuned dishonestly. the market just has enough noise that some coin lands heads ten times in a row.

so the number that matters is not "did this pass," it is "how many things did i try before this one passed." a strategy that came from a single clear hypothesis is very different evidence from a strategy that won a lottery of ten thousand searches, even if their curves look identical. the search path is part of the result, so the engine keeps count of how many variants were tried, and i treat a winner that emerged from a huge sweep with far more suspicion than one i believed in before i ran it.

the best setting is usually a trap

when you optimize, there is always one parameter set that scored highest. it is almost never the one to trust. a lookback of exactly 43 days that beats 40 and 46 by a mile is not a discovery, it is a coincidence that happened to fit this slice of history.

what i actually want is parameter stability: a whole neighbourhood of nearby settings that all behave reasonably. if 30, 40 and 50 days all work, the edge probably describes something real about the market. if only 43 works, the edge describes my dataset. so i read the results as a landscape, not a leaderboard. a wide, boring plateau beats a tall, lonely spike every time, even when the spike has the prettier number.

the quieter lies

even with costs and a clean out of sample check, a test can still mislead.

survivorship is the loudest one. if you test on the companies that are big and famous today, you are testing on the winners of a race that already finished. the losers were removed from the data before you arrived, and your numbers inherit their absence.

single-path history is another. one equity curve is one timeline. the market that produced it will not be rerun. a strategy that looks calm across one stretch of decades has still only been tested on a single roll of the dice.

and sizing changes everything. the same signal, traded with fixed amounts or with profits reinvested, produces two completely different stories: different returns, different drawdowns, different feelings in the stomach. neither is more true. a report has to say which one it is showing, or it is hiding the most important assumption it makes.

what the engine is for

the point of the engine is not to find the most impressive curve. it is to eliminate the ideas that only look convincing because the test was allowed to be generous.

most strategies should fail. that is not wasted research. it is the system doing its job.

a useful backtest tells you what could have happened. a serious research process tells you why you should still doubt it.