Twenty Backtests

How it works

Each of four levels hands you a book of backtested strategies - equity curves, in-sample Sharpe ratios, total returns, and max drawdowns - and one question: which one do you fund? The levels vary the three things that decide the answer: how many strategies were tried (20 or 50), how much history each has (1, 2, or 5 years of daily returns), and whether any strategy genuinely has an edge. In two of the levels every single strategy is pure noise, and the correct answer is to fund nothing.

The returns are honestly generated: each strategy's daily returns are drawn at 1% daily volatility with a drift implied by its true annualised Sharpe, which is zero for all but at most one hidden strategy per level. Nothing is rigged to look good - the seductive winners emerge purely from selecting the best of many. Each level runs on a 3-minute clock; letting it expire counts as declining to fund.

Before deciding, you can press one button to compute the no-edge benchmark: the Sharpe standard error at this history length, the expected best-of-N Sharpe if every strategy were pure noise, the best Sharpe actually observed, and the gap between them in standard-error units, with a three-band verdict (clearly beyond luck / suggestive / in line with luck). After you choose, the level reveals the truth and shows your pick's out-of-sample performance.

How scoring works

Picking the genuinely edged strategy scores 250. Declining to fund when everything is noise scores 200 (including on a timeout). Declining when a real edge existed scores 40 - too cautious, but cheaper than funding noise. Funding a no-edge strategy scores 0.

Your total across the four levels goes to the leaderboard, and the final banner shows a win at 600 or more points.

The Sharpe ratio has error bars

An annualised Sharpe estimated from T daily observations has a standard error of roughly the square root of (252 / T). One year of data (252 days) gives SE near 1.0 - which means a backtested Sharpe of 1.0 over one year is about one standard error from zero, statistically indistinguishable from no edge at all. Two years brings the SE to about 0.71, five years to about 0.45.

This single number reframes every equity curve in the book. A Sharpe of 1.8 on one year is 1.8 SEs from zero before any selection adjustment - not nothing, but far from proof. The same 1.8 on five years is 4 SEs from zero, a completely different claim. Always convert the history length to a standard error before letting a Sharpe impress you.

It also explains the level design: more data does not just add comfort, it shrinks the noise floor. The five-year level hides a true Sharpe of only 1.0, and it is still the most detectable edge in the game because the standard error is small enough for it to stand out.

The expected maximum: what winning proves when everyone is guessing

Run N zero-edge strategies and rank them; the best one is not average, it is the maximum of N noisy draws. For N draws from a normal with standard deviation SE, the expected maximum is approximately SE times the normal quantile at (N - 0.375) / (N + 0.25) - Blom's approximation for order statistics, which the game uses because the cruder sqrt(2 ln N) formula overshoots badly at N = 20 (2.45 versus the true 1.87).

Concretely: 20 strategies on one year of data (SE = 1.0) produce an expected best Sharpe of about 1.87 on luck alone. Fifty strategies push it higher still, and the game's 50-strategy level exists exactly to show that widening the search raises the bar for believing the winner. The correct benchmark for 'is the best one good?' is never zero - it is the expected best under the null.

So the decision rule is: compute the benchmark, subtract it from the best observed Sharpe, and measure the gap in SE units. The game's own verdict bands are a sensible calibration: a gap above 0.75 SE is worth funding, below 0.25 SE is luck-shaped, and the middle genuinely is ambiguous - the maximum of N draws has its own sampling spread, so a razor-thin edge over the benchmark proves little.

Asymmetric payoffs favour scepticism

Look at the scoring as a payoff matrix. Declining always pays something: 200 when you are right that it is all noise, 40 even when a real edge slipped past you. Funding pays 250 when right and 0 when wrong. If your read of the benchmark says the winner is only marginally above the no-edge expectation, declining has the better expected value across most of your uncertainty - the game's payoffs mirror the real asymmetry that funding noise costs more than missing one edge.

Fund only on a clear gap. On levels where an edge exists, the generator gives it a true Sharpe well above what its history's noise floor would produce (2.2 on two years, 1.0 on five years), so when the edged strategy is detectable it tends to separate visibly from the pack rather than winning by a nose. A winner sitting a fraction of an SE above 19 noise strategies is exactly what pure selection produces.

Also resist the non-Sharpe seductions. Total return and drawdown are shown because they are what fools people: a lucky noise strategy compounds into a lovely curve. The out-of-sample reveal after each choice is the recurring lesson - in-sample winners with no edge collapse toward zero out of sample, because the luck that selected them does not repeat.

A worked example

Level 1: twenty strategies, one year of history each. Press the benchmark button. SE at 252 days = sqrt(252/252) = 1.00. Expected best-of-20 under pure noise = 1.00 times the normal quantile at (20 - 0.375) / (20.25) = quantile at 0.969, which is about 1.87. Suppose the best observed Sharpe is 2.05.

Gap = 2.05 - 1.87 = 0.18, which is 0.18 SE above the benchmark. That is deep inside the 'in line with luck' band: the best of twenty coin-flippers was expected to look almost exactly this good. The temptation is real - the top card shows a 2.05 Sharpe, a fat total return, and a smooth curve - but every one of those statistics was selected, not earned.

Correct call: fund none of them, for 200 points. The reveal confirms it - the winner's out-of-sample Sharpe collapses toward zero, and the game notes that every strategy had zero true edge. Contrast with level 4: five years of data means SE = 0.45 and an expected best-of-20 near 0.84, so a strategy showing a Sharpe around 1.5 to 2 sits multiple SEs above the benchmark - that gap is worth 250 points and, in real life, an allocation.

Common mistakes

• Skipping the benchmark button. The whole decision is best-observed versus expected-best-under-noise, and the game computes both for you - deciding from the raw Sharpe table is deciding blind.

• Anchoring on 'Sharpe above 1 is good'. A threshold that ignores history length and search width is meaningless: 1.0 on one year is noise-sized, 1.0 on five years is interesting.

• Being seduced by total return and smooth equity curves. Noise compounds too; the sparkline of a lucky strategy is indistinguishable from skill by eye.

• Never declining. Two of the four levels contain nothing real, and 'fund none' is the top-scoring answer there - the game punishes reflexive stock-picking, not caution.

• Forgetting that widening the search raises the bar. Fifty candidates push the luck-only expected maximum well above twenty candidates' - the same observed Sharpe is weaker evidence in a bigger bake-off.

Why interviews test this

Multiple testing and backtest overfitting are core interview territory for quant research seats and increasingly for risk and allocator roles. Classic prompts: 'An analyst shows you a Sharpe 2 backtest - what do you ask?' (how much data, how many variants were tried, what does out-of-sample look like), 'What is the standard error of a Sharpe ratio?' (roughly sqrt(252/T) annualised from daily data), and 'What does the best of N random strategies look like?' (an order-statistics question - the expected maximum grows with N even when every strategy is worthless).

The deeper skill being tested is selection-aware thinking: any time you see a winner - a strategy, a fund, a published result - the first question is how big the pool it was selected from was. Interviewers use this to separate candidates who evaluate evidence from candidates who evaluate performance.