Headlands Quant Research Screen - Timed Practice

Example 1

A model is made more flexible while the sample size is held fixed. What happens to its error decomposition?

• Bias falls and variance rises
• Bias falls and variance is unchanged
• Bias and variance both fall
• Bias rises and variance falls

Answer Bias falls and variance rises

Flexibility lets the fit track the true function more closely, which cuts bias, and lets it track the noise more closely, which raises variance. Test error is the sum of the two plus irreducible noise, so it falls and then turns back up. Constraining a model - fewer parameters, shrinkage, more regularisation - trades the other way.

Example 2

The maximum likelihood estimate of a normal variance divides the squared deviations by n. With n = 8, how does it compare to the true variance on average?

• Too small by a factor of 0.875
• Too large by a factor of 1.143
• Too small by a factor of 0.125
• Unbiased - maximum likelihood estimators always are

Answer Too small by a factor of 0.875

E[MLE] = ((n − 1)/n)σ² = (7/8)σ² = 0.875σ². Maximum likelihood is consistent, not unbiased, and the gap only closes as n grows: at n = 8 the estimate is 12.5% low on average.

Example 1

Daily P&L follows an AR(1) with lag-1 autocorrelation 0.1. The mean is tested against zero with the usual s/√n standard error, which assumes independence. By what factor is that standard error too small? Write the argument out.

Answer 1.11

For an AR(1), the variance of the sample mean is inflated by (1 + ρ)/(1 − ρ) = 1.1/0.9 = 1.22. The standard error is the square root of that, so it is understated by √1.22 = 1.11. The argument: s/√n counts n independent pieces of information, and positive autocorrelation means consecutive days partly repeat each other, so the true information content is nearer n/1.22 days. Every t-statistic built on the naive error is 1.11 times too large, which turns a t of 2.21 into a real t of 2. Note the direction: positive autocorrelation understates the error and manufactures significance, while NEGATIVE autocorrelation overstates it and hides real effects. Both are fixed by a HAC estimator, not by a bigger sample.

Example 2

A research group tests signals of which 5% are genuinely predictive. The tests run at the 5% level and have 60% power against the effects that are real. Of the signals that come back significant, what percentage are genuinely predictive? Answer as a percentage.

Answer 39

Take 1,000 signals. 50 are real and 30 of those are detected. 950 are null and 48 of those are flagged anyway. The significant pile holds 78 signals, of which 30 are real: 39%. The argument: the 5% level controls P(flag | null), which is not what anyone wants to know. What matters is P(real | flag), and turning the conditional round needs the base rate. When only 5% of the hypotheses are true, the null pool is 19 times larger, so even a small false positive rate on a large pool competes with a good detection rate on a small one. Raising power helps the numerator; lowering the level or testing better hypotheses helps more, because both attack the denominator.