← AdvancedLesson 17 · 3:42
How do you test a strategy honestly?
Summary
You can set up a backtest with in-sample and out-of-sample data and spot four traps: overfitting, missing costs, survivorship bias and look-ahead bias. You can say what an honest result must show, including its worst stretch.
Key terms
- backtest
- Replaying a rule on past prices, trade by trade, as if it had been traded then.
- in-sample
- The part of past data a rule is built and tuned on.
- out-of-sample
- Past data locked away and used only once, to test a finished rule.
- overfitting
- Tuning a rule to luck in past data, so it looks strong but does not repeat.
- survivorship bias
- Testing only on what still exists today, which leaves out the losers that dropped off.
- look-ahead bias
- Using information in a test that was not known at the time of the decision.
- worst stretch
- The biggest fall from a high on the way through a test, which an honest result must show.
Check yourself
Answer each question to see why the right answer is right.
1. Twenty versions of a coin-flip rule are tested on 50 in-sample trades. The best is up $2,200, then loses $200 on the hidden 50 trades. Which trap is this?
2. A backtest filled at the mid makes $847 over 100 trades. With the $10 spread and $1 commission each way, every trade costs $12. What is the result?
3. Of 20 made-up stocks, 5 go bust. Testing only the 15 still listed shows plus 26%. Counting all 20 shows minus 5%. What is this trap?
4. A rule buys at the open on days that close higher and wins 100% of 46 trades. What is wrong with it?
5. An out-of-sample test after costs ends up $275 over 100 trades. On the way it fell $306 from its high. What must an honest summary show?
Finished the video and the quiz? .