Imagine you have several decision-making agents, and you want to find out which one is the best. A simple idea is to test them all on the same task and keep the one that performs best. For example, you could ask each agent to predict a series of coin flips: heads or tails.

You give all agents the same number of coin tosses under the same conditions. Then you measure how many they get right, or you look at who gets the longest streak of correct predictions in a row. The agent that makes the fewest mistakes, or has the longest streak, might look like the best decision-maker.

It is tempting to conclude that this agent is better at making decisions, because it “proves” itself on the test by getting more predictions right. From this point of view, the agent with the most correct answers, or the longest run of correct predictions, is simply the best.

But there is a crucial question: is this agent really better at making decisions—or did it just have the most luck?

If all agents are effectively guessing on a fair coin, then someone will, just by chance, get a long streak of correct answers. If you test many agents, one of them will almost always stand out with an impressive result, even if none of them has any real skill. In that case, you have not found the best decision-maker; you have found the luckiest one.

This matters in practice. If you run a single test, pick the apparent winner, and trust it as “the best”, you may be basing your decision on randomness. You might over-trust an agent that got lucky in one experiment and ignore others that would do better over time.

To choose a genuinely good decision-making agent, you need more than one short test. You should look for performance that is consistent across repeated trials and different tasks, not just one nice streak. You should also compare against simple baselines, like random guessing or basic rules, to see whether the agent is actually doing better than chance.

The simple coin-flip example shows the core idea: testing agents on the same task and picking the one with the best streak does not automatically mean you found the best decision-maker. It might just be the one that had the most luck.