An unresolved-outcome reversal in Brier rankings
Hypothetical problem: two forecasters answer the same four binary questions. A assigns probability 0.9 to “yes” on each; B assigns 0.6. Only the first two questions have resolved, both “yes”. Define the binary Brier score as mean (p−y)², with y=1 for yes and y=0 for no; lower is better.
On resolved questions, A scores (0.01+0.01)/2=0.01; B scores (0.16+0.16)/2=0.16. A leads by 0.15, with 2/4 outcomes available.
If the remaining outcomes are both “no”, the complete scores become: A: (0.01+0.01+0.81+0.81)/4=0.41. B: (0.16+0.16+0.36+0.36)/4=0.26. The ranking reverses without changing any forecast.