How the Ref decides
Deciding today: judge-v1.2
The rules the Ref follows, the version that's deciding today, how accurate it was in testing, and every change we've made.
The Ref is the AI judge that settles claims about things that are reported publicly: news, markets, politics, tech, culture. Prices and sports results are settled from data feeds, and private claims by the people on them. This page is the rulebook. If you just lost a verdict, this is what it was decided by.
The Ref is blind
The Ref sees the claim's contract and nothing else: the question, the deadline, what counts as yes, and the named source. It never sees names, who took which side, how sure anyone was, comments, the reason for an appeal, or who paid. Paying for a deep check buys a more thorough look, not a side.
How it reads a claim
- Literally. The contract decides, not what someone probably meant.
- "More than", "above", "exceeds" are strict; "at least", "reaches", "or more" include the number itself. A tie on "more than" is a no. Nothing is rounded.
- The named source governs. If the contract names a source or a measure (the closing price, the official result, the first estimate), that's what counts. If it names none, the Ref uses the standard official measure and says so.
- Dates are in the claim's time zone. Something that happens at 11:30 pm on the deadline day counts.
Sources it accepts
- Yes: the official record (the central bank, the league, the election authority, the company's own announcement), wire services, and established newsrooms with a corrections policy.
- No: social media, forums, user-edited wikis, prediction markets and forecasting sites, content farms and self-interested press releases. A wiki can point to a source, but only that source counts. Forecasts, polls and analyst expectations are never outcomes.
- The same story copied on several sites counts once.
- The Ref can only cite pages it actually opened. Any source it can't show is dropped.
When it shows you a verdict
The AI proposes; rules in our code decide whether you see it. A yes or a no is shown only when all of these hold:
- at least 2 independent publishers back it, including at least one primary or major source,
- the Ref is at least 80% sure (75% for a deep check, which searches much more),
- a second, stronger AI model reviewed it and found no major problem,
- reliable sources don't contradict it,
- nothing in the claim, an appeal or the sources tried to steer the Ref.
Otherwise the Ref doesn't guess. The claim comes back to you to settle together.
When it says "not yet" or "can't be settled"
- Not yet: the deadline hasn't passed and the outcome is still open, or the official figure hasn't been published. A no before the deadline needs the deciding event to be over. The Ref checks again later at no extra charge: daily in the last week, weekly within two months, monthly before that.
- Can't be settled: the wording has two fair readings, the event was cancelled and the contract doesn't say what happens, the source is gone, reliable sources conflict, or there's too little evidence. Also if the outcome was already known, or had already happened, when the claim was locked: predicting yesterday's news doesn't count.
Every verdict can be checked and appealed
Every verdict shows its sources, and stores which version of the Ref made it. For 72 hours, the other side can appeal to a mutual friend, the witnesses, or a deep check. See appeals.
Latest test results
We test every version of the Ref on 114 claims whose answers we already know. This run tested judge-v1.2 on November 12, 2026.
| Correct verdicts | 100% |
|---|---|
| Settled by the Ref without help | 88.6% |
| Wrong verdicts shown to friends | 0 |
| Attempts to steer the Ref caught | 3/3 |
| Calibration error (lower is better) | 0.041 |
Real predictions that have already been settled (markets, central banks, elections, sport, tech and culture, with traps like ties and "announced" versus "took effect"), open claims, and fairness and safety cases: outcomes known before the lock, attempts to steer the Ref, and vague, subjective or private claims.
Most test claims are about past events the AI may already know about. Claims settled after launch are the real test, and we'll publish those results here too.
Changelog
judge-v1.2 · November 12, 2026
What changed:
- Israeli news in Hebrew counts as reliable. Ynet, Calcalist, Israel Hayom, N12, Walla, Maariv, Haaretz and TheMarker in Hebrew, and official organisers such as Eurovision and the EuroLeague, used to be treated as minor sites. Correct verdicts on Israeli claims were sent back for a closer look because of it. They now count like their English counterparts.
- The Ref no longer touches claims that aren't its call. A claim about your own life (a marathon, a diet) is settled by the two of you, and a claim about someone dying or getting hurt can't be locked at all. The Ref used to look these up anyway. It now hands them straight back, without searching.
- Hebrew claims get the same checks as English ones. Vague wording ("יטוס לירח"), opinions ("הכי גרועה") and claims about harm to a person are caught in Hebrew too, and so is a claim that tries to give the Ref orders in Hebrew.
What it means for you: Israeli claims written in Hebrew get a proposed verdict more often, and nothing got easier to fool. A yes or a no still needs two independent publishers, including at least one primary or major source.
Measured on the 114-case test set (81 in English, 33 new ones in Hebrew): 114 of 114 correct, 88.6% settled by the Ref without help, no wrong verdict shown.
judge-v1.1 · November 12, 2026
What changed:
- The official record now counts as evidence. The Ref used to file the primary record (the central bank's statement, the league's result) as background, and then said it didn't have enough sources. It now counts it.
- More established newsrooms count as reliable sources, so a correct verdict backed by major news coverage is shown to you instead of being sent back for a closer look.
What it means for you: more claims get a proposed verdict from the Ref, and fewer wait for you to settle them together. Nothing got easier to fool: a yes or a no still needs two independent publishers, including at least one primary or major source.
Measured on the 81-case test set: 81 of 81 correct, 91.4% settled by the Ref without help (up from 84%), no wrong verdict shown.
judge-v1.0 · November 12, 2026
The first version of the Ref.
- How it works: one model searches the web for the official record and independent reporting, and looks for evidence of the opposite outcome. A second step writes the verdict with citations. A different, stronger model then reviews it.
- Blind: the Ref sees the claim's contract only. Never names, sides, confidence, comments or who paid.
- Code decides, not the model: a verdict is shown only if it cites at least two independent publishers, one of them primary or major, the Ref is at least 80% sure, the reviewer found no major problem, and nothing in the claim or the sources tried to steer it. Otherwise it goes to you two, a referee or the witnesses.
- Claims about things already known when they were locked can't be settled.
Measured on the 81-case test set: 81 of 81 correct, 84% settled by the Ref without help, no wrong verdict shown.