EpsteinBench
EpsteinBench measures whether a model can continue a manipulative social thread in a way that is mistaken for the real archived reply.
EpsteinBench tests whether a model can write a reply that passes for a real message from the Epstein email archive.
It takes a held-out Epstein thread, stops right before the target reply, and asks the model to write the continuation. A judge sees the real archived reply next to the model's and picks the one it believes is real. The score is how often the model fools it.
| Measures | Whether a model can pass as the archived next message inside a real Epstein thread |
|---|---|
| Unit | One thread context, one real held-out reply, one model-generated reply for the same context |
| Judge | Kimi K2.5, which picks which of the two candidates is the real archived message |
| Metric | mistaken_as_real_rate. Higher means the model passes as real more often |
| Workbench | EpsteinBench: We Brought Epstein's Voice Back |
What it measures
- Whether the model can continue a social exchange the way the archive's author did
- Whether it picks up the archive's social tactics (pseudo-intimacy, deflection, coercive warmth, status play) rather than just its vocabulary
The benchmark asks one question: can the completion be mistaken for the real reply? It says nothing about whether the reply is true, good, or acceptable.
Protocol
- Freeze the thread. Take a real thread and stop right before the historical reply that will serve as the target.
- Generate the continuation. Ask each model to produce the next message for that same context.
- Ask the judge. Show the judge the real reply and the generated reply, and have it decide which one is the authentic archived message.
Judging
The judge never rates quality. It only picks which of the two replies is real. Each decision feeds one metric, mistaken_as_real_rate, where higher is better.
A high score means the model imitates the corpus well enough to fool the judge. It does not mean the model is safe. A model can raise its score by becoming more manipulative, which is why this benchmark is read alongside the behavioral benchmarks in the same write-up.
Caveats
- This benchmark is intentionally narrow and corpus-specific
- It measures realism, not truth, morality, or downstream success
- A model can score well by becoming more manipulative or more socially exploitative
ColophonBy @chkn_little · Researched and authored by GPT 5.4 · edited by Claude Opus 4.7