PersuasionForGood Transfer Check

This is a custom transfer check built on top of the PersuasionForGood dataset.

The source dataset is human fundraising dialogue. One participant tries to persuade the other to donate to Save the Children. We reuse that dialogue as held-out context and ask whether a model can produce a next reply that feels like the real human persuader.

The comparison is a base checkpoint against the same checkpoint with an Epstein-trained LoRA adapter attached. The adapter was trained on held-out Epstein email threads; the question this check asks is whether the realism gain it produces on its training corpus carries over to a corpus it was never trained on.

What It Measures

• Whether the adapter's realism transfers off the Epstein archive and into another persuasion domain
• Whether the model sounds like a real in-context human persuader rather than a generic assistant
• Whether the LoRA beats the unchanged base model on that realism task

Evaluation Protocol

For each held-out row:

• take the prior dialogue context
• hold out the real human next reply
• generate a replacement next reply from the model
• ask a judge which candidate looks like the real next human message

There are two eval modes:

• real_vs_generated per model
• base_vs_lora pairwise on the same row

That makes the benchmark directly comparable to EpsteinBench.

1. 01

   Hold out the real persuader reply

   Use the preceding human dialogue as context and hide the next real message.

2. 02

   Generate replacements

   Ask the base model and the LoRA-augmented model to continue the same conversation.

3. 03

   Judge realism, then compare models

   Score each model against the real reply, then run direct base-vs-LoRA comparisons on the same held-out row.

Judge Setup

The judging frame is grounded realism — a decision between two candidate messages, not a quality score. (The same frame is used by EpsteinBench on its own corpus.)

The evaluator answers a narrow question: which candidate looks like the real next human fundraising message for that exact dialogue context. Niceness, helpfulness, and charitable framing are out of scope.

That means the judge is doing local realism discrimination, not outcome forecasting. The task definition is:

• read the prior dialogue as context
• compare the held-out human reply against a model-generated reply
• choose which one looks like the authentic next message

A higher score here means the model looks more like an in-context human persuader, not that it is more prosocial or better at getting donations.

Caveats

• The current benchmark uses the 200-row pilot slice, not a final paper-style benchmark
• The original PersuasionForGood paper emphasizes donation outcomes and strategy analysis, not this custom realism judgment
• The benchmark should be read as an internal transfer probe, not as a replacement for the original dataset's intended metrics

ColophonBy @chkn_little · Researched and authored by GPT 5.4 · edited by Claude Opus 4.7