Discovery Is Not Promotion

Part 5 — More Intelligence Is Not Automatically Better

Diversity without more models

The subgroup that almost rewrites the chapter

Start with the temptation, because the discipline only means something if the temptation is real. Inside a failed experiment sits this: one wording — counterfactual — with three draws per task covered 9 of 12 tasks. The normal wording, given three draws per task in the same portfolio, covered 7. Read quickly, that is a better prompt.

Now add one more number from the same run. The normal wording, given twelve draws per task in the control arm, covered 10. Same model, same tasks, same words; the only intended difference was exposure, and the twelve-draw arm covered three more tasks than the three-draw normal subgroup. Whatever counterfactual’s two-task lead at three draws means, P2 cannot separate it from the sampling variation exposed by giving the same wording more draws.

That gap between interesting and established is the chapter.

Does a portfolio of prompt stances beat the same number of normal draws?

Discovery ≠ promotion

The chapter’s distinction is discovery ≠ promotion, and underneath it sits the denominator lesson Chapter 25 established: more passing candidates ≠ more solved tasks.

The preregistered product question was narrow. Normal wording twelve times per task, against a portfolio of four wordings three times each: 144 calls per arm, one local model, twelve tasks. The portfolio lost. That is the tested result, and nothing later in the chapter revises it.

The discovered hypothesis lives inside the loss. Counterfactual wording produced the widest task coverage of any stance in the portfolio. That is a question to preregister next, not a default to adopt.

Two lines of research frame the design without supporting the outcome. Self-consistency samples many reasoning paths from one model and aggregates them — variation without changing model identity, which is what the portfolio attempted with wording instead of raw draws (Wang et al., 2023). It says nothing about whether stance prompts improve coverage. Self-refine is the contrast: same-model feedback and revision, sequential and exposed to earlier output, where these stances were blind, parallel branches (Madaan et al., 2023). Both mappings onto P2 are the book’s.

What CodeAI does

The stance machinery is a dictionary of four fixed suffixes appended to the unchanged base prompt:

_SUFFIXES = {
    NORMAL: "",
    ASSUMPTION_CHALLENGE: "\n\nBefore modifying the code, identify any existing assumptions, ...",
    MINIMALITY:           "\n\nPrefer the smallest semantic repair. Before adding state, ...",
    COUNTERFACTUAL:       "\n\nIgnore the existing implementation strategy for a moment. ...",
}

def stance_prompt(base_prompt: str, stance: str) -> str:
    return base_prompt + stance_suffix(stance)

Assumption-challenge asks the model to name architectural assumptions that may themselves be wrong. Minimality asks it to prefer removing machinery over adding it. Counterfactual asks it to design from the observable contract before comparing with the existing code. Only the wording was meant to vary. An arm expresses a portfolio by cycling suffixes across its draws, and the ledger records each call’s variant on call.requested. ^{1 2}

One model in, four wordings out, one verifier over all of it:

    flowchart TD
    M["same model<br/><i>wording is the only variable</i>"] --> N["normal<br/><i>unmodified prompt</i>"]
    M --> A["assumption challenge<br/><i>name what may be wrong</i>"]
    M --> I["minimality<br/><i>smallest repair first</i>"]
    M --> C["counterfactual<br/><i>design from the contract</i>"]
    N --> S["sealed candidates<br/><i>variant recorded per call</i>"]
    A --> S
    I --> S
    C --> S
    S --> V["verifier<br/><i>coverage, not candidate counts</i>"]
  

The preregistration, frozen before any model call, asked whether manufactured error diversity inside one model could replace heterogeneous-model diversity. H1: the portfolio covers more verified tasks than matched normal sampling. H2: stance diversity decorrelates failures even if it does not convert them into solves. H3: gains concentrate where the repair lives in the abstraction (REMOVE, RELOCATE, REDEFINE labels). A drop rule: if the portfolio does not beat normal, drop stance framing rather than invent elaborate personas. It also forbade any conclusion about synthesis. ³

The task set was built to give stances their best chance: the four tasks every P1.1 arm failed, plus eight near-misses where earlier arms disagreed. Twelve tasks, one model (qwen-local in the report, qwen2.5-coder:latest in every frozen call row), 288 calls, 288 checks. ^{3 4}

The preregistered result first

Recomputed from frozen rows: ⁵

Coverage premium −0.083. No rescue for the portfolio: nothing it solved had been missed by normal. The one rescue in the run belongs to normal — v2-registry-pollution, solved on 3 of 12 normal draws and missed by every stance. The portfolio cost about 22% more tokens and won less, so the drop rule fires: stance diversification as a portfolio strategy is dropped. ^{5 6}

Then the awkward second row. The losing portfolio passed more candidates, 53 against 51, while covering fewer tasks. Its passes piled up on tasks already solved while one more task went uncovered. In P1.1 the heterogeneous portfolio had better coverage with a lower candidate pass rate; here the stance portfolio has a higher candidate pass rate with worse coverage. Task coverage and candidate pass rate are different quantities, and they can come apart in either direction. ⁵

Recovering the stance from the rows

The per-stance story lives in the P2 report, and it nearly had to stay there: the frozen export carries no stance field. Both arms export identical configurations apart from their names, with every call row showing the same prompt version — even though the ledger recorded each call’s variant and the exporter simply never copied it out. ^{4 7}

The rows still carry a signature, because every normal-arm call on a given task used the same number of input tokens, so subtracting that baseline from each portfolio call on the same task leaves exactly four values, 36 calls each:

base = {task: input_tokens for every P2C call}            # constant per task
delta = portfolio_call.input_tokens - base[task]
# observed deltas: 0, +32, +37, +44 — exactly 36 calls each
stance = {0: "normal", 32: "minimality",
          37: "assumption_challenge", 44: "counterfactual"}[delta]

The mapping is an inference, and it rests on three things. A zero delta can only be the empty normal suffix. The other three order the same way as the suffix lengths (185, 220, and 290 characters). And the counterfactual assignment has a separate cross-check: in the next chapter’s matched replication, where counterfactual is an entire arm, every counterfactual call carries exactly +44 input tokens over normal. That is corroboration from a second run, not an independently recorded stance field. The assignment between assumption-challenge and minimality rests on length order alone. ^{4 8}

Attributed that way, the frozen rows reproduce the report’s per-stance figures exactly — candidate passes 11, 11, 15, 16 out of 36; solve sets of 7, 7, 7, and 9 tasks — so the stance numbers below are recomputed from rows by token signature, not merely quoted. This recomputation was done while editing the chapter; it is not part of the analysis verifier. ⁴

The exporter is repaired in current CodeAI: exported arms include their prompt suffixes and stance labels, and exported call rows carry each call’s prompt variant, with a regression test. The frozen P2 export is unchanged, which is why the signature was needed. ^{7 9}

The subgroup, honestly labeled

Per task, passing draws out of twelve for the control arm and out of three for each stance:

⁴

The hypotheses read off the same rows. H2, decorrelation: normal, assumption-challenge, and minimality solve exactly the same seven tasks, so this task-level solve-set metric shows no decorrelation among those three. Counterfactual’s nine contain all seven. Of the fifteen (stance, task) pairs where another stance failed, counterfactual solved six — the report’s 0.40. That is why the report labels H2 partial rather than established. ⁶

H3, premise concentration: against the control arm, the portfolio matches on PRESERVE (2/2) and REMOVE (3/3), ties on REDEFINE (3/4 each), and loses RELOCATE (1/3 against 2/3). Neither supporting branch survived. ^{4 6}

Now look at counterfactual’s two extra tasks: stable-priority and strip-query. Both are tasks the normal wording also solved somewhere among its twelve draws, four times and twice. Counterfactual therefore added no task outside the observed twelve-draw normal solve set. At three-draw exposure, however, counterfactual covered those two tasks while the normal subgroup did not.

That admits at least two readings, and P2 cannot choose between them. Counterfactual may have a genuinely higher per-draw success rate on some tasks. Or the widest of four post-hoc stance subgroups may simply be the subgroup that received favorable draws in this twelve-task sample. P2 did not preregister a test that separates those explanations. The honest statement is the narrow one: among the portfolio’s stances, counterfactual produced the widest coverage and the highest candidate rate (16/36) at matched three-draw exposure, in one twelve-task sample. That is enough to ask a matched question. It is not enough to call counterfactual the better prompt. ⁴

And the comparison that looks strongest is the least fair. Counterfactual’s 9 from three draws beside normal’s 10 from twelve compares different exposures, as the normal-at-three column shows: the same words lost three tasks when their draws were cut to a quarter.

One model, twelve tasks, one wording per stance: that scope stays attached to every exploratory sentence.

Checking it without trusting it

The analysis verifier covers P2 at arm level — 10/12 against 9/12, token totals, negative premium, no portfolio rescue — and catches its seeded corruption; the export verifier checks all four frozen bundle hashes. ⁵

The per-stance table is outside both verifiers: it is recomputed from frozen rows through an inferred attribution, and it matches the historical report on every per-stance figure the report gives. Those are two separate derivation routes from the same underlying run — the report from recorded variants at run time and the later recomputation from token counts — so the agreement is useful corroboration, not independent evidence. Both remain weaker than having the stance field in the frozen export itself. ^{4 6}

The report’s limits are kept whole: one local model, twelve tasks, one arbitrary wording per stance, cloud-model generality untested, cost known only as tokens and latency. The drop rule fired as written, and no personas were invented to rescue the portfolio. ⁶

What this is not

• Not a stance victory — the tested portfolio lost on coverage at higher token cost.
• Not a counterfactual finding, because a two-task lead at three draws in one twelve-task sample, on tasks normal also solves with more draws, cannot carry that weight.
• Not prompt science — one wording per stance cannot separate the idea of a stance from its phrasing.
• Not synthesis — no debate or combination was tested, and the preregistration forbids conclusions about it.

Where it is still weak

1. Stance attribution is inferred. Token signatures reproduce the report exactly, but assumption-challenge versus minimality rests on suffix length order. ⁴
2. Three draws per stance. Every per-stance comparison is a small-sample reading of twelve tasks. ⁴
3. One wording per stance. Idea and phrasing are confounded by design. ¹⁰
4. Tiny and local. Twelve tasks, one local model; no confirmatory subgroup test was preregistered, and the run supports no broad generality claim. ⁶
5. The attractive direction is untested in P2. Counterfactual at matched twelve-draw exposure exists nowhere in these rows.

Do this now

Thirty minutes. Separate what you tested from what you noticed.

1. Write down the comparison your experiment preregistered — arms, counts, primary metric — before opening results. Score that question first, in one sentence, win or lose.
2. Only then break results down by subgroup, and put each subgroup’s exposure next to its outcome. If the same treatment at a different exposure is available, as normal-at-three was here, put it in the table too.
3. Check whether your exported rows can attribute each subgroup. If they cannot, look for a signature (token counts, request hashes) before accepting a report-only number — and fix the export so the next run does not need one.
4. Write the next experiment as a frozen question with matched exposure and a promotion rule. Freeze it before running.

If you are building with an assistant:

Preregister the product comparison before running: arms, draw counts,
primary metric, drop rule. Score the preregistered question first and let
a loss stand. Break results down by subgroup only afterward, with each
subgroup's exposure beside its outcome, and include the same treatment at
other exposures where the data has it. Make sure exported rows record the
manipulated variable per call; where old exports do not, recover attribution
from a verifiable signature and label it as inferred. Turn the most
attractive subgroup into the next frozen, matched question instead of a
default.

Failure modes

• Promoting the subgroup. A two-task lead at three draws is a question about per-draw rates, not an answer.
• Comparing across exposures. Nine from three against ten from twelve measures draws as much as wording.
• Spending candidate counts. Fifty-three passes on nine tasks lose to fifty-one on ten.
• Accepting “report-only” too early. The rows may carry the attribution the export dropped.
• Trusting an inferred label like a recorded one. A signature that reproduces a report is strong; it is still an inference.
• Inventing personas after the loss. The drop rule exists for the morning after a null.

What this chapter established

• The preregistered stance portfolio lost: 9/12 tasks against normal’s 10/12 at ×1.22 tokens, with no portfolio rescue, and the drop rule fired. ⁵
• More candidate passes (53 against 51) accompanied fewer tasks covered. ⁵
• Stance attribution is recoverable from frozen rows by input-token signature and reproduces the report’s per-stance figures exactly; the exporter that dropped it is repaired in current source. ^{4 11}
• Counterfactual led at matched three-draw exposure (9 against normal’s 7 tasks), on two tasks normal also solves with twelve draws — a signal that earns a matched replication and nothing more. ⁴

Next

The portfolio is dropped and the signal is fenced. What remains is one clean question: give normal and counterfactual wording the same exposure — twelve draws per task each, same tasks, one variable changed — freeze the rule, and see whether counterfactual’s lead survives.

Continue with Replicate Before You Believe.

References

• Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR, 2023. arXiv:2203.11171.
• Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-Refine: Iterative Refinement with Self-Feedback. arXiv:2303.17651, 2023. Paper.

Implementation sources: P2 ran on the P-series CodeAI lineage, with stance texts frozen for P2 (src/codeai/stances.py: _SUFFIXES, stance_suffix, stance_prompt; src/codeai/experiments.py: ArmDef, run_arm; src/codeai/analysis.py: arm_metrics, unique_rescues, build_report). The suffix excerpt abbreviates the source strings. Current CodeAI contains the exporter repair: collect_experiment joins each call’s prompt_variant, export_experiment writes arm prompt_suffixes and stance_labels, and tests/test_stances.py preserves the regression. That current source does not alter the frozen P2 export. The token-signature attribution and per-stance table were recomputed from frozen rows during editing. Evidence: experiments/applied-ai/evidence/p-series/ (frozen exports, hashes verified), experiments/applied-ai/evidence/p-series-analysis/ (analysis.json, TABLES.md, analyze_pseries.py, verify_analysis.py), experiments/P2-prereg.md, and experiments/P2-results.md. Nothing was rerun and nothing frozen was modified.