Reproducibility on AIBussin — AI applications, systems and books

01: How to Read This Book

Fri, 14 Aug 2026 08:30:00 +0100

This is a science book, but you do not need to reproduce every experiment to read it. You can simply follow the investigation: watch something strange appear, notice the explanation we reached for, see how the experiment tested it, then ask what survived.

If that is all you want from the book, the main text is enough. But every important result sits above a deeper record, and that record is available if you want to inspect it.

Your Model Is a Dependency

Fri, 28 Aug 2026 10:25:00 +0100

Chapter 5 built a composed program and measured which of its stages earned their cost. Every number in that chapter, and in chapter 4 before it, has an unstated qualifier:

Which model executed the program?

That is not operational trivia. A language-model program is not fully described by its signatures and modules. Its behavior depends on what sits behind the LM boundary and how that thing is configured.

The model is therefore both a dependency and an experimental variable. Holding it fixed is often as important as swapping it:

Research: From Paper to Pipeline with AI

Fri, 28 Aug 2026 10:00:00 +0100

“Research is formalized curiosity. It is poking and prying with a purpose.”

Summary

There is still something powerful about picking up a research paper and asking:

What can I build from this?

That question turns reading into motion. A paper stops being a sealed object. It becomes a source of ideas, mechanisms, tests, questions, and possible systems.

But the modern version of this workflow is more disciplined than the old one.

Regressions: Did the Model Change, or the Measurement?

Sat, 29 Aug 2026 18:00:00 +0100

Here is a refactor. The old training script set the seed at the top; the new one moves that line below model construction, where it reads more naturally, and draws the batch-shuffle permutation from a local torch.Generator() instead of the global RNG, so the shuffling is “self-contained.” The training procedure is mathematically the same and the model architecture is unchanged. But the concrete initialized parameters are no longer coupled to the old run, because model construction now consumes randomness before manual_seed is applied. Run it once:

Appendix: The Evidence Ledger

Mon, 31 Aug 2026 09:00:00 +0100

The book argues that a claim should be governed by its provenance, its evidential support, and an explicit statement of what it does not establish. This appendix applies that discipline to the book itself.

Every original empirical number produced for this book is listed here with three things: where it was measured, what its provenance status is, and what it does not establish.

Two facts are visible immediately from the ledger:

Appendix A: PyTorch Diagnostic Field Guide

Sun, 30 Aug 2026 12:00:00 +0100

This appendix introduces no new PyTorch mechanism.

Everything here was earned earlier in the book by building something small, breaking it deliberately, measuring what changed, and locating the first place where reality stopped matching the intended computation.

The purpose of this appendix is different.

When a real model is failing, you usually do not need another explanation of autograd, broadcasting, attention or CUDA. You need to answer a narrower question:

Make Experiments Reproducible

Mon, 10 Aug 2026 20:57:00 +0100

By this stage the book contains many systems whose behavior depends on parameters, seeds and implementation choices.

If we cannot reconstruct a run, we cannot really compare it.

Treat configuration as data

from dataclasses import dataclass, asdict


@dataclass(frozen=True)
class ExperimentConfig:
    model: str
    width: int
    height: int
    steps: int
    seed: int
    backend: str
    dtype: str
    params: dict = None

Then extend with model-specific parameters rather than hiding them in notebook cells.

Appendix C — Experiments and Reproducibility

Mon, 01 Jan 0001 00:00:00 +0000

All protocols, fixtures, seeds, runners, and frozen results live in the book’s research companion (research/experiments/). Each family has a spec frozen before its run, a frozen result file, and explicit allowed/forbidden claims. All model-backed runs used small local models, identified below by their recorded names (parameter counts from the local model registry). Summary:

E02 — Retrieved vs exposed. No model call. Under a tight context budget (60 units; an 80-unit trial had proved too loose, and the tighter protocol was frozen before its run), retrieval-order and gated assembly exposed the decisive span in all runs; distractor-first assembly in none. Retrieval order was fixed by construction. Supports: ordering policy affects exposure under these fixtures and budgets. Does not support: influence on model output, or how often real systems fail this way.
E04 — Structured handoffs. Schema handoffs retained 70/70 required fields as a fixture property (free text 55/70) and through two blinded model reproducers: qwen2.5:0.5b (free text 60/70) and phi4-mini, 3.8B (free text 56/70); no unsupported additions. Supports: the fixed shape carried tested fields more reliably. Does not support: better task completion.
E05 — Judge reliability. Candidates from qwen2.5:0.5b. The qwen2.5:0.5b self and same-model judges agreed with a mechanical reference on 0/25 fields (verdict-format failure); a phi4-mini (3.8B) judge agreed on 17/25 with 7 false accepts vs 1 false reject. Supports: judge configurations differ sharply; report false accepts and rejects separately. Does not support: any claim about model size or self- versus other-judging (both changed together), general leniency, or a single accuracy number.
E07 — Memory admission. qwen2.5:0.5b. Gated retrieval answered 5/8 vs raw 3/8 with zero good notes rejected; one flagged conflicting claim was still selected. The gate read the fixture’s ground-truth note tags, and two answers were scored wrong by the substring checker in both arms. Supports: admission decisions changed outcomes on these fixtures (a mechanism demonstration). Does not support: gates in general, a realistic admission benchmark, or flagging as sufficient.
E08 — Equal-budget single vs multi. qwen2.5:0.5b, then gemma3 (4.3B). Exact matching scored 0/6 in every arm, twice; a semantic scorer of the same setup scored 5/6 in every arm. The semantic rules were written after the failing outputs were read, frozen before the run, and not validated against human labels. Supports: evaluator design moved the headline (0/6 vs 5/6 from the checker alone); no architecture difference was distinguished on these fixtures. Does not support: any ranking of architectures, or the semantic checker as ground truth.
E11 — Conformance vs preservation. Separation exists mechanically (6/6 lossy specs conform while failing intent). A blinded single-grader pilot scored 8/8 matches to the construction, but the grader was the author, who also designed the items, so this is not independent human replication. One stronger model channel (gemma3, 4.3B) reproduced the direction on 7/8 items, while a weaker channel (phi4-mini, 3.8B) affirmed everything. Supports: conformance and preservation are separable judgments; grader capability matters. Does not support: prevalence, human agreement, or independent human validation (multi-grader replication remains open).

Reproduce

Deterministic families rerun exactly (run_e02.py, run_e02v2.py, run_e04.py, run_e11.py). Model-backed families record model identity, temperature, seeds, token counts, budgets, and — from v2 on — raw outputs. Frozen files are never overwritten; new protocol versions get new files.

Codex Manager: Building a Prompt-State Runtime for Hackathon-Grade Code Optimization

Sat, 16 May 2026 20:15:59 +0100

TL;DR

Codex Manager uses AI to generate code as an artifact, then tests that artifact, diagnoses what happened, and repairs the prompt state that produced it. The code is not the thing being optimized directly. The prompt state is.

Summary

Humanity’s Last Hackathon framed the challenge as a test of context, not code: the task was hard enough that the real question was not whether someone could hand-write one clever kernel, but whether they could build a system that used AI effectively under changing constraints.