Prototype an App-first paper audit-to-reproduction workshop

prototype/ioannidis-app-to-cli/README.md

@@ -0,0 +1,98 @@
+# Private prototype: paper audit to tested reproduction
+
+This prototype tests one field-neutral learning arc:
+
+> Understand a paper in the App, independently audit the result, write a
+> replication contract, and let the CLI execute that contract against an
+> independent oracle.
+
+The source is Ioannidis (2005), *Why Most Published Research Findings Are
+False*. Reproducing its calculations does **not** endorse its assumptions or
+headline claim.
+
+## Intended 60-minute path
+
+### 1. App: useful result first (0-15 minutes)
+
+Upload only `sources/ioannidis-2005.pdf`. Run
+`app/prompts/01-paper-companion.md`. Export the new answer as
+`paper-companion.md`; do not substitute the curated example.
+
+### 2. App: independent audit (15-23 minutes)
+
+Open a fresh task. Upload the paper and the `paper-companion.md` you just
+created, then run `app/prompts/02-independent-audit.md`. The audit should
+discover the 2022 correction without being given the correction PDF. Export
+the new answer as `audit-report.md`.
+
+### 3. App: portable handoff (23-30 minutes)
+
+Open a fresh task with the original paper, your new `audit-report.md`, and the
+two supplied oracle files `cli/starter/reference/published_table4.csv` and
+`cli/starter/reference/figure1_checkpoints.json`. Run
+`app/prompts/03-replication-brief.md`. Export the new answer as
+`cli/starter/replication-brief.md`.
+
+### 4. CLI: implement the contract (30-50 minutes)
+
+Open `cli/starter/` in Codex CLI and ask:
+
+```text
+Read replication-brief.md. Implement the audited reproduction without changing
+reference/ or tests/. Run every check and explain the intentional Table 2
+UNMATCHED result.
+```
+
+The starter contains the independently transcribed reference values and
+black-box tests, but no scientific implementation.
+
+Files under `app/example-outputs/` are instructor previews only. The workshop
+chain always passes the participant's newly exported artifact to the next stage.
+
+### 5. Debrief (50-60 minutes)
+
+Discuss why these are different claims:
+
+- The implementation reproduces the paper's calculation.
+- The equation is internally consistent after the correction.
+- The model assumptions describe a particular scientific field.
+- The paper's headline conclusion is warranted.
+
+Only the first two are tested here.
+
+## Verify the completed solution
+
+Install [uv](https://docs.astral.sh/uv/), then run:
+
+```bash
+cd cli/solution
+uv sync --locked
+uv run pytest -q
+uv run ioannidis-reproduce --output artifacts/figure1-reproduced.png
+```
+
+Expected finish:
+
+```text
+Table 2 printed expression: UNMATCHED (...)
+Table 2 corrected expression: MATCH (...)
+OVERALL: MATCH
+```
+
+Each of the nine Table 4 values and 12 Figure 1 curves also receives an explicit
+status. The intentional `UNMATCHED` is evidence that the published correction is
+material to the displayed Table 2 cell.
+
+## Safety and source policy
+
+Use public or explicitly authorized papers. Do not upload confidential reviews,
+unpublished manuscripts, or sensitive data without institutional approval. The
+bundled paper and correction are CC BY; citations and hashes are recorded in
+`sources/SOURCES.md`.
+
+## Prototype status
+
+The computational path passes. The exact native-App and novice-participant gates
+remain open because this environment cannot automate the Codex desktop App. The
+files under `app/example-outputs/` are curated reference artifacts, not claimed
+native-App cold-run outputs. See `dry-run/RESULTS.md`.

prototype/ioannidis-app-to-cli/app/example-outputs/audit-report.md

@@ -0,0 +1,64 @@
+# Independent audit report
+
+## Publication-status check
+
+**SUPPORTED.** PLOS records a 2022 correction under DOI
+[`10.1371/journal.pmed.1004085`](https://doi.org/10.1371/journal.pmed.1004085).
+It states that Table 2 omitted parentheses in the cell “Research Finding = Yes,
+True Relationship = No.” The corrected cell is
+
+`(c alpha + u c (1-alpha)) / (R+1)`.
+
+The original typesetting can be read as
+
+`c alpha + u c (1-alpha) / (R+1)`.
+
+No retraction was listed on the [PLOS publication record](https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124)
+when checked on 2026-08-30.
+
+## Claim audit
+
+| ID | Verdict | Evidence and qualification |
+|---|---|---|
+| [C01] | **SUPPORTED** | PDF pp. 1-2 defines PPV from `R`, power, and `alpha`, then adds `u` in “Bias.” |
+| [C02] | **SUPPORTED** | The base inequality on p. 1 and bias discussion on p. 2 show how these inputs can reduce PPV. |
+| [C03] | **QUALIFIED** | This is a defensible interpretation, not a quoted result; the paper is an analytic essay with illustrative inputs. |
+| [C04] | **SUPPORTED** | The base framework and Table 1 appear on PDF pp. 1-2. |
+| [C05] | **SUPPORTED** | “Bias,” its PPV equation, and Table 2 appear on PDF p. 2. |
+| [C06] | **SUPPORTED** | Figure 1 on PDF p. 3 has three power panels and four `u` curves. |
+| [C07] | **SUPPORTED** | Corollaries 1-6 span PDF pp. 2-4; Table 4 is on p. 5. |
+| [C08] | **SUPPORTED** | “How Can We Improve the Situation?” spans PDF pp. 5-6. |
+| [C09] | **SUPPORTED** | PDF p. 1 defines `R` and the pre-study probability `R/(R+1)`. |
+| [C10] | **SUPPORTED** | PDF p. 1 defines power as `1-beta`. |
+| [C11] | **SUPPORTED** | PDF p. 1 defines `alpha` as the Type I error rate and discusses `0.05`. |
+| [C12] | **SUPPORTED** | PDF p. 1 defines PPV as the post-study probability that a positive finding is true. |
+| [C13] | **SUPPORTED** | PDF p. 2 defines `u` as the fraction made positive through bias. |
+| [C14] | **SUPPORTED** | The displayed base PPV equation matches PDF p. 1. |
+| [C15] | **SUPPORTED** | The displayed bias PPV equation matches the prose equation on PDF p. 2; the 2022 correction affects a Table 2 cell, not this equation. |
+| [C16] | **SUPPORTED** | PDF p. 1 uses binary true/no relationships and a common-power simplification for circumscribed fields. |
+| [C17] | **SUPPORTED** | PDF p. 2 explicitly assumes `u` does not depend on whether a true relationship exists. |
+| [C18] | **QUALIFIED** | The paper treats `R` and `u` as model inputs and uses illustrative settings; it does not estimate universal values. |
+| [C19] | **SUPPORTED** | Corollaries 1-6 on PDF pp. 2-4 make these conditional model claims. |
+| [C20] | **SUPPORTED** | In Figure 1, all plotted powers exceed `alpha=0.05`, so PPV decreases as `u` rises; p. 2 notes the exception outside that plotted regime. |
+| [C21] | **SUPPORTED** | PDF p. 5, Table 4 and its footnote give nine PPVs using `alpha=0.05`, from `0.85` to `0.0010`. |
+| [C22] | **SUPPORTED** | The calculations establish conditional sensitivity; they do not measure the real-world inputs, so the stated separation is warranted. |
+| [C23] | **SUPPORTED** | The 2022 correction changes the readable Table 2 identity, showing why checking the DOI record before implementation matters. |
+| [C24] | **SUPPORTED** | Figure 1 and all nine Table 4 entries are precise, independently transcribable reproduction targets. |
+| [C25] | **SUPPORTED** | Calculation agreement tests arithmetic consistency only, not whether the assumptions or headline apply to a field. |
+
+## Effect of the correction
+
+- **SUPPORTED:** The correction repairs the parenthesization of one Table 2 cell.
+- **SUPPORTED:** With the correction, the two cells in the “True Relationship =
+  No” column sum to the published column total `c/(R+1)`.
+- **QUALIFIED:** The prose PPV equation with bias already contains the intended
+  combined numerator and therefore does not require a new formula.
+- **SUPPORTED:** Figure 1 and Table 4 use that prose PPV equation; their numerical
+  targets are unchanged by the typesetting correction.
+
+## Audit conclusion
+
+The companion is suitable after adding the correction explicitly. The executable
+lesson should intentionally show the original printed expression as `UNMATCHED`
+and the corrected expression as `MATCH`. This reproduces a calculation and checks
+an internal identity; it does not endorse the model assumptions or headline.

prototype/ioannidis-app-to-cli/app/example-outputs/paper-companion.md

@@ -0,0 +1,107 @@
+# Paper companion: *Why Most Published Research Findings Are False*
+
+Personalization: researcher outside biomedicine; goal is to understand how
+assumptions affect the reliability of published claims; background includes basic
+hypothesis testing but not Bayesian statistics.
+
+## 90-second explanation
+
+[C01][PAPER] The paper asks a conditional question: after a study reports a
+statistically significant relationship, how likely is that relationship to be
+true? Its answer is the positive predictive value (PPV). PPV depends on the
+pre-study odds that tested relationships are real, statistical power, the false
+positive rate, and—in extensions of the model—bias and repeated testing by
+multiple teams (PDF pp. 1-3, “Modeling the Framework,” Tables 1-3).
+
+[C02][PAPER] Low prior odds, low power, and greater bias can make a nominally
+significant finding more likely false than true within this model. Table 4 applies
+the model to nine illustrative research settings (PDF p. 5, Table 4).
+
+[C03][INFERENCE] The headline is best read as a warning about the joint effect of
+assumptions, study design, and selection—not as an empirical count of every
+published result.
+
+[OPEN QUESTION] How well do the chosen pre-study odds and the single bias
+parameter represent the participant's own field?
+
+## Reading map
+
+- [C04][PAPER] **First understand the base model:** PDF p. 1, “Modeling the Framework
+  for False Positive Findings,” then Table 1 on p. 2.
+- [C05][PAPER] **Then inspect the bias extension:** PDF p. 2, “Bias,” the PPV equation,
+  and Table 2.
+- [C06][PAPER] **See the sensitivity visually:** PDF p. 3, Figure 1. Compare the three
+  power panels and four values of bias, `u`.
+- [C07][PAPER] **Connect the model to research practice:** PDF pp. 3-5, the six
+  corollaries and Table 4.
+- [C08][PAPER] **Finish with remedies and limitations implicit in the discussion:**
+  PDF pp. 5-6, “How Can We Improve the Situation?”
+
+## Plain-language glossary
+
+- [C09][PAPER] **Pre-study odds, R:** expected true relationships divided by expected
+  non-true relationships among those tested; the corresponding probability is
+  `R/(R+1)` (PDF p. 1, model definition).
+- [C10][PAPER] **Power, 1-beta:** probability of detecting a relationship when it is
+  real (PDF p. 1, model definition).
+- [C11][PAPER] **Alpha:** probability of declaring a relationship when none exists;
+  the paper usually takes `alpha = 0.05` (PDF pp. 1-2, Table 1 discussion).
+- [C12][PAPER] **PPV:** probability that a reported positive finding is true within the
+  model (PDF p. 1, equation after Table 1).
+- [C13][PAPER] **Bias, u:** fraction of otherwise negative analyses presented as
+  positive because of design, analysis, or reporting choices (PDF p. 2, “Bias”).
+
+## Mathematical model and assumptions
+
+[C14][PAPER] Without the bias extension,
+
+`PPV = (1-beta) R / (R - beta R + alpha)`
+
+(PDF p. 1, displayed equation beside the Table 1 discussion).
+
+[C15][PAPER] With bias,
+
+`PPV = ([1-beta]R + u beta R) / (R + alpha - beta R + u - u alpha + u beta R)`
+
+(PDF p. 2, “Bias,” equation following Table 2).
+
+- [C16][PAPER] The model treats hypotheses as true or false and assumes a common power
+  for the relationships considered in a circumscribed field (PDF p. 1).
+- [C17][PAPER] The bias parameter is assumed not to depend on whether a relationship is
+  actually true (PDF p. 2, “Bias”).
+- [C18][INFERENCE] `R` and `u` are modeling inputs, not quantities the paper estimates
+  for science as a whole.
+- [OPEN QUESTION] Would a continuous effect-size model or heterogeneous powers
+  materially alter the conclusions for a given domain?
+
+## Main claims and evidence
+
+- [C19][PAPER] Smaller studies, smaller effects, less selective hypothesis searches,
+  more analytical flexibility, stronger interests, and more competing teams can
+  lower PPV in the proposed framework (PDF pp. 2-4, Corollaries 1-6).
+- [C20][PAPER] Figure 1 shows PPV decreasing as `u` increases at fixed power and
+  pre-study odds (PDF p. 3, Figure 1).
+- [C21][PAPER] Table 4 reports nine PPVs obtained from the model at `alpha = 0.05`,
+  ranging from `0.85` to `0.0010` for the selected examples (PDF p. 5, Table 4).
+- [C22][INFERENCE] These calculations demonstrate sensitivity to assumptions; they do
+  not independently establish the real-world values of `R` or `u`.
+
+## Questions worth asking
+
+- [OPEN QUESTION] Which inputs are empirically measurable, and which require
+  expert judgment?
+- [OPEN QUESTION] Does “a published finding” mean a hypothesis test, an effect
+  estimate, or an entire paper in the intended application?
+- [OPEN QUESTION] How sensitive are the headline conclusions to alternative
+  models of publication and analytical selection?
+- [OPEN QUESTION] Does the paper's current publication record contain corrections
+  that affect any equation or table?
+
+## Suggested next steps
+
+1. [C23][INFERENCE] Audit the paper's DOI for corrections before implementing its
+   equations.
+2. [C24][INFERENCE] Reproduce Figure 1 and all Table 4 values from an independently
+   transcribed oracle.
+3. [C25][INFERENCE] Report calculation agreement separately from agreement with the
+   assumptions or headline claim.

prototype/ioannidis-app-to-cli/app/example-outputs/replication-brief.md

@@ -0,0 +1,93 @@
+# Replication brief
+
+## Objective
+
+Reproduce the scientific content of Ioannidis (2005) Figure 1, all nine Table 4
+PPVs, and the Table 2 correction identity. Emit `MATCH` or `UNMATCHED` for every
+check. Reproduction does not endorse the model assumptions or headline claim.
+
+## Source provenance
+
+- Original: DOI `10.1371/journal.pmed.0020124`, local PDF SHA-256
+  `ffc1005680cb620eec4c913437dfabbf311b535cfe16cbaeb2faec1f92afc362`.
+- Correction: DOI `10.1371/journal.pmed.1004085`, local PDF SHA-256
+  `3cf06b98e2f61177d80c53ada12cd91949400dfdd114fefe9e2b47b87948c65e`.
+
+## Inputs and symbols
+
+- `R >= 0`: pre-study odds of true to non-true relationships.
+- `power = 1-beta`, constrained to `[0,1]`.
+- `alpha`: false-positive rate; use `0.05` for all published targets.
+- `u`: bias parameter, constrained to `[0,1]`.
+- `c`: number of relationships considered; set `c=1000` for the Table 2
+  identity because the choice cancels from the comparison.
+
+## Equations
+
+With `beta = 1-power`, calculate
+
+`PPV = (power R + u beta R) / (R + alpha - beta R + u - u alpha + u beta R)`.
+
+For the corrected Table 2 false-positive cell, calculate
+
+`(c alpha + u c (1-alpha)) / (R+1)`.
+
+For the original printed expression under normal operator precedence, calculate
+
+`c alpha + u c (1-alpha) / (R+1)`.
+
+The companion cell for a negative finding when no relationship exists is
+
+`(1-u)c(1-alpha)/(R+1)`,
+
+and the required column total is `c/(R+1)`.
+
+## Independent reference values
+
+The implementation must read these values rather than generate its own oracle.
+
+| power | R | u | published PPV |
+|---:|---:|---:|---:|
+| 0.80 | 1 | 0.10 | 0.85 |
+| 0.95 | 2 | 0.30 | 0.85 |
+| 0.80 | 1/3 | 0.40 | 0.41 |
+| 0.20 | 1/5 | 0.20 | 0.23 |
+| 0.20 | 1/5 | 0.80 | 0.17 |
+| 0.80 | 1/10 | 0.30 | 0.20 |
+| 0.20 | 1/10 | 0.30 | 0.12 |
+| 0.20 | 1/1000 | 0.80 | 0.0010 |
+| 0.20 | 1/1000 | 0.20 | 0.0015 |
+
+The supplied immutable oracle files are relative to `cli/starter/`:
+
+- `reference/published_table4.csv` contains the nine rows above; SHA-256
+  `a55c621c928f043bb4f7f7fbe36c0d81de3ff9fe980ca795e4e3fba9073e35f4`.
+- `reference/figure1_checkpoints.json` contains 36 checkpoints: every
+  combination of powers `(0.8, 0.5, 0.2)` and biases
+  `(0.05, 0.2, 0.5, 0.8)` at `R=(0.1, 0.5, 1.0)`; SHA-256
+  `8eaa83bdd8cad9b8ece9e10e005b9cccdd28c9fb5a89a37ee4f2e1f8b5c20cf4`.
+
+Treat these files as read-only. The implementation must consume them and must
+not derive, overwrite, or replace its own oracle.
+
+## Figure 1 target
+
+- Three vertically arranged panels: powers 80%, 50%, and 20%.
+- Four curves per panel: `u=0.05`, `0.20`, `0.50`, and `0.80`.
+- Horizontal axis: pre-study odds `R` from 0 to 1.
+- Vertical axis: PPV from 0% to 100%.
+- Verify numerical curve data; do not require pixel-identical rendering.
+
+## Acceptance tests
+
+1. **Table 4:** format each calculated PPV to the precision printed in the paper;
+   all nine lines must be `MATCH`.
+2. **Table 2 printed expression:** with `c=1000`, `R=0.1`, and `u=0.3`, the
+   two relevant cells must fail to equal `c/(R+1)`; print `UNMATCHED`.
+3. **Table 2 corrected expression:** the same column sum must equal `c/(R+1)`;
+   print `MATCH`.
+4. **Figure 1:** all 12 curves must match their three independent checkpoints
+   with absolute tolerance `1e-12`; print one status per curve.
+5. Save a readable PNG containing all three panels.
+6. Print final `OVERALL: MATCH` only when all expected outcomes occur, including
+   the deliberate `UNMATCHED` status for the original printed expression.

prototype/ioannidis-app-to-cli/app/prompts/01-paper-companion.md

@@ -0,0 +1,18 @@
+# Create my paper companion
+
+Read `sources/ioannidis-2005.pdf`. Do not browse or use outside sources in this
+first pass.
+
+Personalization: I am a researcher outside biomedicine reading this paper to
+understand how assumptions affect the reliability of published claims; I know
+basic hypothesis testing but not Bayesian statistics.
+
+Create a concise companion with: a 90-second explanation, a purpose-specific
+reading map, a plain-language glossary, the mathematical model and assumptions,
+the main claims and evidence, questions worth asking, and suggested next steps.
+Prefix every substantive item with `[PAPER]`, `[INFERENCE]`, or `[OPEN QUESTION]`.
+Also give every substantive `[PAPER]` and `[INFERENCE]` item a stable sequential
+claim ID such as `[C01]`; later stages will audit each ID.
+For `[PAPER]` items, cite the PDF page and the nearest section, table, figure, or
+displayed equation. Do not silently repair ambiguities or claim that reproduction
+would validate the paper's interpretation.

prototype/ioannidis-app-to-cli/app/prompts/02-independent-audit.md

@@ -0,0 +1,13 @@
+# Independently audit the companion
+
+Work in a fresh task. Read the uploaded original paper and the newly generated
+`paper-companion.md`. Use the paper DOI
+`10.1371/journal.pmed.0020124` to check the current publication record for
+corrections, retractions, or updated versions.
+
+Audit every `[Cnn]` companion claim exactly once against the paper and current
+record. Preserve each claim ID; classify it as `SUPPORTED`, `QUALIFIED`, or
+`UNSUPPORTED`; and give exact evidence. Report any correction and whether it
+changes the model, the numerical targets, or only the presentation. Separate
+successful calculation reproduction from agreement with assumptions or
+conclusions. Do not rewrite the companion.