Implement verify automatically on macOS and Linux

[bettercallsaulj]

No description provided.

[dIvYaNshhh]

scripts/verify-example-native-probe.py is the best thing in this whole stack — checking the eight required symbols via hasattr on the loaded CDLL is exactly the missing-symbol detection I asked for on #10, and it correctly surfaces get_load_error_message() from #13 when the load fails outright. That file alone would catch several of the failure modes we've been discussing.

The macOS native checks are good too. Removing Homebrew to simulate a clean machine is a genuinely nice idea — it catches the class of bug where a dylib picks up /opt/homebrew paths that exist on the developer's machine and nowhere else — and the otool -L | grep '/usr/local/|/opt/homebrew/' check pairs with it well. The @loader_path loop uses process substitution so missing=1 survives in the parent shell, which is the correct pattern (and notably better than the pipe-into-while in #13's Docker script).

The harness itself is well built: enumerated --mode, regex-constrained --only, a real example registry, mktemp -d rather than a predictable path, permissions: contents: read at the top. No complaints about the structure.

Where I think it needs work is the verification semantics — right now I don't believe a green run tells us much:

1. It verifies PyPI, not the PR. Nothing installs the checked-out branch, so a PR that breaks the SDK still passes.
2. The live assertion passes unconditionally — grep -i "draft" over output that contains the echoed prompt "List my draft mails". Same for the Agent Response marker, which is the example's own header printed before the answer.
3. live is the default mode on PRs, and it's the only mode that hard-fails on missing secrets — which forks never have.
4. Live agent answers get printed into the CI log, which with a real GOPHER_API_KEY means real user data.

Fix 1 and 2 and this becomes a genuinely useful gate. Details inline, along with some smaller notes on concurrency/cost, the push: [iml_verify_auto] trigger, and the trap ordering.

[dIvYaNshhh]

Re-reviewed at 7c62bde0. Six of eight addressed, and the two fixes I cared most about are both good. Resolving those threads.

Confirmed fixed:

• Verifies the PR, not PyPI (5e19d0d7) — pip install -e . on pull_request, PyPI specs retained for dispatch/scheduled runs, and SDK_INSTALL_SPEC threaded through to the harness so it's consistent end to end. This was the big one.
• The tautological live assertion (2e776c61 / 378ae4d5) — nicely solved. VERIFY_EXPECTED_ANSWER is now a unique draft id (r-2553040815323886578) that can't appear in the prompt, so the check can actually fail. The VERIFY_EXPECTED_ANSWER_TERMS addition is a good generalisation.
• Mode default (af370aa6) — auto, plus concurrency: with cancel-in-progress and timeout-minutes: 30. All three of my cost/robustness notes in one go.
• Output redaction (59200a8a) — logs answer_bytes=N and live: OK (answer redacted) instead of the agent's answer. Exactly the verdict-not-content shape I was after, and it keeps real user data out of CI logs.
• Trap ordering (e3398837) — trap cleanup EXIT now precedes create_project, so a failed pip install no longer leaks a venv.
• Missing-env check (97c19a22) — ERROR: missing-required-env: <keys> replaces the English-prose grep. Much more durable.
• Linux native dependency checks (b778873d) — closes the macOS/Linux asymmetry.

Suite at the tip of the stack: 151 passed, 116 skipped, 3 failed — all three reproduce identically on origin/main (they need ./build.sh), so nothing regressed.

Two things inline: the push: [iml_verify_auto] trigger is still there, and the newest commit downgraded the OpenSSL bundling check from exit 1 to a warning — understandable timing-wise, but it's the only enforcement of #13's OpenSSL fix and there's nothing tracking its restoration.

Also still empty on the description. Same note as the rest of the stack.

[dIvYaNshhh]

85769cdf handles the OpenSSL point exactly the way I hoped — resolving that thread.

if [ "${{ github.event_name }}" = "pull_request" ]; then
  echo "OpenSSL libraries must remain system-provided in PR-built packages."
  exit 1
fi
echo "WARNING: OpenSSL libraries should remain system-provided in newly published packages."

PR builds use the checkout, so they're held to the strict rule and #13's fix is genuinely enforced; PyPI-sourced runs stay a warning so the live path doesn't red until a clean package ships. That's the right split, and it means the check tightens automatically rather than needing someone to remember to turn it back on.

Suite at the tip of the stack: 153 passed, 116 skipped, 3 failed — all three reproduce identically on origin/main (they need ./build.sh), so nothing regressed.

One item left: push: branches: [iml_verify_auto], inline. Suggestion block attached, should be a 10-second fix.

Two process notes, neither a code change:

1. The 12.4 MB of binaries removed in #12 are still reachable in that branch's history, and this PR sits on top of it. With allow_merge_commit / allow_rebase_merge enabled they'd reach main on merge — squash-merging the stack avoids it. Detail on #12.
2. Description still empty across all four PRs.

[dIvYaNshhh]

Status: 1 item left

Everything else on this PR is resolved — 10 threads closed, including the two I cared most about (verifying the PR checkout instead of PyPI, and the live assertion that couldn't fail). Test suite at the tip of the stack is 153 passed / 116 skipped / 3 failed, and those 3 fail identically on origin/main (they need ./build.sh), so nothing here regressed.

The only outstanding item is the leftover dev trigger in .github/workflows/verify-examples.yml:

on:
  pull_request:
    branches: [main]
  push:
    branches: [iml_verify_auto]   # <- this

After merge this keeps firing on any push to iml_verify_auto and never fires on main. Since the job now makes real Anthropic calls and real MCP fetches across a 2-platform matrix, that's ongoing spend on a branch nobody is watching, triggered by a workflow that by then lives on main.

Either fix works:

  push:
    branches: [main]

or just drop the push: block — pull_request + workflow_dispatch already cover the real gating.

There's a suggestion block on the inline thread if you want to commit it straight from the UI.

---

Two non-blocking things for the stack as a whole, both flagged on #12:

1. Squash-merge recommended. The 12.4 MB of binaries removed in de69016c are still reachable in feature_token_protect's history (added e0c4bc47, removed de69016c). With allow_merge_commit / allow_rebase_merge enabled they'd land in main's history permanently even though no commit on main would contain them. A squash merge carries only the final tree and avoids it — cheap to decide now, needs a history rewrite later.
2. All four PR descriptions are still empty.