Skip to content

Commit 34b854a

Browse files
authored
fix(update): ship and fetch native binaries from the GitHub release (#355)
## Requirement or Bug Native `pythinker update` fails with `native binary download returned HTTP 404`, and the 2.4.0 and 2.4.1 release runs on main end red. Related: #354. ## Bug Reproduction Steps 1. Install 2.1.0 with the native installer. 2. Run `pythinker update` and accept 2.4.0 (or 2.4.1). 3. `error: failed to download update 2.4.0: native binary download returned HTTP 404`. On 2.2.0–2.4.1 the same command fails earlier, because `manifest.json` comes back as an HTML page. Release runs 36917123003 (2.4.0) and 36942551995 (2.4.1) fail in `Update Homebrew tap` and `Release lane summary`. ## Root Cause Three separate defects. All three are fixed at the root, with no workarounds. 1. **Missing release asset (affects 2.1.0 clients).** `produce-manifest.mjs` writes `manifest.json` and `latest.json` entries that point at the bare binary `pythinker-code-<target>[.exe]`. It hashes that binary and then deletes it, so only `.zip`, `.zst` and `.tar.gz` get uploaded. A 2.1.0 client downloads the bare file and gets a 404. 2. **Wrong download base (affects 2.2.0–2.4.1 clients).** #323 moved the manifest and binary URLs to `code.pythinker.com/pythinker-code/binaries/<v>/…`. That route has never served a file: the site answers it with its SPA page and HTTP 200. These clients cannot update, and only a reinstall fixes them (#354). 3. **Brew job race (the red X).** `assertPublishedNpmVersion` ran `npm view` once, 24 s after `changeset publish`. npm showed 2.4.1 at 23:56:55, about 4 minutes after the check at 23:53:10. The job failed, and `Release lane summary` fails whenever `BREW_RESULT` is not success. ## Code Changes - `produce-manifest.mjs`: copy the binary into the upload set under the name the manifest uses, with a `.sha256` sidecar. The existing `dist-native-release/*` upload step then ships it. - `constant/app.ts`, `native-manifest.ts`, `native-stage.ts`: replace `pythinkerCodeCdnBinariesBase()` with `pythinkerCodeReleaseAssetUrl(version, filename)`. It returns the GitHub release URL with the encoded tag, which is the same shape as 2.1.0 and as `latest.json`. The `.zst`-first staging path does not change. - `update-brew-formula.mjs`: delete the one-shot `npm view` check. `downloadNpmTarball` already polls for 600 s, retries on 404, and refuses when the budget runs out, so it is the only gate now. - `cdn-consistency.mjs` + `verify-release-consistency.mjs`: the release gate now HEADs every URL that `latest.json` and the release `manifest.json` advertise, and fails on anything unreachable. It retries 403/429. Against the live 2.4.1 release it reports the 6 missing bare binaries, so it would have caught defect 1 on 2.3.0. - `README.md`: a pointer for stuck native installs to #354. ## Behavior Changes and Affected Users | Behavior | Before | After | Who relies on the old behavior | Escape hatch | |---|---|---|---|---| | Native release assets | `.zip`, `.zst`, `.tar.gz` (+ sidecars) | also the bare binary + `.sha256` | nobody; it adds an asset that manifests already named | n/a | | Native client download URLs (built from this tree) | `code.pythinker.com/pythinker-code/binaries/<v>/…` | `github.com/PyModel/pythinker-code/releases/download/<encoded tag>/…` | nobody: the CDN route never returned a file (SPA HTML, 200) | n/a | | Brew tap job when npm is slow to show the version | fails at once | polls the tarball up to 600 s | nobody; the old failure was a false negative | `RELEASE_LANE_BREW=disabled` (unchanged) | | Release consistency gate | checks version strings only | also fails on any unreachable advertised download | release operators | none needed; it only reports real 404s | Populations: - Native installs on 2.1.0 and earlier: they update normally from the next release on (defect 1). - Native installs on 2.2.0–2.4.1: they need one reinstall, because their own updater has the dead URL built in (defect 2, #354). - npm and Homebrew installs: not affected. Contract file touched: `apps/pythinker-code/src/cli/update/*` (CLI tripwire). The removed `/binaries/` base never served data, so no client from a previous release loses anything. Test coverage: - `release-artifacts.test.ts`: every file the manifest names exists in the upload set. Failed before the fix, passes after. - `native-manifest.test.ts`: exact encoded GitHub URLs. - `cdn-consistency.test.mjs`: URL collection, 404s, 403/429 retry. - `update-brew-formula.test.mjs`: the poll is the only gate. - `pnpm test:release` 54/54; app update suites 325/325; `tsc` clean; `pnpm lint` 0 errors. ## Checklist - [x] I have read the [CONTRIBUTING](https://github.com/PyModel/pythinker-code/blob/main/CONTRIBUTING.md) document. - [x] I have linked a related issue (external PRs: issue must have a maintainer's `/approve`). - [x] I have added tests that prove my feature works. - [x] The behavior-change table above is complete, and every removed behavior or flipped default is named in the changeset and either has an escape hatch or was explicitly approved by a maintainer in this PR. - [x] Ran `gen-changesets` skill, or this PR needs no changeset. - [x] Ran `gen-docs` skill, or this PR needs no doc update. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Native `pythinker update` now downloads the latest binary from the GitHub release, restoring updates for supported native installations. * If you’re using a native install on version 2.2.0–2.4.1, rerun the installation command once to receive a version that can update normally. After reinstalling, use `pythinker update` for future updates. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
1 parent 5d166ef commit 34b854a

13 files changed

Lines changed: 236 additions & 69 deletions

File tree

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
'@pymodel/pythinker-code': patch
3+
---
4+
5+
Native `pythinker update` downloads the new binary from the GitHub release again; native installs on 2.2.0–2.4.1 need one reinstall to receive it (see #354).

‎README.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -82,6 +82,8 @@ The CLI ships as a native binary, so there is no Node.js prerequisite.
8282
| Nix | `nix run github:PyModel/pythinker-code` |
8383
| npm | `npm install -g @pymodel/pythinker-code` (needs Node.js 24.15+) |
8484

85+
> Native install on 2.2.0–2.4.1? `pythinker update` cannot download new versions there. Run the install command again once to get a later version ([#354](https://github.com/PyModel/pythinker-code/issues/354)).
86+
8587
```sh
8688
cd your-project
8789
pythinker

‎apps/pythinker-code/scripts/native/produce-manifest.mjs‎

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,10 @@
99
* (produced by package.mjs across the 6 native-build matrix runners). The
1010
* zip is the only form in which binaries leave the matrix runners, so this
1111
* script extracts each bare executable and emits next to it:
12+
* pythinker-code-<target>[.exe] the bare binary; `filename` in the
13+
* manifest and `url` in the CDN latest.json
14+
* both name it, and updaters without zstd
15+
* support download it
1216
* pythinker-code-<target>.zst zstd -19, consumed by the staged updater
1317
* pythinker-code-<target>.tar.gz consumed by install.sh / install.ps1
1418
* <artifact>.sha256 sidecars in `<hex> <name>` format
@@ -23,7 +27,7 @@
2327
import { execFile } from 'node:child_process';
2428
import { createHash } from 'node:crypto';
2529
import { createReadStream } from 'node:fs';
26-
import { mkdtemp, readdir, rm, writeFile } from 'node:fs/promises';
30+
import { copyFile, mkdtemp, readdir, rm, writeFile } from 'node:fs/promises';
2731
import { tmpdir } from 'node:os';
2832
import { basename, join, resolve } from 'node:path';
2933
import { promisify } from 'node:util';
@@ -71,8 +75,8 @@ for (const sumFile of sumFiles.sort()) {
7175
const target = basename(sumFile, '.sha256').replace(/^pythinker-code-/, '').replace(/\.zip$/, '');
7276
const zipName = `pythinker-code-${target}.zip`;
7377
const exeName = target.startsWith('win32') ? 'pythinker.exe' : 'pythinker';
74-
// The CDN bare-binary layout carries the .exe suffix on Windows
75-
// (src/constant/app.ts); the updater's fallback downloads this filename.
78+
// Windows keeps the .exe suffix. Every file the manifest names is uploaded
79+
// to the release, so the updater can always fetch it.
7680
const binaryName = target.startsWith('win32') ? `pythinker-code-${target}.exe` : `pythinker-code-${target}`;
7781
const artifactBase = `pythinker-code-${target}`;
7882
const zstName = `${artifactBase}.zst`;
@@ -83,6 +87,8 @@ for (const sumFile of sumFiles.sort()) {
8387
await run('unzip', ['-o', resolve(inputDir, zipName), '-d', workDir]);
8488
const exePath = join(workDir, exeName);
8589
const binaryChecksum = await sha256File(exePath);
90+
await copyFile(exePath, resolve(inputDir, binaryName));
91+
await writeFile(resolve(inputDir, `${binaryName}.sha256`), `${binaryChecksum} ${binaryName}\n`);
8692
await run('zstd', ['-T0', '-19', '-q', '-f', '-o', resolve(inputDir, zstName), exePath]);
8793
await run('tar', ['-C', workDir, '-czf', resolve(inputDir, tarballName), exeName]);
8894

‎apps/pythinker-code/src/cli/update/native-manifest.ts‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
/**
2-
* Per-release native artifact manifest (`/binaries/<version>/manifest.json`).
2+
* Per-release native artifact manifest (`manifest.json` on the GitHub release).
33
*
44
* Published alongside the release and consumed by the install scripts; the
55
* staged updater reuses the same file so checksums and file names have a
@@ -12,7 +12,7 @@
1212
import { valid } from 'semver';
1313
import { z } from 'zod';
1414

15-
import { pythinkerCodeCdnBinariesBase } from '#/constant/app';
15+
import { pythinkerCodeReleaseAssetUrl } from '#/constant/app';
1616

1717
const MANIFEST_FETCH_TIMEOUT_MS = 10_000;
1818

@@ -47,11 +47,11 @@ export type NativeReleaseManifest = z.infer<typeof NativeReleaseManifestSchema>;
4747
export type NativePlatformEntry = z.infer<typeof PlatformEntrySchema>;
4848

4949
export function nativeManifestUrl(version: string): string {
50-
return `${pythinkerCodeCdnBinariesBase()}/${version}/manifest.json`;
50+
return pythinkerCodeReleaseAssetUrl(version, 'manifest.json');
5151
}
5252

5353
export function nativeBinaryUrl(version: string, filename: string): string {
54-
return `${pythinkerCodeCdnBinariesBase()}/${version}/${filename}`;
54+
return pythinkerCodeReleaseAssetUrl(version, filename);
5555
}
5656

5757
/**

‎apps/pythinker-code/src/cli/update/native-stage.ts‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,10 @@
33
* without touching the running executable. The actual swap happens on the
44
* next startup (see `native-swap.ts`).
55
*
6-
* The CDN serves the bare platform binary (e.g. `pythinker-code-win32-x64.exe`),
7-
* whose sha256 comes from the per-release manifest over HTTPS — a staged
8-
* binary is byte-exact what the release pipeline produced.
6+
* The GitHub release serves the bare platform binary (e.g.
7+
* `pythinker-code-win32-x64.exe`) and its `.zst` variant, whose sha256 comes
8+
* from the per-release manifest over HTTPS — a staged binary is byte-exact
9+
* what the release pipeline produced.
910
*/
1011

1112
import { createHash } from 'node:crypto';
@@ -440,7 +441,7 @@ export async function stageNativeUpdate(
440441
// would still be adopted here and reported as success, only for the
441442
// startup swap's claim-time re-verify to reject and discard it. Compare
442443
// the actual digest before adopting; a mismatch falls through and
443-
// re-stages from the CDN (published under a new generation name — the
444+
// re-stages from the release (published under a new generation name — the
444445
// damaged exe is left for the age-gated orphan cleanup).
445446
const digest = await hashFileSha256(stagedExePath(options.exePath, existing));
446447
if (digest === existing.sha256) {

‎apps/pythinker-code/src/constant/app.ts‎

Lines changed: 9 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -108,11 +108,15 @@ export function pythinkerCodeCdnLatestUrl(): string {
108108
export function pythinkerCodeCdnLatestJsonUrl(): string {
109109
return `${pythinkerCodeCdnBase()}/latest.json`;
110110
}
111-
// Per-release native artifacts: `/binaries/<version>/manifest.json` +
112-
// `/binaries/<version>/pythinker-code-<target>[.exe]` — the bare platform binary
113-
// (same layout install.ps1 consumes).
114-
export function pythinkerCodeCdnBinariesBase(): string {
115-
return `${pythinkerCodeCdnBase()}/binaries`;
111+
// Per-release native artifacts live on the GitHub release for that version:
112+
// `manifest.json` plus every file it names (bare binary and `.zst`). The
113+
// release pipeline uploads them and `latest.json` points at the same URLs.
114+
// The CDN serves no `/binaries/` route — it answers any unknown path with the
115+
// site's HTML and a 200.
116+
const PYTHINKER_CODE_GITHUB_RELEASES_BASE = 'https://github.com/PyModel/pythinker-code/releases/download';
117+
export function pythinkerCodeReleaseAssetUrl(version: string, filename: string): string {
118+
const tag = encodeURIComponent(`${NPM_PACKAGE_NAME}@${version}`);
119+
return `${PYTHINKER_CODE_GITHUB_RELEASES_BASE}/${tag}/${filename}`;
116120
}
117121
// The marketplace env override name lives in the shared agent-core-v2 plugin
118122
// domain (agent-gateway consumes it from there). Deep-path import: this module is

‎apps/pythinker-code/test/cli/update/native-manifest.test.ts‎

Lines changed: 4 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,6 @@ import {
66
nativeManifestUrl,
77
selectPlatformEntry,
88
} from '#/cli/update/native-manifest';
9-
import { pythinkerCodeCdnBinariesBase } from '#/constant/app';
109

1110
const VERSION = '0.7.0';
1211

@@ -195,10 +194,9 @@ describe('selectPlatformEntry', () => {
195194
});
196195

197196
describe('url helpers', () => {
198-
it('builds the manifest and binary URLs from the binaries base', () => {
199-
expect(nativeManifestUrl(VERSION)).toBe(`${pythinkerCodeCdnBinariesBase()}/${VERSION}/manifest.json`);
200-
expect(nativeBinaryUrl(VERSION, 'pythinker-code-win32-x64.zip')).toBe(
201-
`${pythinkerCodeCdnBinariesBase()}/${VERSION}/pythinker-code-win32-x64.zip`,
202-
);
197+
it('points the manifest and binaries at the GitHub release for the version', () => {
198+
const base = `https://github.com/PyModel/pythinker-code/releases/download/%40pymodel%2Fpythinker-code%40${VERSION}`;
199+
expect(nativeManifestUrl(VERSION)).toBe(`${base}/manifest.json`);
200+
expect(nativeBinaryUrl(VERSION, 'pythinker-code-win32-x64.exe')).toBe(`${base}/pythinker-code-win32-x64.exe`);
203201
});
204202
});

‎apps/pythinker-code/test/scripts/native/release-artifacts.test.ts‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -97,6 +97,8 @@ describe('native release artifacts', () => {
9797
for (const name of [
9898
`pythinker-code-${target}.zip`,
9999
`pythinker-code-${target}.zip.sha256`,
100+
`pythinker-code-${target}`,
101+
`pythinker-code-${target}.sha256`,
100102
`pythinker-code-${target}.zst`,
101103
`pythinker-code-${target}.zst.sha256`,
102104
`pythinker-code-${target}.tar.gz`,
@@ -168,6 +170,11 @@ describe('native release artifacts', () => {
168170
},
169171
},
170172
});
173+
const bare = resolve(artifactsDir, `pythinker-code-${target}`);
174+
expect(readFileSync(bare, 'utf-8')).toBe(binaryContent);
175+
expect(readFileSync(`${bare}.sha256`, 'utf-8')).toBe(
176+
`${sha256(Buffer.from(binaryContent))} pythinker-code-${target}\n`,
177+
);
171178
});
172179

173180
it('keeps the .exe suffix in Windows manifest filenames', async () => {
@@ -199,6 +206,7 @@ describe('native release artifacts', () => {
199206
expect(entry.filename).toBe('pythinker-code-win32-x64.exe');
200207
expect(entry.checksum).toBe(sha256(binaryContent));
201208
expect(entry.compressed.filename).toBe('pythinker-code-win32-x64.zst');
209+
expect(await readFile(join(releaseDir, entry.filename))).toEqual(binaryContent);
202210
} finally {
203211
rmSync(releaseDir, { recursive: true, force: true });
204212
}

‎scripts/release/cdn-consistency.mjs‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -114,3 +114,48 @@ export async function pollCdnUntilCaughtUp(options) {
114114
await sleep(intervalMs);
115115
}
116116
}
117+
118+
/**
119+
* Every download URL a client can be sent to for `version`: each
120+
* `platforms[*].url` in the CDN `latest.json`, plus every file the release
121+
* `manifest.json` names, resolved against the release asset base.
122+
*
123+
* A matching version string proves nothing about these: 2.4.0 and 2.4.1
124+
* shipped with the CDN in sync while every bare-binary URL returned 404.
125+
*/
126+
export function collectReleaseDownloadUrls({ latestJson, releaseManifest, releaseAssetUrl }) {
127+
const urls = new Set();
128+
for (const entry of Object.values(latestJson?.platforms ?? {})) {
129+
if (typeof entry?.url === 'string') urls.add(entry.url);
130+
}
131+
for (const entry of Object.values(releaseManifest?.platforms ?? {})) {
132+
for (const name of [entry?.filename, entry?.compressed?.filename, entry?.zstd?.file]) {
133+
if (typeof name === 'string') urls.add(releaseAssetUrl(name));
134+
}
135+
}
136+
return [...urls].sort((left, right) => left.localeCompare(right));
137+
}
138+
139+
/**
140+
* HEAD each URL and return the ones that do not answer 2xx. A transport error,
141+
* 5xx, 403 or 429 (GitHub rate limiting) is retried `attempts` times in total;
142+
* any other 4xx is final at once.
143+
*/
144+
export async function findUnreachableUrls({ fetchImpl, sleep, urls, attempts = 3, retryDelayMs = 5_000 }) {
145+
const unreachable = [];
146+
for (const url of urls) {
147+
let status = 'unreachable';
148+
for (let attempt = 1; attempt <= attempts; attempt += 1) {
149+
try {
150+
const response = await fetchImpl(url, { method: 'HEAD' });
151+
status = response.status;
152+
if (response.ok || (status >= 400 && status < 500 && status !== 403 && status !== 429)) break;
153+
} catch (error) {
154+
status = error instanceof Error ? error.message : 'unreachable';
155+
}
156+
if (attempt < attempts) await sleep(retryDelayMs);
157+
}
158+
if (typeof status !== 'number' || status < 200 || status >= 300) unreachable.push({ url, status });
159+
}
160+
return unreachable;
161+
}
Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
import assert from 'node:assert/strict';
2+
import test from 'node:test';
3+
4+
import { collectReleaseDownloadUrls, findUnreachableUrls } from './cdn-consistency.mjs';
5+
6+
const BASE = 'https://github.com/PyModel/pythinker-code/releases/download/%40pymodel%2Fpythinker-code%402.4.1';
7+
const releaseAssetUrl = (name) => `${BASE}/${name}`;
8+
9+
void test('collects latest.json urls and every file the release manifest names', () => {
10+
const urls = collectReleaseDownloadUrls({
11+
latestJson: {
12+
version: '2.4.1',
13+
platforms: { 'darwin-arm64': { url: `${BASE}/pythinker-code-darwin-arm64`, sha256: 'a' } },
14+
},
15+
releaseManifest: {
16+
platforms: {
17+
'darwin-arm64': {
18+
filename: 'pythinker-code-darwin-arm64',
19+
compressed: { filename: 'pythinker-code-darwin-arm64.zst' },
20+
},
21+
'win32-x64': { filename: 'pythinker-code-win32-x64.exe', zstd: { file: 'pythinker-code-win32-x64.zst' } },
22+
},
23+
},
24+
releaseAssetUrl,
25+
});
26+
assert.deepEqual(urls, [
27+
`${BASE}/pythinker-code-darwin-arm64`,
28+
`${BASE}/pythinker-code-darwin-arm64.zst`,
29+
`${BASE}/pythinker-code-win32-x64.exe`,
30+
`${BASE}/pythinker-code-win32-x64.zst`,
31+
]);
32+
});
33+
34+
void test('collects nothing from manifests without platforms', () => {
35+
assert.deepEqual(collectReleaseDownloadUrls({ latestJson: {}, releaseManifest: {}, releaseAssetUrl }), []);
36+
});
37+
38+
void test('reports a 404 at once and passes a 200', async () => {
39+
const calls = [];
40+
const unreachable = await findUnreachableUrls({
41+
fetchImpl: async (url, init) => {
42+
calls.push([url, init.method]);
43+
return new Response(null, { status: url.endsWith('.zst') ? 200 : 404 });
44+
},
45+
sleep: async () => {},
46+
urls: ['a.zst', 'a'],
47+
});
48+
assert.deepEqual(unreachable, [{ url: 'a', status: 404 }]);
49+
assert.deepEqual(calls, [
50+
['a.zst', 'HEAD'],
51+
['a', 'HEAD'],
52+
]);
53+
});
54+
55+
void test('retries transport errors and 5xx, then reports the last failure', async () => {
56+
let calls = 0;
57+
const sleeps = [];
58+
const unreachable = await findUnreachableUrls({
59+
fetchImpl: async () => {
60+
calls += 1;
61+
if (calls === 1) throw new Error('socket hang up');
62+
return new Response(null, { status: 503 });
63+
},
64+
sleep: async (ms) => {
65+
sleeps.push(ms);
66+
},
67+
urls: ['a'],
68+
attempts: 3,
69+
retryDelayMs: 10,
70+
});
71+
assert.equal(calls, 3);
72+
assert.deepEqual(sleeps, [10, 10]);
73+
assert.deepEqual(unreachable, [{ url: 'a', status: 503 }]);
74+
});
75+
76+
void test('recovers when a retry succeeds', async () => {
77+
let calls = 0;
78+
const unreachable = await findUnreachableUrls({
79+
fetchImpl: async () => {
80+
calls += 1;
81+
return new Response(null, { status: calls === 1 ? 502 : 200 });
82+
},
83+
sleep: async () => {},
84+
urls: ['a'],
85+
});
86+
assert.deepEqual(unreachable, []);
87+
assert.equal(calls, 2);
88+
});
89+
90+
void test('retries a rate-limited 429 instead of reporting it at once', async () => {
91+
let calls = 0;
92+
const unreachable = await findUnreachableUrls({
93+
fetchImpl: async () => {
94+
calls += 1;
95+
return new Response(null, { status: calls === 1 ? 429 : 200 });
96+
},
97+
sleep: async () => {},
98+
urls: ['a'],
99+
});
100+
assert.deepEqual(unreachable, []);
101+
assert.equal(calls, 2);
102+
});

0 commit comments

Comments
 (0)