feat: hydrogen skills check for checking if skills are up to date

.changeset/skills-check-command.md

@@ -0,0 +1,5 @@
+---
+"@shopify/hydrogen": minor
+---
+
+Add `hydrogen skills check`, which exits non-zero when the project's synced skills do not match the installed `@shopify/hydrogen`, or were never synced. It prints the summary `skills sync` would act on and writes nothing. The default `--mode=error` gates CI; `--mode=warn` prints the same message and exits zero for dev scripts.

.docs/dependencies.md

@@ -32,13 +32,14 @@ When changing `@shopify/hydrogen/vite` local HTTPS behaviour, update these toget
 - `templates/react-router/vite.config.ts`
 - framework example `dev:https` scripts and configs under `examples/*`
 
-`scripts/preview-template-dist.ts` syncs `packages/hydrogen/skills` into template `.agents/skills` and `.claude/skills` when preparing the dist branch, so template source directories should not duplicate those generated skill copies.
+`scripts/preview-template-dist.ts` syncs `packages/hydrogen/skills` into each template's harness skill directories (`.claude/skills` and `.agents/skills`) when preparing the dist branch, so template source directories should not duplicate those generated skill copies.
 
 ## Hydrogen Skills Sync
 
 When changing how synced skills are stamped or verified (the frontmatter `metadata` block with `source`, `version`, and `hash`, or the overwrite/skip/remove rules), update these together:
 
 - `packages/hydrogen/src/cli/skills.ts` (single writer and reader of the metadata block)
 - `scripts/preview-template-dist.ts` (imports `syncSkills` so template copies carry the same metadata)
+- `hydrogen skills check` in the same file (reads `getSkillsSyncStatus`, prints `describeSkillsSyncStatus`)
 - `packages/hydrogen/README.md` ("Keeping skills in sync")
 - root `README.md` ("Set up in your own project")

README.md

@@ -68,7 +68,7 @@ Then add Hydrogen from your project directory:
 npx @shopify/hydrogen@preview setup
 ```
 
-`setup` installs `@shopify/hydrogen` with your project's package manager and copies Hydrogen's agent skills into your project's skills directory — matched to the version you just installed. After upgrading Hydrogen later, run `npx @shopify/hydrogen skills sync` to bring the skills back in line (see the [package README](packages/hydrogen/README.md#keeping-skills-in-sync)).
+`setup` installs `@shopify/hydrogen` with your project's package manager and copies Hydrogen's agent skills into your project's skills directory — matched to the version you just installed. After upgrading Hydrogen later, run `npx @shopify/hydrogen skills sync` to bring the skills back in line, and `npx @shopify/hydrogen skills check` to verify they are (it exits non-zero when they are not, so it can run in CI). See the [package README](packages/hydrogen/README.md#keeping-skills-in-sync).
 
 Now ask your coding agent to build the storefront:
 

packages/hydrogen/README.md

@@ -31,6 +31,18 @@ Each synced `SKILL.md` records the package version and a content hash in its fro
 
 `--force` discards local state in favour of the package: it overwrites edited and colliding skills and removes edited stale ones.
 
+Gate CI on the same comparison. The command exits non-zero when a sync would change anything, or when skills were never synced:
+
+```bash
+npx @shopify/hydrogen skills check
+```
+
+For dev scripts, `--mode=warn` prints the same message and exits zero so the next command still runs, and prints nothing at all when skills are current:
+
+```json
+{ "scripts": { "dev": "hydrogen skills check --mode=warn && vite dev" } }
+```
+
 ## GraphQL Tooling
 
 Enable Storefront and Customer Account API editor diagnostics with Hydrogen's packed TypeScript plugin:

packages/hydrogen/skills/hydrogen-setup/steps/11-verify.md

@@ -20,6 +20,7 @@ If Playwright is present, run it headless.
 
 - [ ] Every applicable script was executed in this session, after the last code change, and passed
 - [ ] `hydrogen gql check` ran as part of static checks (not skipped)
+- [ ] `dev` still starts with `hydrogen skills check --mode=warn` and prints no skills warning
 
 ## Run Runtime Smoke Tests
 

packages/hydrogen/skills/hydrogen-setup/steps/2-scaffold.md

@@ -18,6 +18,25 @@ Invoke the `hydrogen-routing` skill and create the shared route template manifes
 
 - [ ] Route templates are set up
 
+## Keep Skills In Sync
+
+The packaged skills describe the installed Hydrogen version. Chain `hydrogen skills check --mode=warn` in front of the app's `dev` script so a later `@shopify/hydrogen` bump prints the resync command the next time the dev server starts, without blocking it:
+
+```json
+{
+  "scripts": {
+    "dev": "hydrogen skills check --mode=warn && <framework dev command>"
+  }
+}
+```
+
+Keep the framework's own dev command exactly as it was; only prefix it. Apply the same prefix to any sibling dev script such as `dev:https`. Do not add the plain `hydrogen skills check` (error mode) to `dev`; that belongs in CI, where a stale skill copy should fail the build.
+
+### Continue when
+
+- [ ] `dev` runs `hydrogen skills check --mode=warn` before the framework dev command
+- [ ] Running the `dev` script prints no skills warning (skills were synced by `hydrogen setup`)
+
 ## Use Standard Environment Names
 
 Use these canonical environment variable names throughout the app (kept in sync with `hydrogen-storefront-client`):

packages/hydrogen/src/cli/__tests__/setup.test.ts

@@ -222,7 +222,7 @@ describe("setupHydrogen", () => {
     });
 
     await setupHydrogen({
-      args: ["--force"],
+      force: true,
       cwd: appRoot,
       packageRoot,
       runCommand: createRunCommandSpy(),

packages/hydrogen/src/cli/__tests__/skills.test.ts

@@ -15,7 +15,17 @@ import { describe, expect, it, vi } from "vitest";
 
 import { assert } from "../../core/test-utils";
 import { isObjectRecord } from "../../core/utils/record";
-import { syncSkills, type SyncSkillsResult, type SyncSkillsRootResult } from "../skills";
+import {
+  checkSkills,
+  describeSkillsSyncStatus,
+  parseSkillsCheckArgs,
+  getSkillsSyncStatus,
+  parseSkillsSyncArgs,
+  syncSkills,
+  type SkillsSyncStatus,
+  type SyncSkillsResult,
+  type SyncSkillsRootResult,
+} from "../skills";
 
 const REAL_SKILLS_ROOT = join(import.meta.dirname, "../../../skills");
 
@@ -93,12 +103,21 @@ function readMetadata(content: string): { version: string; hash: string } {
   return { version, hash };
 }
 
+function statusFixture(overrides: Partial<SkillsSyncStatus> = {}): SkillsSyncStatus {
+  return {
+    version: "2026.2.0",
+    pending: { add: 0, update: 0, remove: 0, modified: 0 },
+    conflicts: [],
+    ...overrides,
+  };
+}
+
 /** Tests that expect a prompt pass their own confirm; everything else must never ask. */
 const rejectPrompt = (question: string): Promise<boolean> =>
   Promise.reject(new Error(`Unexpected prompt: ${question}`));
 
-function sync(appRoot: string, packageRoot: string, args: string[] = []) {
-  return syncSkills({ cwd: appRoot, packageRoot, args, log: vi.fn(), confirm: rejectPrompt });
+function sync(appRoot: string, packageRoot: string, force = false) {
+  return syncSkills({ cwd: appRoot, packageRoot, force, log: vi.fn(), confirm: rejectPrompt });
 }
 
 function rootResult(result: SyncSkillsResult, harness: string): SyncSkillsRootResult {
@@ -181,7 +200,7 @@ describe("syncSkills", () => {
     });
     expect(readSkill(appRoot, "hydrogen-cart-ui")).toContain("My local note.");
 
-    const forced = await sync(appRoot, newPackageRoot, ["--force"]);
+    const forced = await sync(appRoot, newPackageRoot, true);
     expect(agents(forced)).toMatchObject({ updated: 1, skipped: [] });
     expect(readSkill(appRoot, "hydrogen-cart-ui")).toContain("New.");
     expect(readSkill(appRoot, "hydrogen-cart-ui")).not.toContain("My local note.");
@@ -209,7 +228,7 @@ describe("syncSkills", () => {
     const skillFile = join(appRoot, ".agents/skills/hydrogen-cart-ui/SKILL.md");
     writeFileSync(skillFile, readFileSync(skillFile, "utf8") + "My local note.\n");
 
-    const result = await sync(appRoot, packageRoot, ["--force"]);
+    const result = await sync(appRoot, packageRoot, true);
 
     expect(agents(result)).toMatchObject({ updated: 1, unchanged: 0, skipped: [] });
     expect(readSkill(appRoot, "hydrogen-cart-ui")).not.toContain("My local note.");
@@ -394,7 +413,7 @@ describe("syncSkills", () => {
     it("removes without asking when forced", async () => {
       const { appRoot, skillFile, newPackageRoot } = await createEditedStaleSkill();
 
-      const result = await sync(appRoot, newPackageRoot, ["--force"]);
+      const result = await sync(appRoot, newPackageRoot, true);
 
       expect(agents(result)).toMatchObject({ removed: 1, kept: [], skipped: [] });
       expect(existsSync(skillFile)).toBe(false);
@@ -425,7 +444,7 @@ describe("syncSkills", () => {
     expect(existsSync(join(appRoot, ".agents/skills/hydrogen-setup"))).toBe(false);
     expect(readSkill(appRoot, "hydrogen-cart-ui")).toContain("Handwritten.");
 
-    expect(agents(await sync(appRoot, packageRoot, ["--force"]))).toMatchObject({
+    expect(agents(await sync(appRoot, packageRoot, true))).toMatchObject({
       added: 1,
       updated: 1,
     });
@@ -472,11 +491,10 @@ describe("syncSkills", () => {
     expect(existsSync(join(appRoot, ".claude/skills"))).toBe(false);
   });
 
-  it("rejects unknown arguments", async () => {
-    const appRoot = createAppRoot();
-    const packageRoot = createPackageRoot("2026.1.0");
-
-    await expect(sync(appRoot, packageRoot, ["--yolo"])).rejects.toThrow();
+  it("parses --force and rejects unknown flags", () => {
+    expect(parseSkillsSyncArgs([])).toEqual({ force: false });
+    expect(parseSkillsSyncArgs(["--force"])).toEqual({ force: true });
+    expect(() => parseSkillsSyncArgs(["--yolo"])).toThrow("--yolo");
   });
 
   describe("package resolution", () => {
@@ -565,4 +583,204 @@ describe("syncSkills", () => {
       readMetadata(readSkill(appRoot, skillName));
     }
   });
+
+  describe("getSkillsSyncStatus", () => {
+    it("reports everything pending for a project that never synced", () => {
+      const appRoot = createAppRoot();
+      const packageRoot = createPackageRoot("2026.1.0", { "hydrogen-cart-ui": "Cart.\n" });
+
+      const status = getSkillsSyncStatus({ cwd: appRoot, packageRoot });
+
+      expect(status).toEqual({
+        version: "2026.1.0",
+        pending: { add: 1, update: 0, remove: 0, modified: 0 },
+        conflicts: [],
+      });
+    });
+
+    it("reports nothing pending right after a sync", async () => {
+      const appRoot = createAppRoot();
+      const packageRoot = createPackageRoot("2026.1.0", { "hydrogen-cart-ui": "Cart.\n" });
+      await sync(appRoot, packageRoot);
+
+      const status = getSkillsSyncStatus({ cwd: appRoot, packageRoot });
+
+      expect(status.pending).toEqual({ add: 0, update: 0, remove: 0, modified: 0 });
+    });
+
+    it("counts updates, additions, removals, and local modifications after an upgrade", async () => {
+      const appRoot = createAppRoot();
+      await sync(
+        appRoot,
+        createPackageRoot("2026.1.0", {
+          "hydrogen-cart-ui": "Cart.\n",
+          "hydrogen-legacy": "Legacy.\n",
+          "hydrogen-money": "Money.\n",
+        }),
+      );
+      for (const harness of [".claude", ".agents"]) {
+        const moneyFile = join(appRoot, harness, "skills/hydrogen-money/SKILL.md");
+        writeFileSync(moneyFile, readFileSync(moneyFile, "utf8") + "Mine.\n");
+      }
+      const newPackageRoot = createPackageRoot("2026.2.0", {
+        "hydrogen-cart-ui": "Cart v2.\n",
+        "hydrogen-money": "Money v2.\n",
+        "hydrogen-image": "Image.\n",
+      });
+
+      const status = getSkillsSyncStatus({ cwd: appRoot, packageRoot: newPackageRoot });
+
+      expect(status).toEqual({
+        version: "2026.2.0",
+        pending: { add: 1, update: 1, remove: 1, modified: 1 },
+        conflicts: [],
+      });
+      expect(readSkill(appRoot, "hydrogen-cart-ui")).toContain("Cart.\n");
+    });
+
+    it("counts an edited skill the package no longer ships as locally modified", async () => {
+      const appRoot = createAppRoot();
+      await sync(appRoot, createPackageRoot("2026.1.0", { "hydrogen-legacy": "Legacy.\n" }));
+      const skillFile = join(appRoot, ".agents/skills/hydrogen-legacy/SKILL.md");
+      writeFileSync(skillFile, readFileSync(skillFile, "utf8") + "Mine.\n");
+
+      const status = getSkillsSyncStatus({
+        cwd: appRoot,
+        packageRoot: createPackageRoot("2026.2.0"),
+      });
+
+      // The .claude copy is untouched and would be removed; the edited .agents copy waits for consent.
+      expect(status.pending).toEqual({ add: 0, update: 0, remove: 1, modified: 1 });
+    });
+
+    it("surfaces name collisions without throwing", () => {
+      const appRoot = createAppRoot();
+      writeSkill(join(appRoot, ".agents/skills"), "hydrogen-cart-ui", "Handwritten.\n");
+      const packageRoot = createPackageRoot("2026.1.0", { "hydrogen-cart-ui": "Cart.\n" });
+
+      const status = getSkillsSyncStatus({ cwd: appRoot, packageRoot });
+
+      expect(status.conflicts).toEqual([join(appRoot, ".agents/skills/hydrogen-cart-ui")]);
+    });
+  });
+
+  describe("describeSkillsSyncStatus", () => {
+    it("returns nothing when everything is current", () => {
+      expect(describeSkillsSyncStatus(statusFixture())).toBeUndefined();
+    });
+
+    it("lists pending updates, additions, and removals with the sync command", () => {
+      expect(
+        describeSkillsSyncStatus(
+          statusFixture({ pending: { add: 2, update: 5, remove: 1, modified: 0 } }),
+        ),
+      ).toBe(
+        "Hydrogen skills are out of date with @shopify/hydrogen 2026.2.0 (5 to update, 2 new, 1 removed upstream). Run `npx @shopify/hydrogen skills sync`.",
+      );
+    });
+
+    it("points at --force when only locally modified skills are behind", () => {
+      expect(
+        describeSkillsSyncStatus(
+          statusFixture({ pending: { add: 0, update: 0, remove: 0, modified: 2 } }),
+        ),
+      ).toBe(
+        "Hydrogen skills are out of date with @shopify/hydrogen 2026.2.0 (2 locally modified). Run `npx @shopify/hydrogen skills sync --force` to reset them.",
+      );
+    });
+
+    it("reports locally modified skills alongside other pending changes", () => {
+      expect(
+        describeSkillsSyncStatus(
+          statusFixture({ pending: { add: 1, update: 3, remove: 0, modified: 2 } }),
+        ),
+      ).toBe(
+        "Hydrogen skills are out of date with @shopify/hydrogen 2026.2.0 (3 to update, 1 new, 2 locally modified). Run `npx @shopify/hydrogen skills sync`, or `npx @shopify/hydrogen skills sync --force` to also reset the locally modified ones.",
+      );
+    });
+
+    it("explains collisions", () => {
+      const message = describeSkillsSyncStatus(
+        statusFixture({ conflicts: ["/app/.agents/skills/x"] }),
+      );
+
+      expect(message).toContain("/app/.agents/skills/x");
+      expect(message).toContain("--force");
+    });
+  });
+
+  describe("checkSkills", () => {
+    it("passes and reports the version when skills are current", async () => {
+      const appRoot = createAppRoot();
+      const packageRoot = createPackageRoot("2026.1.0", { "hydrogen-cart-ui": "Cart.\n" });
+      await sync(appRoot, packageRoot);
+      const log = vi.fn();
+
+      checkSkills({ cwd: appRoot, packageRoot, log });
+
+      expect(log).toHaveBeenCalledWith(
+        "Hydrogen skills are up to date with @shopify/hydrogen 2026.1.0.",
+      );
+    });
+
+    it("fails when the package moved on", async () => {
+      const appRoot = createAppRoot();
+      await sync(appRoot, createPackageRoot("2026.1.0", { "hydrogen-cart-ui": "Cart.\n" }));
+      const newPackageRoot = createPackageRoot("2026.2.0", { "hydrogen-cart-ui": "Cart v2.\n" });
+
+      expect(() =>
+        checkSkills({ cwd: appRoot, packageRoot: newPackageRoot, log: vi.fn() }),
+      ).toThrow("out of date with @shopify/hydrogen 2026.2.0 (1 to update)");
+    });
+
+    it("fails when skills were never synced", () => {
+      const appRoot = createAppRoot();
+      const packageRoot = createPackageRoot("2026.1.0", { "hydrogen-cart-ui": "Cart.\n" });
+
+      expect(() => checkSkills({ cwd: appRoot, packageRoot, log: vi.fn() })).toThrow("1 new");
+    });
+
+    it("warns and returns instead of throwing in warn mode", () => {
+      const appRoot = createAppRoot();
+      const packageRoot = createPackageRoot("2026.1.0", { "hydrogen-cart-ui": "Cart.\n" });
+      const log = vi.fn();
+      const warn = vi.fn();
+
+      checkSkills({ cwd: appRoot, packageRoot, mode: "warn", log, warn });
+
+      expect(warn).toHaveBeenCalledTimes(1);
+      expect(warn.mock.calls[0]?.[0]).toContain("1 new");
+      expect(log).not.toHaveBeenCalled();
+    });
+
+    it("stays silent in warn mode when skills are current", async () => {
+      const appRoot = createAppRoot();
+      const packageRoot = createPackageRoot("2026.1.0", { "hydrogen-cart-ui": "Cart.\n" });
+      await sync(appRoot, packageRoot);
+      const log = vi.fn();
+      const warn = vi.fn();
+
+      checkSkills({ cwd: appRoot, packageRoot, mode: "warn", log, warn });
+
+      expect(log).not.toHaveBeenCalled();
+      expect(warn).not.toHaveBeenCalled();
+    });
+
+    it("parses --mode and rejects unknown values", () => {
+      expect(parseSkillsCheckArgs([])).toEqual({ mode: "error" });
+      expect(parseSkillsCheckArgs(["--mode=warn"])).toEqual({ mode: "warn" });
+      expect(parseSkillsCheckArgs(["--mode", "error"])).toEqual({ mode: "error" });
+      expect(() => parseSkillsCheckArgs(["--mode=loud"])).toThrow("Unknown --mode 'loud'");
+      expect(() => parseSkillsCheckArgs(["--yolo"])).toThrow("--yolo");
+    });
+
+    it("never writes", () => {
+      const appRoot = createAppRoot();
+      const packageRoot = createPackageRoot("2026.1.0", { "hydrogen-cart-ui": "Cart.\n" });
+
+      expect(() => checkSkills({ cwd: appRoot, packageRoot, log: vi.fn() })).toThrow();
+      expect(existsSync(join(appRoot, ".agents/skills"))).toBe(false);
+      expect(existsSync(join(appRoot, ".claude"))).toBe(false);
+    });
+  });
 });