Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 24 additions & 8 deletions .github/skills/add-e2e-test/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,27 +87,43 @@ carries the API mechanics.
`test("change UI language repeatedly [Test Case ID 69]", ...)` — so the code and the
inventory stay tied. Read the card's Test Steps checkboxes; they are the behavior
contract. When the automated test lands, set the card's `Automation` property to
`Automated` — or to `Partial` when the automated test covers only part of the steps,
and say which part in `Automation Notes`. While the test is still in an open PR, the
card belongs in `PR Pending` instead, with the PR URL in `Automation Notes`. The title string is the whole mechanism;
`Automated`. While the test is still in an open PR, the card belongs in `PR Pending`
instead, with the PR URL in `Automation Notes`. The title string is the whole mechanism;
the library provides no helper or annotation for it, deliberately, so that grepping
`Test Case ID` across `src/BloomE2E/tests/` finds every tie.
- **When the test covers only part of the card's steps, split the card.** A card marked
`Automated` while some of its steps are still human-run hides those steps: nobody reads
`Automation Notes` when planning a manual run. So a card is never half automated.
1. Rename the original to `<title> [Automated portion]`. It keeps its `Test Case ID`,
because the test source carries that id, and keeps only the steps the test covers.
2. Add a row `<title> [Manual portion]` with the next free `Test Case ID` and the same
`Test Suite Run`, `Areas`, `Priority`, and `Dokimion ID`. Move the uncovered steps into
it. Start its body with a callout that names the automated card and says, per step,
why it is not automated (microphone, native dialog, WinForms surface) and which
`AUTOMATION-DEBT.md` entry covers it. Its `Automation` is `Manual`, or `Keep manual`
when the steps can never be automated.
3. Link the two rows through the `Related Cases` relation property, in both directions,
and name the other card's id in each `Summary`.
Do this when you set `PR Pending`, not after the merge. Example: Test Case ID 349,
"Duplicate Page [Automated portion]", and its manual portion, Test Case ID 810.
- **Writing a new e2e test that has no manual card:** add a row to the inventory so it
remains the inventory of ALL tests, not only human-run ones. Allocate the next free
`Test Case ID`, fill in the title, Summary, and Areas, and set `Automation` to
`Automated`.
- **The `Automation` select property** holds the case's automation lifecycle:
`Manual` → `Planned` → `Building` → `PR Pending` → `Automated` (or `Partial`), with
`Keep manual` as the deliberate opt-out.
`Manual` → `Planned` → `Building` → `PR Pending` → `Automated`, with `Keep manual` as
the deliberate opt-out.
- Empty means the same as `Manual` — the legacy rows were not bulk-stamped.
- `Planned` marks a case the team judged a good automation candidate. To find work,
filter the current suite run on `Automation = Planned`.
- `Building` means someone is automating it right now. Set it when you start, so two
people or agents do not automate the same case; set `Automated` (or `Partial`,
with the covered part named in `Automation Notes`) when the test lands.
people or agents do not automate the same case; set `Automated` when the test lands,
after splitting the card if the test covers only part of its steps.
- `PR Pending` means the test exists in an open PR that has not merged. Put the PR URL
in `Automation Notes`. The `improve-test-automation-coverage` skill leaves cards here;
a human (or a later sweep) moves them to `Automated` or `Partial` after the merge.
a human (or a later sweep) moves them to `Automated` after the merge.
- `Partial` is retired. A card that would have been `Partial` is split instead (see
above). A card still marked `Partial` is one that still needs the split.
- `Has automation problems` means an automation attempt found the card not automatable as
written. `Automation Notes` says which step blocks it and what the card, or Bloom, needs.
The developer who owns the card fixes that and sets `Planned` again.
Expand Down
31 changes: 28 additions & 3 deletions .github/skills/improve-test-automation-coverage/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,8 +49,11 @@ forbidden: marking a PR ready for review, moving any Orca card to Peer Review, s
## The Automation lifecycle this skill drives

`Planned` → `Building` (you, at claim time) → `PR Pending` (worker, when the draft PR exists,
with the PR URL in `Automation Notes`) → `Automated` or `Partial` (a human, after merge; a
separate sweep of `PR Pending` cards is planned). A worker that finds a case not feasible as
with the PR URL in `Automation Notes`) → `Automated` (a human, after merge; a separate sweep of
`PR Pending` cards is planned). A worker whose test covers only part of a card's steps splits the
card when it sets `PR Pending`, as `add-e2e-test` describes: the original becomes the
`[Automated portion]` and keeps its id, the uncovered steps move to a new `[Manual portion]` row.
A card is never left half automated. A worker that finds a case not feasible as
written sets it to `Has automation problems` with a dated note that says what the card, or
Bloom, needs; that is the queue for the developer who wrote the card. The developer telling the worker
that the card is not ready counts as such a finding, as much as a technical block does. Such a card is out of
Expand Down Expand Up @@ -183,7 +186,8 @@ The worker has asked you to review its test, and is blocked until you reply.
`add-e2e-test`: the title carries `[Test Case ID <id>]`; the test builds its own collection
unless a fixture is justified; the behavior under test goes through the real UI, setup may use
the API; waits are state-based; no native dialogs; helpers reused rather than re-implemented;
the covered and uncovered Test Steps match what the worker says; `AUTOMATION-DEBT.md` records
the covered and uncovered Test Steps match what the worker says, and any uncovered step means
the card was split into an automated and a manual portion; `AUTOMATION-DEBT.md` records
anything the worker could not automate cleanly. Run the `code-review` skill on the worktree
for a second opinion when the diff touches C# or the shared helpers.
2. If you want to see it run, run it yourself through the lock, from that worktree's
Expand Down Expand Up @@ -215,6 +219,27 @@ in the state the outcome implies (`PR Pending` with a PR URL; `Has automation pr
dated note; or `Planned` with a `Blocked:` note) and fix it with `notion_automation.py set` if the worker forgot. Do not delete the worktree: the
PR lives on that branch.

## Resuming a stalled run

A run stalls when the controller or a worker stops for a reason outside the work: a Claude
usage limit, a machine sleep, an Orca restart. Symptoms: `check` shows an `escalation`
"Agent exited unexpectedly", or heartbeats "rejected ... capability is revoked", and the cards
stay `Building`. A new controller can take the run over:

1. `orca orchestration worker-list --json` filtered on the run id gives every dispatch, its
task, and its worktree. `git -C <worktree> status --short` shows what the dead worker left.
Nothing is lost: the work is uncommitted in the worktree.
2. For each task whose dispatch is `failed` or `abandoned`, start a replacement in the SAME
worktree: `worker-start --task <task_id> --retry-of <old dispatch> --worktree id:<worktree id>
--agent claude --model claude-fable-5-1`. Then send the new dispatch a follow-up that says
the predecessor died, that its work is in the worktree, to read `git status` and `git diff`
first and continue from it, and where the brief file is. Restate any review fixes you had
already sent the dead worker.
3. Acknowledge the stale inbox messages, then continue Step 3 as usual.

Do not reset a card to `Planned` because its worker died; the claim and the worktree are
still good.

## Step 4 — Report

One message to the developer, in this order:
Expand Down
10 changes: 6 additions & 4 deletions .github/skills/improve-test-automation-coverage/worker-brief.md
Comment thread
hatton marked this conversation as resolved.
Original file line number Diff line number Diff line change
Expand Up @@ -94,8 +94,7 @@ Use the task id and dispatch id from the dispatch preamble at the top of your pr
### 3. Implement the test

Follow the add-e2e-test skill. Put `[Test Case ID {{TEST_CASE_ID}}]` in the test title. Prefer
`collectionSpec` (the test builds its own collection). Cover every to_do step you can; if you
cover only part, remember which part for the `Partial` note later.
`collectionSpec` (the test builds its own collection). If you cannot implement each step, that is a problem. If a substantial portion can be automated and there is a clean split, then the notion test must be split into manual vs. automated. Otherwise,  you can just fail the implementation of this test and set the Automation property to "Has automation problems".

### 4. Run it through the lock, three times

Expand Down Expand Up @@ -131,8 +130,11 @@ proceed without a `ship`.
Also say which Test Steps the test covers and which it does not, and give any change
outside `src/BloomE2E` its own **Bloom production code changes** heading, as the add-e2e-test
summary does.
2. Set the card to `PR Pending` with the PR URL in the note. If the test covers only part of the
steps, say which part in the same note:
2. If the test covers only part of the card's steps, split the card first, as `add-e2e-test`
describes: the original keeps its `Test Case ID` and becomes `<title> [Automated portion]`;
the uncovered steps move to a new `<title> [Manual portion]` row. A card is never left half
automated. Then set the card to `PR Pending` with the PR URL in the note, and say in the same
note which steps the test covers:

```powershell
py {{SKILL_DIR}}/notion_automation.py set {{TEST_CASE_ID}} "PR Pending" --note "[improve-test-automation-coverage {{TODAY}}] PR: <pr url>. Covers steps: <which>. Not covered: <which, or 'none'>."
Expand Down
2 changes: 1 addition & 1 deletion build/testing-inputs.pin
Original file line number Diff line number Diff line change
Expand Up @@ -18,4 +18,4 @@
# Format: one `key=value` per line. Blank lines and lines starting with # are ignored.

repo=https://github.com/BloomBooks/bloom-testing-inputs.git
commit=3b69cfb30892202c9564f58290fa05a00f80f2ee
commit=9b58f6be71491201b121debd184bb38ec14aa43f
40 changes: 40 additions & 0 deletions src/BloomE2E/AUTOMATION-DEBT.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,12 @@ only stable marker available. Fix direction: `data-testid="workspace-tab-collect
(etc.) on each tab and one on the shell root, and drop the label matching.
(Found 2026-09-01 while scaffolding src/BloomE2E.)

seen again 2026-09-01, in the Edit tab's page thumbnail menu: the items
`pageThumbnailList.tsx` renders carry no id, class or `data-testid` (all their styling is
inline), so `src/BloomE2E/helpers/pageThumbnails.ts` has to find "Copy Page" and "Paste Page"
by their English labels, exactly as the top bar does. Same fix: a `data-testid` per command,
taken from the `commandId` the menu already has.

## The component-tester Playwright suites are not in CI

`nightly.yml` runs vitest, C#, and visual-regression only; nothing runs
Expand Down Expand Up @@ -187,6 +193,40 @@ CKEditor does with a programmatic value change; a supported "set the text of thi
path would let long text be set at once.
(Found 2026-09-01 automating Test Case ID 169.)

## The page menu offers commands that silently do nothing while a page is loading

Copy Page and Paste Page go through `EditingModel.SaveThen`, which quietly gives up when the
editing state machine is not in Editing or NoPage (`EditingStateMachine.ToSavePending` returns
false and `CopyPage` passes `() => { }` as its "wrong state" action). The menu does not know
this: `PageThumbnailList.IsContextMenuCommandEnabled` disables commands during SavePending, but
NOT during Navigating, so while a page is still loading both commands look available and both
do nothing at all, with no error and no message. Copy Page itself then saves and reloads the
page, which reopens the same window for the very next click.

Cost, twice over. For a person: click Copy Page and then Paste Page quickly and the paste is
lost with no feedback. For a test: `src/BloomE2E/helpers/pageThumbnails.ts` has to carry
`markEditablePage` / `waitForEditablePageReload`, which stamp the page's document and wait for
Bloom to replace it, purely to know when the model has come back to Editing — the page url
cannot answer it, because Bloom reloads a page to the same in-memory url. Fix direction: make
the enabled test cover the Navigating state too, so a command that cannot run is greyed out;
or, better, queue the command instead of dropping it. Either would let the helper drop the
document-marking dance.
(Found 2026-09-01 while automating Test Case ID 348, copy page preserves everything.)

## Copying a page between two Bloom instances cannot be tested at all

The manual case "Copy Page Preserves Everything" (Test Case ID 348) ends by copying a page from
one running Bloom into a second one. Bloom's page clipboard is a pair of fields on the one
`EditingModel` instance (`_pageDivFromCopyPage`, `_bookPathFromCopyPage`), not the Windows
clipboard, so nothing crosses a process boundary; the feature is known not to work in 6.5. The
e2e fixture is also built around one Bloom per worker, so a test could not stage it today even
if the feature worked. The automated test therefore covers the within-book and between-books
cases only, so the Notion card splits: the cross-instance step belongs on a manual portion
row, per the card-splitting rule in `add-e2e-test`. Fix direction: decide whether cross-instance
copy is a feature we want; if it is, put the page on the real clipboard, and give the launch
fixture a way to run a second instance.
(Found 2026-09-01 while automating Test Case ID 348.)

## Every Bloom of one build shares one user.config, so a run inherits another Bloom's settings

Bloom keeps its user settings (UI language, page zoom, and the rest of `Settings.Default`) in
Expand Down
9 changes: 5 additions & 4 deletions src/BloomE2E/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,10 +115,11 @@ a case, put that id in the test title so the code and the inventory stay tied:
test("change UI language repeatedly [Test Case ID 69]", async ({ page }) => { ... });
```

Then set the card's `Automation` property to `Automated`, or to `Partial` when the test covers only
part of the steps. A new test with no manual card gets a new inventory row, so the inventory stays
the inventory of all tests rather than only the human-run ones. `.github/skills/add-e2e-test/SKILL.md`
has the details.
Then set the card's `Automation` property to `Automated`. If the test covers only part of the card's
steps, split the card first into an `[Automated portion]` that keeps the id and a `[Manual portion]`
with a new id, so no human-run step hides behind an automated card. A new test with no manual card
gets a new inventory row, so the inventory stays the inventory of all tests rather than only the
human-run ones. `.github/skills/add-e2e-test/SKILL.md` has the details.

## Running

Expand Down
Loading