feat(cli): doctor diagnostics commands - #3252
Draft
benallfree wants to merge 61 commits into
Draft
Conversation
…re verify (cherry picked from commit 07b20f0)
(cherry picked from commit 6b4f235)
…/serve) (cherry picked from commit 5da7830)
…otloader) (cherry picked from commit 9b61169)
(cherry picked from commit 9beaf68)
(cherry picked from commit 543c901)
… tooling (.mota reference, EndF/target/vector generators) (cherry picked from commit 4680d10)
…pect/serve/keygen) (cherry picked from commit 2092500)
(cherry picked from commit e3dd6b4)
(cherry picked from commit 54c8b58)
…eeder/console) Extends the USB-serial folder relay to WiFi so an ESP32 companion can both serve .mota and be operated headlessly: - motatool `serve --tcp <host[:port]>`: a TcpTransport sibling of the serial transport (default port 5001). SeederCore/Folder are reused unchanged — the COUNT/DESCRIBE/READ protocol is transport-agnostic. - ESP32 companion: a dedicated OTA seeder port (5001) for `serve --tcp`, plus an OTA text console on 5002 (`nc <ip> 5002` -> `ota status|ls|announce|...`, the same handle_ota_command CLI serial nodes have). Both run alongside the phone-app port (5000); all three coexist. - WiFi.setSleep(false): ESP32 STA mode's modem power-save periodically sleeps the modem/CPU and stalls the SX1262 SPI+DIO servicing, leaving LoRa deaf while WiFi is associated. Disabling it restores the radio (HW-validated: a V3 WiFi companion is then discovered over LoRa and discovers its peers). - docs: serving .mota over WiFi (protocol §10.2 + user guide). (cherry picked from commit 280b9b1)
Discovery was hard to use — a node only advertised at boot, so a peer that ran `ota ls` minutes later saw "no neighbours". - First self-advert ~8s after boot, then a short burst (~1 min), then re-announce at a random 3-10 min interval so a long-running node stays discoverable without all nodes beaconing in lockstep. The beacon is tiny, lowest-priority and duty-gated, so a few-minute cadence is cheap. - `ota ls` now shows the raw target id (`hw XXXXXXXX`) when the env name is not in this build's OtaTargets.h table, instead of a blank "[other hw]". (cherry picked from commit 4cb118b)
Switch these existing variants from `nrf52_base` to `rak4631_hw` (which adds ENABLE_OTA + the in-place flash store + the EndF post-build hook), so they get OTA-over-LoRa. Scope is limited to variants already covered by the Adafruit_nRF52_Bootloader_OTAFIX in-place apply — no new variants are added. OtaTargets.h is regenerated to include their target ids. (cherry picked from commit 0a1cf2b)
…ved-set change The discovery beacon previously re-announced at a random 3-10 min interval. Replace that with a fixed, user-configurable cadence: - OtaManager::advert_mins() — re-advertise every N minutes after the boot burst; 0 disables periodic re-advertise (boot burst only). Default 24h. - Persisted in NodePrefs (CommonCLI) and runtime-tunable: `ota config advert <minutes>` (0..10080; 0 = off), and shown in `ota config`. - When periodic advert is disabled, the scheduler still re-checks the config on a slow timer, so a later `ota config advert <mins>` takes effect live. Also advertise immediately whenever the served set changes — when a motatool folder is attached to / detached from the ESP32 WiFi seeder — so peers learn about newly-available firmware without waiting for the next interval (the `ota folder` serial path already announced on attach). Docs: protocol beacon-cadence note + user-guide `ota config advert`. (cherry picked from commit 9ff1c88)
…rd RAM guard Bound OTA-over-LoRa duty cycle across repeaters with one runtime-tunable, persisted limit (OtaManager::max_hops, `ota config hops <0..8>`, default 3): - Accept-gate: a node ignores OTA that arrived from more than max_hops hops away (neither processes nor relays it). 0 = direct only. - Forward-cap: relay a flood only while still under max_hops, appending this node's path-hash (hop count increments like the mesh flood routing). - RAM guard: relay an OTA flood only while more than OTA_FWD_MIN_FREE packet- pool slots stay free, so heavy OTA (best-effort, lowest-priority) can never monopolise the shared pool and starve real traffic — a dropped relay is re-requested by the source. Persisted in NodePrefs (CommonCLI) and shown in `ota config`. Docs updated. (cherry picked from commit 259802b)
…e handlers Add the device->host WRITE half of the mota-seeder link so a device can capture a .mota it is fetching off-mesh into the host folder (the same --dir used for serving), stored as <mid>.mota — e.g. to grab an exact copy of a device's firmware to build a delta against firmware you don't otherwise have. - MotaSeederProto: OP_STAT / OP_BEGIN / OP_WRITE / OP_SREAD / OP_FIN (keyed by mid). Resume needs no host bookkeeping: BEGIN 0xFF-fills the file; on reconnect the device SREADs the leaves and re-requests only missing blocks (same as flash resume). Partial = <midhex>.mota.part, published to <midhex>.mota on FIN. - motatool `serve --dir <folder>` now also handles the storage ops on the same folder/connection (SeederCore gains the store dir; serve_loop frames the variable-length WRITE). Host round-trip test added. Firmware side (FolderMotaStore + `ota pull <#> <dest>` + pause/resume) follows. (cherry picked from commit bd2ecb8)
…firmware side FolderMotaStore: an OtaStore that streams an in-transit .mota straight to the host folder over the seeder link (OP_BEGIN/WRITE/SREAD/STAT/FIN) instead of RAM/flash — the device holds no local staging for it. Wired as a selectable pull destination: - OtaContext gains a folder_dest (registered by the app while a motatool `serve` link is up) + its human id (e.g. "tcp 192.168.4.5"). - `ota pull <#> <dest>` now takes a MANDATORY destination; `ota pull <#>` with none lists the choices (flash always; folder + its link id when connected). - The ESP32 WiFi seeder registers/clears the folder destination on connect/close — the same connection both serves the folder and accepts pulls into it. Captures an exact copy of a device's firmware over the mesh to build a delta against. Pause/resume on a mid-pull disconnect follows. (cherry picked from commit 10fa434)
If an `ota pull <#> folder` block-write fails mid-transfer (the motatool seeder link dropped), the fetch enters a new PAUSED state instead of failing or falling back to RAM/flash: progress stays on the host, the manager stops requesting, and loop()/stall-detection leave it untouched (it waits indefinitely). On reconnect the ESP32 seeder re-registers the folder destination and, if PAUSED, calls resumeStaged(): OP_STAT re-attaches the host's partial, the leaves are re-read, and only the missing blocks are re-requested (a brand-new/absent file restarts from 0). `ota status` reports the paused state. (cherry picked from commit 28e3df0)
Document `ota pull <#> <dest>` (flash|folder, destination mandatory), the folder pull that captures a device's exact firmware to the host as <mid>.mota (for delta-building), the paused/resume-on-reconnect behaviour, and that a `motatool serve` link doubles as the pull-to-folder store. (protocol §10, user guide, motatool README) (cherry picked from commit 49820b5)
Route both begin() and reopen() through a pure mota_nrf52_stage_plan() that bottom-aligns a received .mota below the filesystem region (trailer ending at FS_START, where the bootloader scans) and refuses any size that would overrun that ceiling or overlap the running image — the single place the FS/prefs-safe bounds are enforced. Add compile-time static_asserts pinning the flash-layout ordering (app < staging ceiling < bootloader; in-place workspace ends at/below the ceiling) so an inconsistent constant fails the build instead of silently corrupting user prefs. Cover the pure planner in test/test_ota/test_ota_flashplan.cpp. (cherry picked from commit 4a60148)
…-diff capture Protocol: OTA_REQ and OTA_GET_MANIFEST now carry a want_mask bitmap, so a fetcher asks for specific fragments (all on the first request, only the still-missing holes on a retry) instead of a whole block/manifest window. The WANT_MANIFEST and FETCHING retry loops re-ask only on a no-progress tick, so a lost fragment costs one fragment to recover and a re-request can't collide with an in-flight multi-fragment burst on half-duplex radios. Warm-start (motatool folder-capture only): new OTA_GET_LEAVES/OTA_LEAVES let `ota pull <#> folder validate` bulk-fetch the target's merkle leaves, authenticate them against the manifest root, diff a similar seed build already staged in the destination, and pull DATA over LoRa only for the blocks that differ. Leaves are bitmap-fragmented + no-progress retry-gated like the manifest, capped at OTA_DIFF_MAX_BLOCKS so the want_mask stays a fixed uint16, and the diff runs a bounded batch per loop tick so it never starves the mesh loop. motatool `serve --seed <build.mota>` injects the seed payload into the destination .part on OP_BEGIN. (cherry picked from commit 8953214)
Add an admin-only `ota stats` reply: one dense line with the running firmware's merkle content-id (mid) AND its EndF body_hash (only body_hash was surfaced before), version, served-set count + digest, live fetch state/progress, and policy — snprintf-bounded to the 160-byte reply. The remote CLI path is already admin-gated, so it's admin-only over the mesh (send it from the app's repeater command screen, or the WiFi/serial OTA console). A new servedDigest() accessor exposes the beacon set-digest. HW-verified on RAK4631: reports the fw identity + live fetch state (incl. during a warm-start capture). (cherry picked from commit b159737)
The EndF post-build stamper read the version via _cppdef('FIRMWARE_VERSION'), which only sees -D build flags — but MeshCore authors FIRMWARE_VERSION as a header #define in each example, so the stamper found nothing and defaulted fw_version to 0. Every .mota / OTA advert then reported v0.0.0 while `ver` (which reads the header at runtime) showed the real version.
Read the header value as a fallback WITHOUT moving where MeshCore authors it: honor a -D override first (the header's #ifndef guard invites it, for release builds), else read the #define from the example this env builds (via build_src_filter), falling back to the repo-wide value when unambiguous. Purely additive to our own EndF tooling — no MeshCore source changed.
HW-verified: RAK4631 now reports `fw v1.17.0` in ota stats/ls (was v0.0.0), matching `ver`.
(cherry picked from commit 0319a81)
… switch) Spell out how the warm-start seed is actually applied (a user asked): the seed is the --seed <file> given to `motatool serve`, NOT a file dropped into the --dir destination (which starts empty). motatool stamps that one seed into the fresh .part on every `… folder validate` begin, so it's always the named file — no guessing. `validate` is the switch (a plain folder pull fetches from 0); a re-pull re-begins fresh (never resumes a stale partial); a mismatched/absent seed just falls back to fetching those blocks over the radio (correct result, only slower). (cherry picked from commit cbdc629)
The T096 is an nRF52840 with the same S140 v6 SoftDevice and flash geometry as the RAK4631, so it can reuse the shared rak4631_hw OTA recipe (ENABLE_OTA + OtaFlashLayout_nrf52.h — APP_BASE 0x26000, staging ceiling 0xD4000, all static_asserts satisfied by the stock nrf52840_s140_v6[_extrafs] ldscripts). With the OTAFIX bootloader (oltaco PR meshcore-dev#42 adds the T096 board) a node applies in place; without it the node still discovers/relays/serves OTA and merely fetch-refuses apply — so a T096 is a useful OTA relay/seeder regardless. MOTA_HW_ID='"Heltec_t096"' tags the EndF trailer so a .mota built for other hardware is refused. Propagates to every T096 role via ${Heltec_t096.*}. Verified: repeater + companion_radio_usb build clean, 14 OTA objects each, EndF stamped (hw='Heltec_t096', v1.17.0). (cherry picked from commit ee6e548)
…ool) The host-side build/verify/inspect/serve CLI now lives as a standalone Rust project at https://github.com/vk496/motatool. Remove the in-tree C++ copy (tools/motatool/) and repoint the docs (ota_protocol.md, ota_user_guide.md, tools/mota/README.md) at the standalone repo. No cross-dependency either direction: MeshCore's firmware build never invoked motatool (only tools/mota/ Python glue runs in the build), and motatool depends only on the shared .mota wire spec in docs/ota_protocol.md. tools/mota/ (pio_endf.py EndF hook, motalib.py reference lib, gen_targets.py) is unchanged. (cherry picked from commit 8724132)
Co-authored-by: Cursor <cursoragent@cursor.com> (cherry picked from commit 7184185)
Co-authored-by: Cursor <cursoragent@cursor.com> (cherry picked from commit dcf051f)
Co-authored-by: Cursor <cursoragent@cursor.com> (cherry picked from commit 97470f4)
Co-authored-by: Cursor <cursoragent@cursor.com> (cherry picked from commit 18be8ac)
Co-authored-by: Cursor <cursoragent@cursor.com> (cherry picked from commit 5116d0b)
No OLED or external sensor drivers (built-in MCU temperature only). Extends the sensor-free rak4631_hw OTA recipe: 434,036 B flash vs 572,336 B for the full repeater (~135 KB saved), growing .mota staging headroom below MOTA_STAGE_CEILING from ~237 KB to ~372 KB. Co-authored-by: Cursor <cursoragent@cursor.com> (cherry picked from commit beeef28)
Gates the Bluefruit BLE DFU fallback (`start ota`) behind BLE_DFU_DISABLED and sets it on RAK_4631_repeater_slim: the BLE stack drops out entirely (434,036 -> 388,656 B, ~45 KB), raising staging headroom to ~416 KB -- enough to stage a full slim .mota. Co-authored-by: Cursor <cursoragent@cursor.com> (cherry picked from commit 31621a3)
Co-authored-by: Cursor <cursoragent@cursor.com> (cherry picked from commit 705724b)
Co-authored-by: Cursor <cursoragent@cursor.com> (cherry picked from commit 376a9d3)
Co-authored-by: Cursor <cursoragent@cursor.com> (cherry picked from commit 2e7d5a6)
Co-authored-by: Cursor <cursoragent@cursor.com> (cherry picked from commit 899ba16)
Co-authored-by: Cursor <cursoragent@cursor.com> (cherry picked from commit 0e70236c7d2ec5be2def6ee782222ee96b2618c8)
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
benallfree
force-pushed
the
feature/doctor
branch
from
August 20, 2026 05:06
da56835 to
448cf52
Compare
benallfree
changed the title
benallfree
force-pushed
the
feature/doctor
branch
from
August 20, 2026 05:33
e103ff1 to
3f1ce22
Compare
Contributor
Author
|
@dotspencer Curious if you have any feedback on the |
Remote admin CLI no longer runs handleCommand inside onPeerDataRecv. Commands queue to MyMesh::loop() so savePrefs and ed25519_sign run at loop-task stack depth, matching USB serial CLI. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
EC-001: MeshCore companion-v1.17 upgrade (freshen overlay + upstream base). Resolve defer-remote-cli init: keep main _remote_cli memset + v1.17 recv_pkt_region. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
benallfree
force-pushed
the
feature/doctor
branch
from
August 25, 2026 08:38
3f1ce22 to
295b478
Compare
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
benallfree
force-pushed
the
feature/doctor
branch
from
August 25, 2026 08:44
295b478 to
b5e4181
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds a
doctorCLI command for repeaters to perform on-device diagnostics. I was hitting LittleFS errors and needed more insight.doctor statFS_STATon serial)doctor gc/packet_log, legacy prefs paths, doctor temp files)doctor checkdoctor lsFS_LSon serial)doctor probeFS_PROBEon serial)doctor dumpAlso appends
(try: doctor gc)hints to the FS save error replies, and upgradeseraseto format+remount when the FS callback is available.Stacked on #3253 (FS error replies), which stacks on #3254 (atomic saves) — merge those first, then rebase this onto
dev. The first two commits here are theirs.