feat: nav2 sensor fix OTA demo#58
Merged
Merged
Conversation
bburda
changed the title
bburda
force-pushed
the
feat/ota-nav2-sensor-fix
branch
from
39a0925 to
fff9581
Compare
2 tasks
bburda
changed the title
bburda
force-pushed
the
feat/ota-nav2-sensor-fix
branch
from
fff9581 to
7a37cc6
Compare
…pdate, publish-and-apply a hotfix Dev-grade OTA demo on the ros2_medkit gateway's ota_update_plugin and the SOVD /updates resource. An RB-Theron AMR runs Nav2 in an AWS small-warehouse world. A regressing lidar update (broken_lidar_3_0_0) is auto-applied at boot; a few metres into a mission the scan sensor develops a stuck sector, Nav2 can no longer make progress, and navigate_to_pose aborts. Generic log/action-status bridges surface that as SOVD faults on bt-navigator and controller-server with a freeze-frame and an MCAP capture - the sensor node never reports itself. The operator diagnoses over SOVD: downloads the MCAP, runs a suite of health-check operations (lidar/localization/drivetrain/costmap) to confirm the lidar is the cause, then publishes the forward hotfix (fixed_lidar_3_0_1) as a hand-provided update descriptor via POST /updates and applies it (prepare + execute), and clears the latched faults. A custom nav_to_pose behaviour tree keeps the abort prompt. Foxglove panels + curl scripts drive the loop; smoke tests cover the plugin, the SOVD envelope, the publish-then-apply flow, and the single-publisher /scan remap. Runs on CycloneDDS (FastDDS segfaults amcl/controller_server on Jazzy).
bburda
force-pushed
the
feat/ota-nav2-sensor-fix
branch
from
7a37cc6 to
7a85e20
Compare
The default recovery (spin, wait 5s, back-up) made a stuck robot thrash for 20-30s before navigate_to_pose aborted. Strip the recovery RoundRobin down to a costmap clear (no motion), keep the top-level retry so the goal still aborts, and shorten the progress-checker window (movement_time_allowance 10 -> 4s). The robot now drives, stalls on the phantom, and aborts in ~15-19s with no spinning.
Group the health-check app under a new 'Health Checks' function (category diagnostics) so the differential-diagnosis operations (lidar / localization / drivetrain / costmap) show up as a first-class functional grouping in the SOVD entity tree, alongside Fleet Diagnostics.
…e gz race The gz_ros2_control cold start races the controller spawners; when it loses, the robot has no odometry/TF and Nav2 aborts every goal instantly. run-demo.sh now waits for the odom->base_footprint TF to actually flow before reporting the demo up, and restarts the sim (bounded to 3 tries) if it does not - so one run-demo.sh hands off a genuinely drive-ready robot instead of leaving it to chance.
… health checks into Fleet Diagnostics The fault_manager defaults snapshots.rosbag.topics to 'entity', which narrows the captured bag to the fault source node's subgraph (bt_navigator: /odom /tf /bond /navigate_to_pose/*) and drops /scan, /cmd_vel and /local_costmap - so the downloaded MCAP had none of the diagnostic topics and Foxglove playback showed nothing. Set topics 'explicit' (write exactly include_topics) + qos_match so the capture records /scan (the phantom), /cmd_vel, /tf and the local costmap. Also add the health-check app to the Fleet Diagnostics function so its operations surface there (dropped the redundant standalone function).
…cute The descriptor shipped automated:false, which the Updates panel surfaces as 'not automated' and uses to disable the Execute button after a completed Prepare. The OTA plugin does apply the update itself on execute (kill + respawn), so it is an automated application - set automated:true. Verified: Prepare alone does NOT auto-swap (broken stays), Execute swaps to fixed - the two-step operator flow is preserved.
…abort BT Add a latched_relay node that re-publishes the latched /robot_description (URDF) and /tf_static at 2 Hz so the fault_manager's post-hoc volatile rosbag capture records them - without them a downloaded MCAP had no robot mesh and no static TF to place /scan against, so playback showed only floating obstacles. Add /robot_description to the capture include_topics. Also bake the nav2 default behavior-tree overwrite into Dockerfile.gateway (was only applied at runtime) so a clean build ships the fast-abort tree.
…hecks operation The 4 std_srvs/Trigger checks each showed a meaningless auto-generated 'structure_needs_at_least_one_member' request field and needed 4 separate clicks. Replace them with a single health_check_msgs/RunHealthChecks service: request bools lidar/localization/drivetrain/costmap (default true) select which checks to run, response returns ok / checks_run / checks_failed and a per-check report. One operator operation, meaningful request fields, richer diagnostic output; the per-check logic and message wording are unchanged.
The relay called declare_parameter('use_sim_time') but rclpy auto-declares that
parameter and the launch already sets it via -p, so the node raised
ParameterAlreadyDeclaredException and died on startup every boot. Rely on the
launch-provided value instead of re-declaring.
…in the MCAP supports_automated() returned false, so the UI's combined 'Prepare & execute' reported 'package does not support automatic updates' even though the plugin applies updates itself on execute - return true. Also capture /local_costmap/published_footprint so MCAP playback shows the robot's outline and pose; the URDF mesh cannot render on replay (Foxglove resolves package:// meshes via the live foxglove_bridge asset service, absent in a bag), so the footprint + /tf frames are how you see where the robot is in the recording.
…pose freshness AMCL only republishes /amcl_pose on a motion-triggered filter update, so once the robot reaches its goal and sits still the pose goes stale even though AMCL is still localized (it keeps broadcasting map->odom). The freshness check then reported 'AMCL pose stale/absent' after the OTA fix, collapsing the whole post-fix 'all healthy' result on a false positive. Judge the cached pose's xy covariance instead - small = converged, regardless of age; a real divergence still trips the bound.
… medkit workspace The gateway image ran a bare 'colcon build' over the full 15-package ros2_medkit clone - including ros2_medkit_opcua (open62541pp, very slow to compile), integration_tests, and the discovery beacons the demo never loads (it only loads ota_update_plugin). Restrict the build to --packages-up-to the four medkit packages the demo actually runs (gateway, fault_manager, log/action-status bridges) plus the demo's own packages, so their deps come in transitively and everything else is skipped.
There was a problem hiding this comment.
Pull request overview
This PR adds a new end-to-end “OTA over SOVD” demo (demos/ota_nav2_sensor_fix/) that exercises the /updates lifecycle against a live ros2_medkit gateway, including a dev-grade OTA plugin, an artifact server, and ROS 2 demo packages used to demonstrate a forward hotfix of a regressed lidar node.
Changes:
- Introduces a complete OTA demo stack (gateway image + C++ OTA plugin + FastAPI artifact server + ROS 2 demo packages + operator scripts).
- Adds automated validation (new OTA smoke tests + CI jobs) to cover
/updatesbehavior, spec-shaped payloads, and process swapping. - Documents the new demo in the repository README and includes third-party notice pinning for build-fetched assets.
Reviewed changes
Copilot reviewed 75 out of 77 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
tests/smoke_test_ota.sh |
New host-driven smoke test for the OTA demo’s /updates and process swap flow |
tests/smoke_lib.sh |
Makes smoke harness exit code reflect recorded failures (and preserve crash exit codes) |
README.md |
Adds the new OTA demo and includes it in the smoke test list |
demos/ota_nav2_sensor_fix/updates/README.md |
Documents manual publishing of updates via POST /updates |
demos/ota_nav2_sensor_fix/updates/fixed_lidar_3_0_1.json |
Ships the hotfix update descriptor used by publish/apply scripts |
demos/ota_nav2_sensor_fix/trigger-bad-update.sh |
Script to re-apply the regressing update via prepare/execute |
demos/ota_nav2_sensor_fix/THIRD_PARTY_NOTICES.md |
Documents pinned upstream assets fetched at image build time |
demos/ota_nav2_sensor_fix/stop-demo.sh |
Demo teardown helper (compose down + optional cleanup) |
demos/ota_nav2_sensor_fix/send-goal.sh |
Helper to publish a Nav2 goal inside the gateway container |
demos/ota_nav2_sensor_fix/scripts/test_pack_artifact.py |
Pytest coverage for pack_artifact.py |
demos/ota_nav2_sensor_fix/scripts/pyrightconfig.json |
Pyright config for demo scripts |
demos/ota_nav2_sensor_fix/scripts/pack_artifact.py |
CLI to build tarballs + SOVD-shaped catalog/update entries |
demos/ota_nav2_sensor_fix/scripts/e2e_webui_smoke.mjs |
Optional Playwright-based E2E smoke driver for the web UI flow |
demos/ota_nav2_sensor_fix/scripts/conftest.py |
Pytest path setup for local script tests |
demos/ota_nav2_sensor_fix/scripts/build_artifacts.sh |
Optional host-side artifact build helper for maintainers |
demos/ota_nav2_sensor_fix/scripts/.gitignore |
Ignores local Python/pytest artifacts for scripts |
demos/ota_nav2_sensor_fix/run-demo.sh |
Main runner for building/starting the demo stack |
demos/ota_nav2_sensor_fix/ros2_ws/.gitignore |
Ignores ROS 2 workspace build/install/log trees |
demos/ota_nav2_sensor_fix/ros2_packages/ota_nav2_sensor_fix_demo/urdf/warehouse_rbtheron.urdf.xacro |
RB-Theron URDF wrapper to avoid closed controller config and keep sensors |
demos/ota_nav2_sensor_fix/ros2_packages/ota_nav2_sensor_fix_demo/package.xml |
Package metadata for the demo launch/config glue |
demos/ota_nav2_sensor_fix/ros2_packages/ota_nav2_sensor_fix_demo/models/person_obstacle/model.sdf |
Simple box obstacle model used in the simulation narrative |
demos/ota_nav2_sensor_fix/ros2_packages/ota_nav2_sensor_fix_demo/models/person_obstacle/model.config |
Model config for person_obstacle |
demos/ota_nav2_sensor_fix/ros2_packages/ota_nav2_sensor_fix_demo/models/person_human/model.sdf |
Human-like obstacle model built from primitives (mesh-free) |
demos/ota_nav2_sensor_fix/ros2_packages/ota_nav2_sensor_fix_demo/models/person_human/model.config |
Model config for person_human |
demos/ota_nav2_sensor_fix/ros2_packages/ota_nav2_sensor_fix_demo/models/aws_small_warehouse/worlds/warehouse.sdf |
gz-port world file referencing AWS warehouse models |
demos/ota_nav2_sensor_fix/ros2_packages/ota_nav2_sensor_fix_demo/config/warehouse_map.yaml |
Map server YAML for the warehouse environment |
demos/ota_nav2_sensor_fix/ros2_packages/ota_nav2_sensor_fix_demo/config/ros_gz_bridge.yaml |
Bridge config, including /scan_sim mapping to avoid /scan stomping |
demos/ota_nav2_sensor_fix/ros2_packages/ota_nav2_sensor_fix_demo/config/navigate_to_pose_fast_abort.xml |
Nav2 BT tuned to abort faster for the demo narrative |
demos/ota_nav2_sensor_fix/ros2_packages/ota_nav2_sensor_fix_demo/config/nav2_params.yaml |
Nav2 params tailored for the RB-Theron + warehouse scenario |
demos/ota_nav2_sensor_fix/ros2_packages/ota_nav2_sensor_fix_demo/config/controllers.yaml |
ros2_control controller config for diff-drive + joint states |
demos/ota_nav2_sensor_fix/ros2_packages/ota_nav2_sensor_fix_demo/CMakeLists.txt |
Installs launch/config/URDF/models/maps into the package share dir |
demos/ota_nav2_sensor_fix/ros2_packages/health_check/src/health_check_node.cpp |
Operator-invoked Trigger services for differential diagnosis checks |
demos/ota_nav2_sensor_fix/ros2_packages/health_check/package.xml |
health_check package manifest |
demos/ota_nav2_sensor_fix/ros2_packages/health_check/CMakeLists.txt |
Builds/installs health_check_node |
demos/ota_nav2_sensor_fix/ros2_packages/fixed_lidar/src/fixed_lidar_node.cpp |
Post-fix clean /scan_sim → /scan republisher |
demos/ota_nav2_sensor_fix/ros2_packages/fixed_lidar/package.xml |
fixed_lidar package manifest |
demos/ota_nav2_sensor_fix/ros2_packages/fixed_lidar/CMakeLists.txt |
Builds/installs fixed_lidar_node |
demos/ota_nav2_sensor_fix/ros2_packages/broken_lidar/src/broken_lidar_node.cpp |
Regressed scan node that overlays a phantom sector after a driven distance |
demos/ota_nav2_sensor_fix/ros2_packages/broken_lidar/package.xml |
broken_lidar package manifest |
demos/ota_nav2_sensor_fix/ros2_packages/broken_lidar/CMakeLists.txt |
Builds/installs broken_lidar_node |
demos/ota_nav2_sensor_fix/README.md |
Full demo narrative + curl equivalents + Foxglove walkthrough |
demos/ota_nav2_sensor_fix/publish-fix.sh |
Registers the hotfix via POST /updates |
demos/ota_nav2_sensor_fix/ota_update_server/tests/test_main.py |
Tests for update server catalog + artifact endpoints |
demos/ota_nav2_sensor_fix/ota_update_server/tests/__init__.py |
Marks test package |
demos/ota_nav2_sensor_fix/ota_update_server/pyrightconfig.json |
Pyright config for the update server |
demos/ota_nav2_sensor_fix/ota_update_server/pyproject.toml |
Python packaging for the update server |
demos/ota_nav2_sensor_fix/ota_update_server/ota_update_server/main.py |
FastAPI app serving /catalog and /artifacts/* |
demos/ota_nav2_sensor_fix/ota_update_server/ota_update_server/__init__.py |
Exposes create_app |
demos/ota_nav2_sensor_fix/ota_update_server/Dockerfile |
Multi-stage build: builds demo artifacts, then serves via FastAPI |
demos/ota_nav2_sensor_fix/ota_update_server/.gitignore |
Ignores venv/build artifacts |
demos/ota_nav2_sensor_fix/ota_update_plugin/test/test_plugin_smoke.cpp |
GTest suite validating plugin behaviors and ProcessRunner spawn failure |
demos/ota_nav2_sensor_fix/ota_update_plugin/test/test_operation_dispatcher.cpp |
Tests operation-kind classification from metadata component arrays |
demos/ota_nav2_sensor_fix/ota_update_plugin/test/test_catalog_client.cpp |
Tests URL parsing logic for the CatalogClient |
demos/ota_nav2_sensor_fix/ota_update_plugin/src/process_runner.hpp |
Declares process-management seam used by the plugin |
demos/ota_nav2_sensor_fix/ota_update_plugin/src/process_runner.cpp |
Implements pgrep/kill/spawn with cmdline-based matching and double-fork |
demos/ota_nav2_sensor_fix/ota_update_plugin/src/plugin_exports.cpp |
Gateway plugin entrypoints + UpdateProvider accessor |
demos/ota_nav2_sensor_fix/ota_update_plugin/src/operation_dispatcher.hpp |
Defines operation-kind classification API |
demos/ota_nav2_sensor_fix/ota_update_plugin/src/operation_dispatcher.cpp |
Implements operation-kind classification |
demos/ota_nav2_sensor_fix/ota_update_plugin/src/catalog_client.hpp |
Declares catalog/artifact HTTP client interface |
demos/ota_nav2_sensor_fix/ota_update_plugin/src/catalog_client.cpp |
Implements HTTP GET for /catalog and artifact downloads |
demos/ota_nav2_sensor_fix/ota_update_plugin/package.xml |
Plugin package manifest |
demos/ota_nav2_sensor_fix/ota_update_plugin/include/ota_update_plugin/ota_update_plugin.hpp |
Plugin class interface (GatewayPlugin + UpdateProvider) |
demos/ota_nav2_sensor_fix/ota_update_plugin/CMakeLists.txt |
Builds static core lib + dlopen-loaded module + tests |
demos/ota_nav2_sensor_fix/manifest.yaml |
Demo manifest defining components/apps/functions (hybrid discovery) |
demos/ota_nav2_sensor_fix/gateway_config.yaml |
Gateway config enabling updates and loading ota_update_plugin |
demos/ota_nav2_sensor_fix/entrypoint.sh |
Launches demo and auto-applies the bad update at boot |
demos/ota_nav2_sensor_fix/Dockerfile.gateway |
Builds the full gateway + Nav2 + gz-sim + demo packages image |
demos/ota_nav2_sensor_fix/docker-compose.yml |
Two-service stack: gateway + update server |
demos/ota_nav2_sensor_fix/clear-fault.sh |
Clears the latched faults after applying the hotfix |
demos/ota_nav2_sensor_fix/check-demo.sh |
Displays current update/fault/process state for the operator |
demos/ota_nav2_sensor_fix/artifacts/.gitignore |
Ignores locally built artifacts |
demos/ota_nav2_sensor_fix/apply-fix.sh |
Applies the published fix via prepare/execute |
.github/workflows/ci.yml |
Adds CI jobs to build/run the OTA demo and execute smoke tests |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
smoke_test_ota.sh asserted broken_lidar_node was the live scan_sensor_node with a one-shot pgrep, but the entrypoint applies broken_lidar_3_0_0 asynchronously after the catalog registers, so the check raced the async apply and flaked. Gate on broken_lidar_3_0_0 reaching 'completed' before the "Initial process state" section, and poll the process assertions instead of checking once (matches the boot gate smoke_test_demo_narrative.sh already uses).
The gateway container multiplexes ~15 nodes into one stdout, so 'docker compose logs gateway --tail=N' on failure showed only discovery chatter and never the nav2 goal handling. Add tests/dump_ota_diagnostics.sh and run it from both OTA jobs on failure: it captures the full compose log plus ROS introspection (lifecycle, /navigate_to_pose server, /scan, /amcl_pose, TF, controllers, faults) and uploads it as an artifact.
… publish send-goal.sh published /goal_pose once with 'ros2 topic pub --once' from a transient docker exec node. On a cold CI runner that single volatile publish raced DDS discovery and was dropped before bt_navigator matched the subscription, so nav2 never drove, never aborted, and the reactive-fault scenario raised nothing (ota-demo-narrative timed out waiting for ACTION_NAVIGATE_TO_POSE_ABORTED). Send through the /navigate_to_pose action instead: the client waits for the server and confirms the goal is accepted, guaranteeing delivery, and returns on acceptance so a healthy-resume goal does not block on the drive. Update the README/run-demo help to match.
Sending via the action exposed a second cold-start race: bt_navigator is lifecycle-active and its action server is discoverable before it can actually accept a goal (amcl has not published map->odom yet, so it has no pose to plan from), so it rejects the goal. A goal fired right after boot was rejected once and dropped. Retry send_goal_async until the goal is accepted (up to 90s), so the reactive-fault step reliably starts the mission on a cold CI runner.
mfaferek93
reviewed
Jul 7, 2026
- pack_artifact.py: --version defaults to '' so an omitted version fails loud for update/install instead of silently packing a 0.0.0 artifact. - process_runner: correct the pgrep doc to cmdline argv0 (not comm, which the kernel truncates to 15 chars). - docker-compose: block privilege escalation on the gateway (no-new-privileges); cap_drop / non-root USER are documented as production hardening, not applied here (they risk breaking gz-sim / DDS in this all-in-one demo container).
…e-then-swap) - Reject catalog ids/components/executables/target packages that are not [A-Za-z0-9._-] before they reach a path or exec (blocks '../' traversal and argument injection). - Extract the artifact with fork+execvp instead of std::system, so a catalog-controlled name can never be interpreted by a shell. - Stage-then-swap: extract and verify the new tree before killing the running node, so a bad tarball no longer leaves the sensor with no lidar. - Track processes the plugin spawned (keyed by component) and also kill the recorded pid on re-apply, guarded by a /proc/<pid>/cmdline argv0 check against pid reuse - fixes the duplicate scan_sensor_node on a repeated apply. - Uninstall kills the recorded executable (not the package name, which never matched argv0) and removes the install fragment keyed on the component, so an uninstalled app no longer lingers in the entity tree. - Emit install-fragment scalars as quoted YAML. - Rewrite the process-runner test stub into a stateful fake covering the swap, duplicate-spawn, uninstall, traversal-rejection and quoting paths. - README: document the demo's trusted-local security model and what a production deployment would add; correct the swap note.
…n CI The plugin image built with -DBUILD_TESTING=OFF and no job ran colcon test or pytest, so the plugin gtest, the ota_update_server tests and the pack_artifact tests never executed anywhere. Add a Dockerfile 'ota-plugin-test' stage that rebuilds the plugin with BUILD_TESTING=ON and runs its gtest, and a 'plugin-tests' CI job that builds that stage and runs both pytest suites.
The healthy-resume step failed intermittently: resuming from the tight spot the robot stops in after the broken-phase recovery, nav2's first replan can log one transient 'Failed to make progress' that the fault_manager latches, even though the fixed lidar is clean. Give the robot time to get moving, clear that one-off controller fault, then assert none reappears. A real sensor regression keeps the controller failing (and the navigate_to_pose abort check already guards that), so a bad fixed_lidar would put the fault straight back after the clear.
mfaferek93
approved these changes
Jul 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds
demos/ota_nav2_sensor_fix/- an end-to-end OTA-over-SOVD demo (gateway plugin + FastAPI artifact server + 4 ROS 2 demo packages) exercising the/updateslifecycle: update a broken lidar node, install a new safety classifier, and uninstall a deprecated package, all over HTTP and spec-compliant.OTA demo
demos/ota_nav2_sensor_fix/end-to-end OTA demoota_update_pluginC++ gateway plugin (UpdateProvider+GatewayPlugin)updated_components/added_components/removed_components)pack_artifact.pyCLI for building tarballs and catalog entriesdocker-compose.yml(gateway + update server); nav2 / Foxglove are bring-your-own (documented in README)Out of scope (deliberate, dev-grade positioning)
Test plan / verification
Unit & integration tests (all clean):
pytest -vforpack_artifact.py(16 tests)pytest -vforota_update_server(5 tests)colcon testforota_update_plugin(24 GTest cases)-Wall -Wextra -Wpedantic -Wshadow -Wconversionbuild_artifacts.shproduces a 3-entry catalog + tarballs end-to-endEnd-to-end smoke:
UpdateProvider(gateway logs: "Update backend provided by plugin")/catalogand registers all 3 catalog entries/updates/fixed_lidar_2_1_0/prepare && /executekillsbroken_lidar_nodeand spawnsfixed_lidar_node/updates/obstacle_classifier_v2_1_0_0/prepare && /executeswaps files and spawnsobstacle_classifier_node/updates/broken_lidar_legacy_remove/prepare && /executereturnsstatus: completedand the legacy process is gonetests/smoke_test_ota.sh- 25/25 pass on a fresh stacktests/smoke_test_demo_narrative.sh- 8/8 pass on a fresh stacktrigger-update.sh,trigger-install.sh,trigger-uninstall.sh,check-demo.sh,stop-demo.shexercised end-to-endNotes
selfpatch/ros2_medkitmainfor the gateway sources (clone happens at image build time)pgrepmatches against/proc/<pid>/cmdlineargv[0] basename (notcomm, which the kernel truncates to 15 chars)artifacts/is baked into theupdate_serverimage at build timeros2_medkit_foxglove_extensionUpdates panel PR (fix(docker): add missing ros2_medkit components and submodules #6 of that repo)