🤖 AgentNav: Agentic Robot Navigation Framework

English | 中文

"Stop hardcoding, start conversing."
AgentNav lets a multimodal AI agent (Claude, Gemini, GPT) drive a real robot through natural language and its own eyes — no waypoint predictor, no policy training, no hardcoded pipeline.

---

💡 Why Agentic Navigation?

A conventional navigation stack is a black box: one goal in, success or failure out. The agent cannot see why a move failed or how to recover.

AgentNav turns navigation into a conversation between an AI brain and a robot body — but the interesting part is the layer in between. The agent supplies semantics; a deterministic Harness supplies geometry, safety and memory. Neither is asked to do the other's job.

	Traditional navigation	🤖 AgentNav
Goal	Predefined coordinates	Point at a pixel, or say what you want
Perception	Separate detector / VLM	The agent looks at the picture itself
Distance	Dense depth map, or nothing	On-demand pixel probes with quality flags
Safety	Costmap only	Swept-corridor check before every move
Progress	Agent is blind until timeout	Stall and divergence surfaced in seconds
Memory	None	BEV map, on-demand recall, persistent facts
Arrival	Planner says "succeeded"	Explicit visual verification
Logic	C++/Python pipelines	Markdown skills, edited like a prompt

---

🚀 Key Features

• 👀 Native multimodal vision. robot_observe() returns real images with a pixel grid overlay. Every image carries a frame_id, and all reasoning about it resolves against that exact snapshot — depth, intrinsics, extrinsics and pose included.
• 📏 On-demand depth. query_depth() probes specific pixels and reports distance, a 5×5 surface-quality window, an action preview and whether the path is clear. Comparing three candidates beats guessing from an image.
• 🛡️ Safety in the loop. move_to_pixel() runs a swept-corridor check before dispatching anything. A blocked path comes back as reselect — a normal answer the agent can act on, not a collision.
• 🧠 Memory that pays for itself. A bird's-eye map every step, recall(step) for detail on demand, and a persistent fact store so the second trip to the kitchen is a straight line.
• ✅ Verified arrival. verify_arrival() makes the agent look before it claims success. "The planner returned SUCCEEDED" is not evidence.
• 🔌 Pluggable motion. Ships driving out of the box: the default teleop backend needs no map and no extra install. Switch to Nav2 for global planning, or add your own policy by implementing BaseS1Client.
• 🧩 Declarative drivers. Each driver declares what it needs; unmet requirements mean the tool is never exposed, and robot_status() explains why.

---

🏗️ Architecture

flowchart TB
    User(["User · Telegram / CLI"])

    subgraph L3["Cognition — nanobot Agent OS"]
        Main["Main agent · semantic planning<br/>skills: navigate · explore · recover"]
        Runner["Skill runner · isolated context<br/>local search, retries, recovery"]
    end

    subgraph L2["Harness — agentnav MCP (deterministic, no LLM)"]
        Obs["Perception<br/>robot_observe · query_depth · nav_map · recall"]
        Act["Motion<br/>move_to_pixel · move_to_pose · move_to_waypoint"]
        Safe["Safety<br/>swept corridor · limits · robot_stop"]
        Mem["Memory<br/>FrameCache · BEV map · FactStore"]
        Ver["Verification<br/>stall detection · verify_arrival"]
    end

    subgraph L1["Execution — pluggable S1"]
        Teleop["TeleopClient — default<br/>needs nothing"]
        Nav2["Nav2Client<br/>needs Nav2 + a map"]
    end

    subgraph HW["ROS2 Humble"]
        Topics["/camera/color · /camera/depth · /camera_info<br/>/odom · /tf · /cmd_vel"]
    end

    User <--> Main
    Main --> Runner
    Runner <-->|"MCP over stdio"| Obs
    Runner <-->|"MCP over stdio"| Act
    Obs --> Mem
    Act --> Safe
    Ver --> Obs
    Safe --> Teleop & Nav2
    Teleop & Nav2 <--> Topics
    Obs <--> Topics

Loading

The rule that shapes everything: semantics belong to the agent, geometry belongs to the Harness. Nothing requiring trigonometry, a coordinate transform or a safety threshold goes into a prompt.

---

🔧 Tool Reference

📸 Perception

Tool	What it does	Safety
robot_observe(views, annotate)	Look around. views=4 turns in place for four angles. Returns frame_id + image.	✅ Safe
query_depth(frame_id, points)	Probe up to 8 pixels: distance, 5×5 stats, action preview, reachability.	✅ Safe
nav_map(waypoints)	Bird's-eye view: trajectory, obstacles, frontiers, remembered places.	✅ Safe
recall(step, view)	Pull one earlier view back into context.	✅ Safe
clearance_report()	Free distance in eight directions, right now.	✅ Safe

🚗 Motion

Tool	What it does	Safety
move_to_pixel(frame_id, u, v)	Drive toward a pixel. Corridor-checked; may return reselect.	⚠️ Caution
move_to_pose(pose, frame)	Drive to explicit coordinates (base_link or odom).	⚠️ Caution
move_to_waypoint(index)	Head for a numbered candidate from nav_map(waypoints=True).	⚠️ Caution
task_status(task_id)	Progress, including stalled / diverging with recovery hints.	✅ Safe
task_cancel(task_id)	Abort one move.	⚠️ Caution
robot_stop()	Emergency stop. Zero velocity first, then flag, then cancel all.	🚨 Danger

🧠 Memory & verification

Tool	What it does	Safety
remember(kind, key, value)	Persist a place, object, route or failure — with provenance.	✅ Safe
memory_query(q, kind, k)	Look up what earlier runs learned. Returns poses you can drive to.	✅ Safe
forget(kind, key)	Delete a memory that turned out to be wrong.	✅ Safe
verify_arrival(goal)	Fresh image + clearance + distance travelled, for an honest arrival check.	✅ Safe
memory_stats()	How many facts are stored, by kind, and where the database lives.	✅ Safe

🔍 Introspection

robot_status() reports pose, battery, sensor health, the loaded motion backend, and any tools that could not be registered.

For an unfamiliar robot: ros_list_nodes() · ros_list_topics() · ros_topic_info() · ros_topic_echo() · ros_topic_pub() · ros_service_list() · ros_service_call(). These need the ros2 CLI on PATH; without it they are not registered at all.

reload_drivers() appears only when AGENTNAV_DEV=1.

---

🧠 Skills

Behaviour is taught in Markdown, not compiled in:

Skill	Covers
navigate.md	Instruction → arrival, including distant targets
explore.md	Memory → map → rotate, and when to give up
recover.md	Every failure mode and what to do about it
ros_introspect.md	Learning an unfamiliar robot's topology

---

🛠️ Quick Start

1. Install

pip install nanobot-ai

pip install -e agentnav/

2. Configure

export ANTHROPIC_API_KEY=sk-ant-...
export TELEGRAM_BOT_TOKEN=123456:ABC-...
export MY_TELEGRAM_ID=123456789
export NAVDP_PYTHON=/opt/conda/envs/navdp/bin/python

That is enough to drive. The motion backend defaults to teleop, the built-in short-range controller — it needs only /cmd_vel, /odom and a depth camera, so it works on any ROS2 base with no map and nothing extra installed.

Once you have a tuned Nav2 stack, switch to it for global planning and routing around corners:

export S1_MODE=nav2

Measure the camera mount once. These are not optional decoration: an uncorrected 15° downward tilt inflates every reported distance by several percent.

export CAMERA_Z_HEIGHT=0.45
export CAMERA_PITCH_DEG=12
export ROBOT_RADIUS=0.25

Skip them only if base_link ← camera is published on /tf — AgentNav picks that up automatically and says so in robot_status().

3. Launch

bash agentnav/scripts/start_robot_agent.sh

4. Try it

"Look around and tell me what you see." "Go to the chair on the left." "Find the coffee machine."

---

🧪 Tests

The Harness runs without ROS, a robot or a network, against a synthetic scene:

python -m pytest tests/agentnav/ -q

Covers reprojection and extrinsics, corridor safety, frame identity, retry and backoff, stall detection, the memory store, driver gating, the teleop controller, every tool end to end, and the whole Habitat evaluation harness against a simulated simulator.

📊 Benchmarking

Score AgentNav on the benchmarks the literature uses, with the drivers that ship — no benchmark-only reimplementation of any tool:

python -m agentnav.scripts.eval_habitat --dry-run

python -m agentnav.scripts.eval_habitat --benchmark r2r-ce --split val_unseen -n 100

Reports SR, SPL, OSR, NE, nDTW and SDTW, alongside the costs that matter in practice: LLM turns, tool calls, tokens and wall time per episode, plus collisions, reselects and stalls. Setup, dataset layout and the one simulator-specific accommodation are documented in docs/EVAL-HABITAT.md.

Habitat itself is Linux/macOS only and is not a dependency — the Harness runs without it, and --dry-run verifies the wiring on any machine.

---

🗺️ Status & Roadmap

• MCP core, declarative drivers, hot reload
• Native vision with frame identity, on-demand depth, ROS2 introspection
• Pixel-pointing motion with swept-corridor safety
• Pluggable S1: built-in teleop by default, Nav2 when a map exists
• Agentic memory: BEV map, recall, persistent facts
• Verification: stall detection + arrival checking
• Habitat evaluation harness — R2R-CE / RxR-CE / HM3D-ObjectNav, scored on SR / SPL / nDTW
• Learned S1 policies via BaseS1Client (NavDP, ViNT)
• Multi-robot coordination

Design rationale, benchmarks and the reasoning behind each choice: docs/DESIGN-OPTIMIZATION.md.

---

🤝 Contributing

New sensor drivers, new skills, new robot platforms and new S1 backends all welcome.

Two rules worth knowing before you start:

1. Semantics to the agent, geometry to the Harness. Anything needing trigonometry or a threshold belongs in bridge_core/, not a prompt.
2. Tool errors must say what to do next. A failure that only reports what went wrong wastes an agent turn.

---

📜 License

MIT. See LICENSE.

---

Built with ❤️ for the Robotics & AI Community.