LocalLens

A private, fully on-device video-intelligence app for iPhone: it watches a room through the camera, remembers every object it sees and where it saw it in 3D, and later walks you back to any of them. Ask "where is my water bottle" and an on-device language model answers with when and how far, a 3D arrow points the way, and a targeting reticle locks onto the bottle when the camera can see it again. Nothing ever leaves the device.

The numbers, in one line: the live perception loop runs at 4.2 ms end-to-end p50 (3.2x faster than the baseline, via Metal GPU letterboxing, pipelining, and an adaptive scheduler), 3D re-anchors agree within a 2.8 cm median (n = 33), and Instruments' network instrument records zero bytes during a full session.

Read the full 13-week worklog: pranav6670.github.io/projects/locallens with every number above, how it was measured, and the two places the first explanation was wrong.

Mapping a room	Guided walk-back	Ask your memory	Found it

What's inside

• Live perception loop: YOLO26n on the Neural Engine every frame, Metal GPU preprocessing, one-frame pipelining, and an adaptive scheduler that trades cadence and resolution against thermal and battery state. End-to-end p50 went 13.5 ms → 4.2 ms on an iPhone 17 Pro.
• Spatial-semantic memory: SORT/ByteTrack tracking, LiDAR ⊕ Depth Anything V2 depth fusion, 3D world anchors in SQLite, MobileCLIP2 embeddings with exact vDSP recall. Re-anchor agreement: 2.8 cm median.
• Object permanence: geometry-first identity for look-alike objects, an absence-evidence sweep (missing objects are marked, not hallucinated), and motion handling with consensus re-anchoring.
• An on-device agent (Apple Foundation Models): ask in English, get a grounded answer and a tappable "Take me there", hardened by typed guardrails, not prompts.
• Developer Mode: the entire benchmark kit (latency benches, scheduler ablation, quantization variant picker, live HUD) ships in the same binary behind a toggle, so every performance number is reproducible on your own device.

Requirements

• macOS with Xcode 26 or newer
• An iPhone with LiDAR running iOS 26 (built and tested on iPhone 17 Pro). The camera + LiDAR pipeline means the app runs on a physical device, not the simulator.
• ~350 MB of Core ML models (fetched separately, see below)

Build and run

git clone https://github.com/pranav6670/LocalLens.git
cd LocalLens

1. Get the models (two options):

• Download LocalLens-models.zip from Releases and unzip into app/LocalLens/Resources/Models/, or

• Rebuild everything from source checkpoints (needs Python 3.10+, ~30 min):

  ./models/convert_all.sh

  The script creates a venv, converts every model (Ultralytics YOLO26n export + quantization variants, multifunction 640/320 and 518/364 merges, MobileCLIP2-S2 image/text encoders, SAM 2.1 Tiny), and copies each .mlpackage to the exact path the app expects.

2. Open and run:

1. open app/LocalLens.xcodeproj
2. Signing & Capabilities → select your development team
3. Select your iPhone as the destination and ⌘R

Tests: the LocalLensTests target (212 tests, all pure logic) runs with ⌘U.

Repository layout

app/
  LocalLens/            SwiftUI app: AR, Camera, Memory, Agent, UI, Developer kit
  LocalLensTests/       unit tests for every pure core
  LocalLens.xcodeproj   ready to open (project.yml for xcodegen users)
models/
  convert_all.sh        one-shot: fetch + convert + place all Core ML models
  *.py                  the individual conversion / parity / eval scripts
assets/                 README images

Privacy

Frames, embeddings, thumbnails, 3D positions, and the agent's reasoning never leave the device. Search runs against a local index. During a full capture-and-search session, Instruments' network instrument records zero bytes on the wire.

License

MIT; see LICENSE.