Connect visual scans to AI interpretation

[cseelus]

What changed

• freezes one deliberate AR camera frame after the four-second visual scan, compresses it locally, and stops the camera before network interpretation begins
• adds a server-owned Cloudflare Worker boundary for OpenAI image understanding; the API key never enters the iOS app
• sends the captured frame together with direct center-depth, ambient-light, and LiDAR availability evidence
• uses the OpenAI Responses API with store: false, low-detail image input, GPT-5.6 Luna, and a strict structured-output schema
• renders live output as an AI synthesis · live estimate, including supporting evidence, limitations, and a useful next reading
• retains the guided demo when the service is absent or fails
• removes the demo's fake “93% confidence”: numerical confidence now appears only for a real interpretation
• adds backend validation/extraction tests and runs them in the native workflow before the Xcode build

Why

The first device slice stopped exactly where the screenshot said it did: FIELD could capture camera and depth context, but could not interpret it. This turns that placeholder into a real, privacy-conscious vertical slice while preserving the product boundary between a direct sensor measurement and an AI estimate.

The prompt explicitly forbids claims about composition, chemicals, toxicity, disease, safety, edibility, structural integrity, face identity, and exact species without uniquely identifying evidence. Ambiguous or poor images must produce lower confidence and an honest limitation.

Privacy and operations

• sensors still start only after Begin scan
• FIELD captures one image only for that explicit scan
• sensors stop before the request starts
• the Worker and app do not persist the image or response
• the OpenAI request sets store: false
• the app falls back locally if FIELD_API_BASE_URL is empty or unreachable
• before a public release, the endpoint needs rate limiting and Apple App Attest; an embedded static secret is deliberately not treated as security

Configuration

1. In backend/, run npm install.
2. Add the server secret with npx wrangler secret put OPENAI_API_KEY.
3. Deploy with npm run deploy.
4. Set the iOS target build setting FIELD_API_BASE_URL to the Worker origin.

Validation

• backend: 5/5 Node tests pass
• web: production build and 2/2 rendered-HTML tests pass
• git diff --check passes
• Xcode is unavailable in the Linux workspace; the PR's macOS Native iOS check is the authoritative compile validation