Skip to content

NOTES.md

Latest commit

 

History

History
474 lines (352 loc) · 34.1 KB

NOTES.md

File metadata and controls

474 lines (352 loc) · 34.1 KB
Raw

Echo — Build notes

Health & Human Optimization Hackathon. Pitch: 8:30 PM.

Current state (PHASE 6 — RESILIENT CAPTURE)

DONE. The capture pipeline no longer assumes getUserMedia is going to work. Before, if the laptop didn't have a mic detectable by PipeWire, the whole flow aborted with NotFoundError. Now each audio source (mic, loopback) is opened in its own try/catch, and the app continues with whatever it managed to get. It only aborts if literally nothing could be opened — and even then the error screen has a "Use demo audio" button so the user is never stranded.

Why it was changed

Real incident during a live demo: the laptop didn't have a physical microphone exposed by PipeWire. The first click on "Speak for 30 seconds" threw NotFoundError: Requested device not found and the app stayed dead until refresh and discovering the shy link to demo mode. Unacceptable for a tool whose pitch is "30 seconds without hardware".

The 4 possible combinations in system mode

micStream sysStream activeCapture UI
✓ ✓ mic-system Cyan badge Mic + System, no warning
✓ ✗ mic-only Amber badge Mic only + soft warning
✗ ✓ system-only Amber badge System only + soft warning
✗ ✗ (not applicable) Error screen with "Use demo audio" button

Pure mic mode: if it opens, cyan badge Mic only. If not, error screen with the demo escape hatch. demo mode: zinc badge Demo audio.

Technical changes

  • handleStart(mode: "mic" | "system") no longer throws at each step; it wraps each getUserMedia and device enumeration in its own try/catch that logs a console.warn and leaves the variable at null.
  • LOOPBACK_HINTS lifted to the module top level (was inside the body of handleStart).
  • New state activeCapture: "mic-only" | "system-only" | "mic-system" | "demo" reflects what was actually opened, NOT what the user asked for. That's what the badge uses.
  • New state captureWarning: string | null for the soft message shown at the top during recording when there was degradation.
  • New CaptureBadge component maps (activeCapture, captureMode) to (label, tone). Cyan tone for complete cases, amber for degraded cases, zinc for demo.
  • RecordingView now receives activeCapture + captureWarning; the badge goes up to the right next to the "Capturing audio" label, and the warning appears as a soft amber pill below.
  • ResultView receives activeCapture; the badge sits next to "REPORT READY" in the header, so the judge sees what was captured when they read the report.
  • mic-error screen: primary button is now Use demo audio (handleStartDemoAudio) and the secondary is Back (handleReset). Before, it only had "Try again" which sent you to idle without a clue.
  • MediaRecorder ties to the stream that's actually playing: the destination of the merger when there are two sources, or the raw stream (mic or sys) when there's only one. Whisper always receives the blob from the real source, never an empty stream.

What was NOT touched

  • lib/meydaCapture.ts, lib/audioFeatures.ts, lib/speechRecognition.ts, all /api/* routes, ResultCards, EvolutionChart, CrossInsightsCard, LiveFeaturesStrip, Waveform remained intact.
  • The loopback auto-detect (LOOPBACK_HINTS) is exactly the same; we only removed the throw when there's no match.
  • demo mode with public/demo-audio.mp3 is identical — it only adds setActiveCapture("demo") so the badge renders correctly.
  • Comparison screen, save profile, individual result: no regressions.

How to test each case locally

cd /home/junrod/Desktop/Echo
npm run dev
# → http://localhost:3000
  1. No mic detected (mic mode): physically disconnect the mic or pactl unload-module module-alsa-source (Linux). Click "Speak for 30 seconds" → should show the "We couldn't record" screen with the Use demo audio primary button and Back secondary. Used to crash.
  2. No mic + no loopback (system mode): same setup as (1), click "Capture mic + system audio" → idem, useful error with the demo escape.
  3. Mic, no loopback (system mode): on a machine without Stereo Mix/Monitor, click "Capture mic + system audio" → records with mic only, amber badge Mic only at the top and amber warning "No system audio detected, capturing only your mic.".
  4. No mic, with loopback (system mode): hard to reproduce in practice because without a mic the device labels come back empty and the loopback matcher doesn't work — the app treats this case as "nothing available" and goes to the error with the demo escape. It's a browser limitation, not ours. Documented.
  5. Demo mode: click "Mic trouble? Use demo audio" → plays public/demo-audio.mp3 just like before, zinc badge Demo audio.
  6. Result + comparison: the captured mode badge is also visible at the top of the report (next to "REPORT READY") and the comparison screen keeps working without changes.

Pitch hint

If the doctor asks what happens when "something doesn't work": the app degrades instead of failing. It tells you what it captured, warns you if it's partial, and offers demo audio as a safety net. Resilience ≠ silencing errors: each fallback has a visible badge that the user (and the judge) can read.

Previous state (PHASE 5 — COMPARISON UI FIX)

DONE. The "profile comparison" screen goes from a cramped 6-column grid (two <ResultCards /> side by side inside an xl:grid-cols-2) to an executive three-tier report: hero scoreboard with deltas, detail rows per dimension, and collapsible cross insights + transcripts.

Why the previous view was broken

  • <ResultCards /> already renders 3 cards in lg:grid-cols-3. Rendering it twice inside xl:grid-cols-2 produced 6 cramped columns on normal screens.
  • Scores (text-3xl) overflowed the card border when shrunk.
  • Asymmetric heights: the "Mental wellbeing" card has pills + actions, the other two only summary — no visual parity between rows.
  • Text wrapped word by word in such narrow columns.
  • No "comparison" narrative: just two identical blocks glued together, no deltas, no visible numeric differentiation.

What changed in the architecture

  • New component: components/ComparisonReport.tsx. Encapsulates the entire comparison report. Receives current, saved, onClose, onClearSaved, onReset.
  • ResultCards NOT touched: the individual result screen remains identical. The comparison was duplicating that component; now it doesn't use it at all, it builds its own layout.
  • ComparisonView in app/page.tsx stays as a thin wrapper passing props to ComparisonReport. All previous inline JSX deleted.

Structure of the new layout

  1. Header: cyan subtitle PROFILE COMPARISON, title Two voices, two profiles, zinc-400 subtitle {saved label} ({hh:mm}) vs {current label} with aria-label.
  2. Hero scoreboard: one row of 3 cards (lg:grid-cols-3, stack on mobile), one per dimension. Each card: icon + title, two text-5xl scores side by side with vertical divider, short label truncated (max 12 chars with tooltip), delta pill (+8 pts ↑ emerald / -5 pts ↓ rose / No change zinc). Border border-cyan-500/25 to set them apart as "score".
  3. Detail rows per dimension: 3 full-width cards in vertical stack, each with a dimension header and two columns (md:grid-cols-2, items-start). Each column: header with label + large score with scoreColor, divider, dimension-specific content:
    • Mental: pills (Stress/Fatigue/Mood) + summary + recommended actions list with check icons.
    • Respiratory and Cognitive: summary only (no invented extra content).
  4. Cross insights: two columns. If both have cross_insights, two <CrossInsightsCard /> side by side. If only one has them, that column + dashed placeholder. If neither has them, the section is omitted.
  5. Transcripts: collapsible section (closed by default). Ghost View transcripts toggle button. When open, two columns with each transcript in a sober card. If a transcript is empty, that column is omitted.
  6. Actions: Back to single report / Delete saved profile / New session — same handlers as before (onClose, onClearSaved, onReset).

New helpers in ComparisonReport.tsx

  • scoreColor(score): identical mapping to ResultCards (≥80 emerald, ≥60 cyan, ≥40 amber, else rose).
  • delta(current, saved): { value, sign, color, label } with sign up/down/flat and Tailwind classes for the pill.
  • truncateLabel(label, max): cuts with … and returns { display, full } for title=.
  • indicatorTone / indicatorLabel: copied from ResultCards for the mental pills.

How to verify visually

cd /home/junrod/Desktop/Echo
npm run dev
# → http://localhost:3000
  1. Click Speak for 30 seconds → record 30s → result → Save as reference → label A.
  2. Start over → record another 30s → result → Compare with A.
  3. See: header with A (HH:MM) vs Current session, row of 3 hero cards with deltas, 3 balanced detail rows, cross-insights section if applicable, View transcripts button closed by default.
  4. Resize to mobile (~360px): everything should stack to 1 column without overflow.
  5. Back to single report → the individual screen looks identical (smoke check for no regressions on ResultCards).

Risks / notes

  • If an older saved profile doesn't have cross_insights (pre-PHASE 3.5), the entire section is cleanly omitted (null when both are empty, dashed placeholder for the one that's missing).
  • Collapsed transcripts by default reduce scroll and give hierarchy to the scoreboard.
  • card-in with animationDelay 0/120/240ms applied to hero cards AND to dimension rows — the stagger appears doubled (first the hero, then the detail rows while the user reads). Deliberate.

Previous state (PHASE 4 — SYSTEM MODE)

DONE. The shared mode (which relied on getDisplayMedia and forced the user to pick a specific tab with a "Share tab audio" tick) was replaced by a system mode that captures mic + system audio with a single button and no tab-picker dialog.

What changed

  • CaptureMode = "mic" | "system" | "demo" (was: "mic" | "shared" | "demo").
  • Deleted the getDisplayMedia({ audio: true, video: true }) path and displayStreamRef. Replaced by two simultaneous getUserMedia calls:
    1. Real mic (default) — opens the mic permission once.
    2. System loopback / Monitor — opened with getUserMedia({ audio: { deviceId: { exact: <loopback.deviceId> } } }).
  • Mix identical to before: both MediaStreamSource → GainNode (merger) → MediaStreamAudioDestinationNode for MediaRecorder. Meyda reads from the merger. Whisper receives the mixed blob.
  • Web Speech does NOT start in system mode (same as before in shared/demo). The transcript comes from Whisper.
  • Downstream pipeline unchanged: Meyda → MediaRecorder → /api/transcribe → /api/analyze → cards.

How the loopback auto-detect works

  1. Request mic permission once (getUserMedia({audio:true})). This unlocks real label values in enumerateDevices(); without that permission, labels come back empty and the matcher doesn't work.
  2. Filter audioinput and look for a device whose label.toLowerCase() contains any of these substrings:
    • monitor (Linux PulseAudio/PipeWire — "Monitor of Built-in Audio")
    • loopback
    • stereo mix (Windows)
    • blackhole (Mac)
    • soundflower (Mac legacy)
    • vb-audio, vb-cable (Windows virtual cables)
    • wave out, what u hear (legacy Windows)
  3. No match → throw with an OS-specific message and clean abort (the permission tracks get stopped).
  4. Match → use permStream as the mic stream (it's the default mic) and open a new stream specific to the loopback.

What to do if the Monitor device doesn't show up

  • Linux (my setup): PulseAudio/PipeWire expose them by default. Verify with pactl list sources short — at least one source with .monitor in the name should appear. If not, check the PulseAudio config or install pavucontrol.
  • Mac: install BlackHole (free, open source). Then, in Audio MIDI Setup, create a Multi-Output Device so audio goes out to both speakers AND BlackHole at the same time.
  • Windows: enable Stereo Mix in Sound Control Panel → Recording → right click → "Show Disabled Devices" → enable Stereo Mix. If it doesn't exist (modern laptops often don't have it), install VB-CABLE and route system output through the virtual cable.

Why this mode is better for the pitch

  • Single click: no "choose a tab" dialog, no manual "Share audio" tick. Click → mic permission → record.
  • Captures EVERYTHING playing: doesn't matter if the doctor is on Meet in a tab, on the Zoom native app, or on a WhatsApp call in another window. If it plays through speakers/headphones, Echo captures it.
  • Pitch flow now: have Meet (or whatever) open in any window → click "Capture mic + system audio" → grant mic once → speak → 30s later the report is there.

Known risk

If the judge's laptop doesn't have loopback configured, the mode falls back to the OS-specific error message. mic mode and demo mode keep working as independent fallbacks.

Previous state (PHASE 3.5 — KILLER VISUALS)

DONE. 5 additive visual features layered on the existing flow. Nothing broke — the mic / shared / comparison flows keep working as before.

What's new

  1. Live indicators during recording (components/LiveFeaturesStrip.tsx): 3 horizontal mini-cards below the waveform showing Energy, Pitch and Clarity moving in real time with each Meyda callback. Bars with transition-all duration-100. startMeydaCapture now accepts an optional onFrame (doesn't break prior callers). The frame is stored in a ref and flushed to React state every 80 ms — so Meyda runs at its own pace (~43 fps) and the render doesn't overload.
  2. Mini evolution chart (components/EvolutionChart.tsx): Recharts LineChart with 5 points (4 mocked historical + the real current one) and 3 lines (Mental cyan, Respiratory emerald, Cognitive amber). Dark-styled tooltip, legend, dots with activeDot. Appears below the 3 score cards.
  3. Cross Insights card (components/CrossInsightsCard.tsx): cyan→emerald gradient border, Sparkles icon, list of 1-3 observations that cross two dimensions. The OpenAI schema now includes cross_insights: string[] as required (strict mode). The system prompt asks for observations like "the pause pattern combined with vocal monotony suggests attention to your cognitive load". Validated with curl: the model returns 2-3 strings coherent with the features sent.
  4. Analysis meta header (ResultMetaStrip inline in page.tsx): shows analysis date + time (new Date() captured when the result arrives), real recorded duration (features.duration_seconds), and the model used ("Processed by GPT-4o + Whisper + Meyda"). Discreet style, under the subtitle.
  5. "Use demo audio" button (sober link on idle screen): pitch life-saver. captureMode = "demo" runs the full flow with a public/demo-audio.mp3 file. Meyda analyzes via MediaElementAudioSourceNode, Whisper transcribes the fetched blob, OpenAI generates the report. The audio plays through the speakers while the analysis runs (source.connect(audioCtx.destination)).

How to activate the "demo audio" button

  1. Upload a demo-audio.mp3 file to the public/ folder of the project: 
    /home/junrod/Desktop/Echo/public/demo-audio.mp3
  2. Any browser-decodable format works (mp3, ogg, wav, m4a). The path in code is /demo-audio.mp3 — to use a different name, change the DEMO_AUDIO_PATH constant in app/page.tsx.
  3. Recommended: 30 seconds of human English voice so Whisper transcribes something coherent and the scores vary.
  4. If the file does NOT exist → clicking takes the flow to "mic-error" with a clear message: "Demo audio not found at /demo-audio.mp3. Upload the file to /public."

Extended OpenAI schema

cross_insights added to ECHO_SCHEMA under required (strict mode requires it). The system prompt adds the specific rule for that key. The frontend reads result.cross_insights ?? [] with graceful fallback for older saved profiles.

Verified smoke tests

  • GET / → 200, 19 KB of HTML (vs 16 KB before — extra UI)
  • POST /api/analyze → response has keys [mental_wellbeing, respiratory_voice, cognitive_performance, cross_insights]. cross_insights.length === 2, first insight: "The high coherence and lexical richness suggest you're in an optimal mental state..."
  • POST /api/transcribe and the other endpoints with no regressions (only features were added, nothing was modified destructively).

Risks and notes

  • localStorage of older saved profiles: if Junior saved a profile BEFORE this update, it has no cross_insights. The card simply does not render for that profile (returns null on empty array). Side-by-side comparison keeps working.
  • Live indicators with a silent stream: if the shared tab isn't playing anything, the 3 bars stay at 0. Not a bug, it's a signal — it tells the judge "no audio coming through, try again".
  • Recharts is ~110 KB minified. Acceptable for a demo.
  • MediaElementAudioSourceNode: if you connect the same audio element to destination AND to Meyda, it plays through the speakers while Meyda reads it. Dramatic bonus for the live demo.

Previous state (PHASE 3 — KILLER DEMO MODE)

DONE. System audio capture (shared tab) + mic, Whisper transcription, and side-by-side comparison of saved profiles.

What's new

  • /api/transcribe route with OpenAI Whisper (whisper-1, language: "en"). Receives FormData with audio field, returns { transcript: string }. Tested with curl: 200 with real audio, 400 without audio, 500 without key.
  • Demo mode on idle screen: two buttons —
    • Primary: "Speak for 30 seconds" (mic only, Web Speech API → instant)
    • Secondary: "Capture shared audio + mic" (getDisplayMedia + mix + Whisper)
  • Tab audio capture: uses navigator.mediaDevices.getDisplayMedia({ audio: true, video: true }). Chrome shows the "share screen" dialog, the user ticks "Share tab audio" and picks the tab (Meet, Zoom Web, YouTube, etc). The video track is discarded immediately — we only care about audio.
  • Stream mixing: tab audio + mic mixed in a Web Audio GainNode. Meyda reads the mix. A MediaStreamAudioDestinationNode exports the mix as a new MediaStream recorded by MediaRecorder for Whisper.
  • Whisper instead of Web Speech: in demo mode, Web Speech does NOT start (it would only listen to your mic). The full MediaRecorder blob is sent to /api/transcribe after the 30s.
  • Extended "Analyzing..." steps in demo mode: adds "Transcribing with Whisper..." between audio processing and extraction.
  • Recompute features with real transcript: in demo mode, features are computed AFTER having the Whisper transcript, so speech_rate and lexical_richness come out accurate.
  • Profile saved in localStorage: "Save as reference" button on the result screen. Prompts for a label (default "Doctor" in demo mode, "My profile" in mic). Persists { label, result, features, transcript, savedAt }.
  • Side-by-side comparison: if there's a saved profile, a "Compare with [label]" button appears. Renders two sets of 3 cards in a 2-column grid (xl:grid-cols-2), each with its label on top.
  • Saved profile indicator on idle: cyan card with the label + trash button to delete it.

The killer pitch flow

  1. Before the pitch (5 min before): record 30s with "Speak for 30 seconds" → "Save as reference" → label "Junior".
  2. The pitch starts on Meet, you share your screen.
  3. You show Echo, open the flow, click "Capture shared audio + mic".
  4. Chrome opens the dialog. CRITICAL: tick "Share tab audio", pick the Meet tab, click "Share".
  5. Ask the doctor-judge to talk 30s about anything in her day.
  6. Echo captures her voice via the Meet tab (codec quality, not acoustic).
  7. Whisper transcribes (3-5s), Meyda extracts features, OpenAI generates the report.
  8. 3 cards appear with her scores and the transcript of what she said.
  9. Click "Compare with Junior" → 6 cards on screen, two distinct profiles.
  10. "Echo distinguishes two voices in 30 seconds each. This is what would come to an EPS or to HR."

Demo risks (READ BEFORE STARTING)

  • Tick "Share tab audio". If you forget, no system audio comes through and we get the error "You didn't tick 'Share audio'... try again and check the option." (specific message). Practice the sequence 2 times before the pitch.
  • Share the right tab: the tab you pick in the dialog is the one that provides the audio. If your Meet is in one tab and you share another, her voice doesn't come through. Share the Meet tab.
  • getDisplayMedia requires a user gesture: already handled, the button triggers it.
  • Whisper adds ~5s of latency. The "Analyzing..." screen with 4 steps covers it visually.
  • macOS / Linux with Wayland: in some cases the system denies getDisplayMedia. If it happens, mic-only mode keeps working.
  • If Whisper fails (rate limit, key, etc): analyze-error screen with "Retry analysis". Features are already saved, no need to re-record.

New endpoints / files

app/api/transcribe/route.ts   # NEW — Whisper proxy
app/page.tsx                   # rewrite: 2 modes, save/compare

page.tsx now has CaptureMode = "mic" | "shared". Core functions (handleStart, finishRecording, runAnalyze) branch on mode.

How to verify end-to-end

npm run dev   # http://localhost:3000

Test mic mode (fast):

  1. Click "Speak for 30 seconds"
  2. Grant mic
  3. Speak for 30s
  4. Result in ~3-5s

Test demo mode:

  1. Open YouTube in another tab, queue up a video with English voice without hitting play yet
  2. Click "Capture shared audio + mic"
  3. Chrome shows the dialog → "Tab" → pick YouTube → tick "Share tab audio" → "Share"
  4. Immediately play the YouTube video
  5. Echo captures + Whisper transcribes + reports
  6. The transcript should show what the video said, not what you said

Test comparison:

  1. After the first report, "Save as reference"
  2. "Start over"
  3. Another report
  4. "Compare with [label]"

Verified smoke tests (curl)

  • GET / → 200, page renders with both buttons
  • POST /api/transcribe with no audio → 400 with clear error
  • POST /api/transcribe with a real WebM (440 Hz tone, 2s) → 200 with transcript field
  • POST /api/analyze with complete features → 200 with valid structured JSON

Previous state (PHASE 2.5 — MEYDA UPGRADE)

DONE. Acoustic features extracted with real Meyda. Manual computation remains as fallback.

What changed in this upgrade

  • Meyda 5.x installed. Replaces the manual RMS sampling with Meyda.createMeydaAnalyzer and bufferSize 1024.
  • Features extracted frame by frame: rms, zcr, spectralCentroid, spectralFlatness, spectralRolloff, perceptualSpread, perceptualSharpness, loudness, energy.
  • AcousticFeatures extended with new spectral and perceptual fields (see lib/types.ts). The payload sent to OpenAI now carries: spectral_centroid_mean, spectral_centroid_std, spectral_flatness_mean, spectral_rolloff_mean, perceptual_spread_mean, perceptual_sharpness_mean, zcr_mean, lexical_richness, in addition to the previous ones.
  • pitch_variability_aprox is now real: std(spectralCentroid) / mean(spectralCentroid) — relative spectral variability, much better proxy for prosody than the energy_std × 4 of the fallback.
  • shimmer_aprox is now real: std(rms) / mean(rms) computed over Meyda frames (bufferSize 1024 at ~44.1 kHz = ~23 ms per frame, academic granularity).
  • lexical_richness = unique_words / total_words computed on the client from the transcript.
  • Provenance: each AcousticFeatures carries source: "meyda" | "manual". Popovers show a Meyda or fallback badge so the judge sees where the numbers come from.
  • Automatic fallback: if startMeydaCapture fails (browser without ScriptProcessor, import error, etc.), it automatically falls back to manual sampling with AnalyserNode + getByteTimeDomainData every 100ms. Doesn't break the flow.
  • Category-specific popovers:
    • Mental: F0 variability, pause ratio, shimmer, speed
    • Respiratory: spectral flatness, spectral rolloff, energy mean, duration
    • Cognitive: speed, lexical richness, pause ratio, spectral centroid
  • System prompt updated with rules that mention the real names of the Meyda features. Validated with curl: high features under stress → indicators "high", low pitch_variability → depression_risk "moderate" with a referral suggestion.

How to verify that Meyda is running (not the fallback)

  1. Open DevTools → Console before recording.
  2. If Meyda fails, you'll see console.warn("Meyda failed to initialize, falling back: ...").
  3. After getting a result, open the Popover on any card. Upper right there's a badge:
    • Meyda (cyan) → Meyda ran
    • fallback (amber) → fell back to manual
  4. If you're in fallback, the spectral fields (spectral_flatness, spectral_rolloff) will come back as 0 — visible in the "Respiratory health" popover.

Why you should see more variation with Meyda

Before (manual fallback), pitch_variability_aprox was approximated as energy_std × 4 — it only captured volume instability, not real pitch. Now it comes from the spectral centroid variance, which DOES move with real pitch (singing a flat note vs speaking with intonation → centroid changes a lot).

Expected result: speaking very flat and monotone (robotic reading) will pull pitch_variability_aprox below 0.1 → activates the "flat prosody → low mood marker" rule → depression_risk rises to mild or moderate. Speaking with normal intonation keeps it minimal.

Base state (PHASE 2 — REAL INTEGRATION)

DONE. End-to-end flow with real LLM, real browser features, and transcription.

What changed from Phase 1

  • Provider: OpenAI (not Anthropic). Model gpt-4o-2024-08-06 with response_format: { type: "json_schema", strict: true }. The response is JSON guaranteed by the model — no defensive parsing or retries.
  • Real features from AudioContext + AnalyserNode (fftSize 2048). RMS sampling every 100 ms during recording.
  • Real Web Speech API (webkitSpeechRecognition) in en-US, continuous + interim. We only accumulate result.isFinal.
  • Clickable tooltips on each card with @radix-ui/react-popover: show the 5 real user features + clinical text hardcoded by category.
  • Visible transcript on the result screen, in a sober card below the 3 scores. If the browser doesn't support Web Speech, an amber-500 warning is shown.
  • Retry without re-recording: on analyze-error, the state keeps features and transcript, so "Retry analysis" reuses the previous capture.
  • Analyzing steps: 3 progressive steps change every 700 ms on the waiting screen.

Feature computation (client)

In lib/audioFeatures.ts:

Feature How it's computed
energy_mean average RMS per frame
energy_std standard deviation of RMS per frame
pause_ratio % frames with RMS < (energy_mean × 0.3)
duration_seconds real performance.now() at start/stop
shimmer_aprox clamp(energy_std / max(energy_mean, 0.001), 0.01, 0.15)
pitch_variability_aprox clamp(energy_std × 4, 0.05, 0.5)
speech_rate transcript_words / duration_seconds

The first 4 are measured. The last 3 are derived from the real ones — not random. If Junior speaks very softly or very loudly, the derived values also change.

API route

app/api/analyze/route.ts:

  • POST receives { features, transcript }
  • Calls OpenAI with SYSTEM_PROMPT + strict ECHO_SCHEMA
  • Returns parsed JSON or { error } with the right status
  • Handling: 401 (invalid key), 429 (rate limit), 400 (bad body), 500 (no key or generic error)

Tested with real curl: POST /api/analyze with a test payload returns 200 in ~3.8 s and JSON conforming to the schema.

Technical decisions taken in Phase 2

  1. Capture logic inline in page.tsx instead of a custom hook: only one component consumes it — the indirection doesn't pay off under time pressure.
  2. MediaRecorder kept even though we don't use the blob: it gives us Chrome's "recording state" (mic LED) and forward compatibility to send audio to the backend if we wanted to.
  3. performance.now() for real duration: key if the user hits "Stop now" before 30 s.
  4. continuous: true on Web Speech: in long sessions Chrome sometimes cuts it and fires onend. We keep what was accumulated up to that point.
  5. Separate mic-error vs analyze-error statuses: different UX. The first requires re-recording, the second allows retry with already-captured data.
  6. low/moderate/high states mapped to colors: emerald (low/minimal), cyan (mild), amber (moderate), rose (high/elevated). Same mapping applied to pills and to the large score color.

Key commands

cd /home/junrod/Desktop/Echo
npm run dev
# → http://localhost:3000
# Endpoint smoke test without recording:
curl -X POST http://localhost:3000/api/analyze \
  -H "Content-Type: application/json" \
  -d '{"features":{"duration_seconds":25,"energy_mean":0.05,"energy_std":0.02,"pause_ratio":0.18,"shimmer_aprox":0.04,"pitch_variability_aprox":0.15,"speech_rate":2.4},"transcript":"today I had a long meeting, I felt fine although a bit tired"}'

Environment variables

# .env.local
OPENAI_API_KEY=sk-...

Without this key, the endpoint returns 500 with a clear message. The frontend shows it in analyze-error with a "Retry analysis" button.

How to test end-to-end (what counts for the demo)

  1. npm run dev, open http://localhost:3000 in Chrome.
  2. Click "Speak for 30 seconds" → grant mic permission.
  3. Speak while the countdown runs. Spoken words appear in gray below the waveform in real time (Web Speech).
  4. Wait 30 s or hit "Stop now".
  5. "Analyzing" screen walks through 3 steps.
  6. 3 cards with real model scores + indicators (Stress/Fatigue/Mood) + 3 recommended actions.
  7. Click the Info icon on any card → Popover with the real acoustic features + clinical rationale.
  8. "What you said" block below with the full transcript.

What to say into the mic to see clear variation

  • Low, slow, monotone voice, negative phrases ("I'm tired today, I don't feel good, everything feels heavy, nothing motivates me..."): should lower mental_wellbeing and push indicators.depression_risk to mild or moderate. High pause_ratio, low pitch_variability_aprox.
  • Energetic, fast voice, positive phrases ("I'm excited for the pitch, things are going well, the team is aligned"): should raise scores overall, low indicators.
  • Very fast voice with no pauses and choppy speech: can raise stress to moderate/high.
  • 5-10 s audio: the model returns scores near 50 with a summary noting "limited duration" — correct behavior per the system prompt.

Project structure (updated)

Echo/
├── app/
│   ├── layout.tsx
│   ├── page.tsx                 # state machine + AudioContext + Speech + fetch /api/analyze
│   ├── globals.css
│   └── api/
│       └── analyze/
│           └── route.ts          # OpenAI structured outputs
├── components/
│   ├── ui/
│   │   ├── button.tsx
│   │   ├── card.tsx
│   │   ├── progress.tsx
│   │   └── popover.tsx           # Radix Popover wrapper
│   ├── Waveform.tsx
│   └── ResultCards.tsx           # now takes `features` + has the 3 Popovers
├── lib/
│   ├── utils.ts
│   ├── types.ts
│   ├── audioFeatures.ts          # computeRmsFromTimeDomain, computeFeatures
│   ├── speechRecognition.ts      # webkitSpeechRecognition wrapper
│   └── mockResult.ts             # legacy from Phase 1, no longer used
└── package.json

mockResult.ts is orphaned. Left as reference until Phase 4 in case Vercel deploy fails and we need to re-enable the fallback.

Pending for PHASE 3 (3:00 PM – 4:30 PM)

  • Mini evolution chart with Recharts (4 mock points) below the cards
  • Secondary "Use demo audio" button (pitch backup in case the venue's mic fails)
  • Mobile polish (the 3 cards should stack well on a narrow screen — already there with lg:grid-cols-3, validate visually)
  • Polish header / hero copy

Pending for PHASE 4 (4:30 PM – 5:30 PM)

  • Deploy to Vercel
  • OPENAI_API_KEY in Vercel env vars (Production)
  • Smoke test on the prod URL
  • Test on at least 2 browsers (Chrome and Safari for the judge just in case — Safari doesn't have Web Speech, validate the fallback)

Known risks / things to know for the pitch

  • Web Speech API doesn't work in Firefox nor in some Safari builds. The amber warning is shown and the scores are computed from acoustic features alone. If the judge uses Chrome/Edge, OK.
  • Mic permissions require https or localhost. Vercel provides https, OK.
  • OpenAI structured outputs: the model RESPECTS the strict schema, no risk of malformed JSON. The only possible errors are 401/429 — handled with retry without re-recording.
  • Latency: the OpenAI call takes ~3-5 s in the smoke test. The "Analyzing" screen with 3 steps covers that visually.
  • Meyda uses ScriptProcessorNode (deprecated but supported in Chrome). If it breaks in modern Safari, the manual fallback kicks in automatically — the app doesn't crash.
  • If the doctor asks how we really measure shimmer/jitter: shimmer_aprox and pitch_variability_aprox are now computed over Meyda frames (bufferSize 1024 ≈ 23 ms at 44.1 kHz). It's still a proxy, not Praat's canonical shimmer. The honest answer: "Meyda gives us spectrum and RMS frame by frame; we compute relative variance as a proxy. For real clinical use we'd move to Praat or to algorithms like YIN/RAPT for pure F0. The detection of pauses, speech rate and lexical richness are direct measures."