AI-powered communication system for ALS patients — type with your eyes, speak with AI.
GazeConnect enables people with ALS and severe motor impairments to communicate using only facial movements. The patient selects keywords through gaze-based zone interaction, a locally running LLM (Gemma 4) expands them into natural first-person sentences, and the system speaks them aloud. Supermemory Local provides persistent, on-device memory that learns from every interaction — making the AI smarter over time while keeping all data completely private.
Built for the Supermemory Local Hackathon (localhost:6767)
| Feature | Description |
|---|---|
| Facial Tracking Input | Hands-free communication using webcam-based gaze zone detection |
| AI Sentence Completion | Gemma 4 running locally converts keywords into natural first-person sentences |
| Supermemory Integration | Persistent on-device memory that learns patterns, stores profile, and retrieves context |
| Patient Profile System | Caretaker configures patient details (family, routine, preferences) once — AI uses them forever |
| Session-Aware Context | Caretaker provides live voice context ("it's medicine time") that shapes AI responses |
| Predictive Suggestions | 4-quadrant suggestion mode shows frequently used words matching current input |
| Text-to-Speech | Generated sentences are spoken aloud via Web Speech API |
| Blink Detection | Sustained eye closure triggers delete/cancel actions |
| 100% Private | All AI, memory, and tracking runs on-device. Zero cloud. No data leaves the system. |
Supermemory Local (localhost:6767) is the core intelligence layer:
-
Sentence Memory — Every sentence the patient generates is stored as a document. When the patient types new keywords, a hybrid search retrieves the most relevant past sentences, which are injected into the LLM prompt for context-aware completions.
-
Patient Profile Storage — Profile facts (family names, daily routine, dietary preferences, medical condition) are stored as individual documents in a dedicated
patient_profilecontainer for precise semantic retrieval. Typing "chai" surfaces the patient's tea preferences and schedule. -
Session Context — Caretaker voice context is stored per-session, so the AI always knows the current situation. Multiple contexts accumulate during a session and are deduplicated before prompt injection.
-
Word Suggestions — Supermemory tracks word frequency across all stored sentences. The suggestion engine queries this data to show the 4 most likely words matching the patient's current input.
-
Behavioral Learning — Auto-learned patterns from past conversations are merged with manual profile data to create a comprehensive context that improves with every use.
A webcam continuously captures the user's face. MediaPipe Face Mesh detects facial landmarks and tracks nose-tip displacement relative to the eye center to estimate gaze direction.
The screen is divided into six gaze zones:
+-------------------+-------------------+-------------------+
| | CENTER-TOP | |
| ZONE 1 | (Switch Mode) | ZONE 2 |
| A B C D E F | | G H I J K L M |
| | CENTER (Rest) | |
+-------------------+ (Safe Zone) +-------------------+
| | | |
| ZONE 3 | CENTER-BOTTOM | ZONE 4 |
| N O P Q R S T | (Send to AI) | U V W X Y Z ␣ |
+-------------------+-------------------+-------------------+
Accuracy features:
- 7-frame smoothing buffer reduces jitter
- Stability gating requires 3 consecutive detections before zone activation
- Wide dead zones around center for safe resting
- Automatic calibration on first launch
Main View — 4 large quadrant zones, each containing 6-7 letters. Dwell for 1.8 seconds to open a zone.
Sub View — The selected zone expands into a 2x2 grid with letter pairs:
+----------+----------+
| A / B | C / D |
+----------+----------+
| E / F | CANCEL |
+----------+----------+
- Dwell 1.5s on a cell to select the displayed letter
- Look at CENTER-TOP for 1.0s to toggle between letters (A -> B)
- Blink to cancel and return to Main View
Toggle via CENTER-TOP. All four zones display the most frequently used words matching the current input:
+-------------------+-------------------+
| ⚡ bathroom | ⚡ hungry |
| SUGGESTION 1 | SUGGESTION 2 |
+-------------------+-------------------+
| ⚡ water | ⚡ tired |
| SUGGESTION 3 | SUGGESTION 4 |
+-------------------+-------------------+
Suggestions are passively visible in Alphabet Mode as well, shown below each zone label.
The patient types keywords, and the LLM generates what the patient wants to say:
Input: "chai"
Context: "it is 5 o'clock, Arjun just had his physiotherapy"
Output: "Can I have my evening chai now, please?"
The prompt is enriched with:
- Patient profile from Supermemory (family, preferences, routine)
- Past sentences retrieved via hybrid search
- Current caretaker context (most recent, deduplicated)
- System prompt enforcing first-person, short, direct sentences
The caretaker clicks the microphone button and speaks situational context:
"Ananya called, she is coming to visit this weekend"
This context is stored in Supermemory and shapes every subsequent AI response until cleared or replaced.
| Component | Technology |
|---|---|
| Backend | Flask (Python) |
| Frontend | HTML, CSS, JavaScript, Feather Icons |
| Facial Tracking | MediaPipe Face Mesh |
| Gaze Estimation | Nose-tip displacement algorithm |
| AI Model | Gemma 4 (local, via llama-cpp-python) |
| Memory Layer | Supermemory Local (localhost:6767) |
| Speech Output | Web Speech API (TTS) |
| Speech Input | Web Speech Recognition API |
| Camera | OpenCV + WebRTC |
GazeConnect/
├── app.py # Flask server, API routes, LLM integration
├── gaze_tracker.py # MediaPipe gaze tracking + zone classification
├── memory.py # Supermemory SDK wrapper (storage, search, suggestions, profile)
├── requirements.txt # Python dependencies
├── .env # Environment variables (API keys, model path)
├── templates/
│ ├── index.html # Main gaze-tracking communication UI
│ └── setup.html # Patient profile setup page
└── static/
├── css/
│ ├── style.css # Main application styling
│ └── setup.css # Setup page styling
└── js/
├── main.js # Client-side state machine, gaze polling, UI rendering
└── setup.js # Setup page logic (profile save/load)
- Python 3.10+
- A standard webcam
- Supermemory Local running on
localhost:6767 - A Gemma 4 GGUF model file (download Q4_K_M from HuggingFace)
git clone https://github.com/ShahbazCoder1/GazeConnect.git
cd GazeConnectpip install -r requirements.txtnpx supermemory localThis will output an API key. Copy it for the next step.
Create a .env file in the project root:
# Supermemory Local
SUPERMEMORY_API_KEY=sm_your_key_here
SUPERMEMORY_BASE_URL=http://localhost:6767
# Gemini API key (used by Supermemory Local for memory extraction)
GEMINI_API_KEY=your_gemini_api_key
# Local LLM Model Path
MODEL_PATH=C:\path\to\your\gemma-4-E4B-it-Q4_K_M.ggufpython app.pyYou should see:
==================================================
GazeConnect — Starting Services
==================================================
[Memory] ✅ Supermemory Local connected at http://localhost:6767
[Memory] Session: xxxxxxxx
[Model] Loading Gemma 4 ...
[Model] ✅ Gemma 4 loaded and ready!
Gaze tracking thread active. Starting Flask server...
* Running on http://127.0.0.1:5000
* Setup page: http://127.0.0.1:5000/setup
- Main App: http://127.0.0.1:5000
- Patient Profile Setup: http://127.0.0.1:5000/setup
- Calibrate — Look at the center of the screen and click "Start Calibration" when prompted
- Setup Profile — Go to
/setupand enter the patient's details (name, family, routine, preferences) - Start communicating — Use gaze zones to type keywords, switch to Suggestion Mode, or send to AI
| Method | Endpoint | Description |
|---|---|---|
GET |
/ |
Main communication UI |
GET |
/setup |
Patient profile setup page |
GET |
/api/gaze |
Returns current gaze zone and blink state |
POST |
/api/complete |
Sends keywords to LLM, returns completed sentence |
POST |
/api/suggest |
Returns top 4 word suggestions based on typed text |
POST |
/api/context |
Stores caretaker voice context in Supermemory |
GET |
/api/context |
Returns current caretaker context |
GET |
/api/model_status |
Returns LLM loading status |
GET |
/api/memory_status |
Returns Supermemory connection status and stats |
POST |
/api/profile |
Saves patient profile facts to Supermemory |
GET |
/api/profile |
Retrieves stored patient profile facts |
POST |
/calibrate/start |
Begins gaze calibration |
GET |
/calibrate/status |
Returns calibration sample count |
POST |
/calibrate/finish |
Finalizes calibration with median baseline |
| Action | Required Dwell | Description |
|---|---|---|
| Enter Zone (Main View) | 1.8 seconds | Opens the letter group |
| Select Letter (Sub View) | 1.5 seconds | Selects the displayed letter |
| Switch / Toggle | 1.0 second | Toggles mode or letter pair |
| Send to AI | 1.8 seconds | Triggers AI sentence generation |
┌──────────────┐ ┌──────────────┐ ┌──────────────────────┐
│ Webcam │────>│ MediaPipe │────>│ Gaze Zone Classifier │
│ │ │ Face Mesh │ │ (gaze_tracker.py) │
└──────────────┘ └──────────────┘ └──────────┬───────────┘
│
v
┌──────────────┐ ┌──────────────┐ ┌──────────────────────┐
│ Browser UI │<───>│ Flask API │<───>│ Supermemory Local │
│ (main.js) │ │ (app.py) │ │ (localhost:6767) │
└──────────────┘ └──────┬───────┘ └──────────────────────┘
│
v
┌──────────────┐
│ Gemma 4 LLM │
│ (llama-cpp) │
└──────────────┘
- Zero cloud dependency — All processing happens on the patient's machine
- No data leaves the device — Supermemory Local, Gemma 4, and gaze tracking all run locally
- No internet required — Once set up, the system works fully offline
- Patient data stays private — Medical information, family details, and conversation history never leave the local machine
- Completely hands-free interaction
- Eye-only typing with large, forgiving gaze zones
- Dedicated safe resting zone to prevent accidental inputs
- Blink-based controls for delete and cancel
- Local AI processing for offline and private operation
- Automatic speech output for natural communication
- Personalized calibration per user
- Caregiver-assisted context with easy clear functionality
- Predictive word suggestions that learn from usage history
- Visual feedback (glowing eyes, loading bars) for all gaze actions
- Memory dashboard to visualize communication trends
- Multi-language support
- Emotion-aware sentence generation
- Iris-based gaze estimation for improved accuracy
- Personalized language model fine-tuning
- Mobile/tablet companion app for caretakers
GazeConnect was built for the Supermemory Local Hackathon (#localhost:6767).
The goal: make communication faster, more natural, and more accessible for people with severe motor disabilities — by combining facial tracking, hierarchical gaze interaction, persistent AI memory, and speech synthesis. All while keeping every piece of data on the patient's own machine.