Add structured PDF and Office understanding

.github/workflows/windows-distribution.yml

@@ -49,6 +49,10 @@ jobs:
           if ($LASTEXITCODE -ne 0 -or $version -ne "Interly 0.6.0") {
             throw "Standalone executable smoke test failed: $version"
           }
+          $documentSupport = & .\dist\interly.exe --check-document-support
+          if ($LASTEXITCODE -ne 0 -or $documentSupport -ne "Document support: PDF DOCX XLSX XLSM PPTX") {
+            throw "Standalone document backend smoke test failed: $documentSupport"
+          }
 
       - name: Install Inno Setup
         run: choco install innosetup --no-progress --yes

README.md

@@ -83,6 +83,9 @@ Development installations upgrade through pipx. Restart Interly after a successf
 - Automatic isolated-browser cleanup after every browser-assisted request
 - Guarded file search, bounded text reads, exact text edits, comparison, creation, copying,
   moving, renaming, and folder creation
+- Read-only structured understanding of PDF, Word (`.docx`), Excel (`.xlsx`/`.xlsm`), and
+  PowerPoint (`.pptx`) documents, including pages or slides, headings/text blocks, tables,
+  spreadsheet sheets/ranges/formulas, speaker notes, and metadata where the format exposes it
 - Approved direct-file downloads with public-URL validation, a 1 GB limit, overwrite protection,
   temporary-file cleanup, final content type, byte count, and SHA-256 reporting
 - Visible top-level Windows window enumeration with exact handles, PIDs, titles, and rectangles
@@ -140,16 +143,30 @@ Sensitive local-read prompts use a different meaning:
 Sensitive `A` approval applies to one command only; it is never remembered for the session.
 
 Raw process lists, application matches, IP and Wi-Fi configuration, users, routes, performance
-metrics, installed-application reports, desktop window listings, OCR results, control inspection,
-and clipboard reads default to local-only output. With `Y`, Interly sends Groq only a short
-completion status. With `A`, the user explicitly authorizes that one output to be included in the
-Groq conversation.
+metrics, installed-application reports, structured document contents, desktop window listings, OCR
+results, control inspection, and clipboard reads default to local-only output. With `Y`, Interly
+sends Groq only a short completion status. With `A`, the user explicitly authorizes that one output
+to be included in the Groq conversation.
 
 `set-free 1` through `set-free 30` temporarily removes repeated prompts for ordinary actions.
 `set-free 0` disables the window immediately. Emergency stop remains active, sensitive local output
 still stays local unless explicitly shared with `A`, and destructive Windows actions still require
 individual confirmation.
 
+## Structured document model
+
+Structured document reads reuse the same permission path as ordinary approved file reads. Interly
+opens supported documents read-only and converts their useful structure into bounded local output.
+PDF reads expose pages, extracted text, heuristic headings, tables, and metadata. Word reads preserve
+ordered headings, paragraphs, and tables. Excel reads expose sheets, used ranges, headers, rows, and
+formula cells. PowerPoint reads expose slides, titles, text blocks, tables, speaker notes, and core
+metadata.
+
+The reader limits input size, Office package entry count and uncompressed size, page/slide/sheet and
+table dimensions, and total extracted text. Corrupt or invalid Office Open XML packages fail without
+being parsed. Document contents remain terminal-only unless the user explicitly chooses `A` at the
+sensitive read approval prompt.
+
 ## Desktop interaction model
 
 Interly is instructed to inspect before acting. For window operations it first resolves exact native
@@ -173,16 +190,17 @@ other sensitive machine inspection tools.
 
 ## Important limitations
 
-Interly is alpha software. Model responses can be wrong, and read-only system or desktop output may
-contain private information. Review proposed actions. Forced process termination can lose unsaved
-work. Generic mouse and keyboard actions affect the currently visible/focused desktop state. Web
-searches and selected page text are sent to external services and Groq.
+Interly is alpha software. Model responses can be wrong, and read-only system, document, or desktop
+output may contain private information. Review proposed actions. Forced process termination can lose
+unsaved work. Generic mouse and keyboard actions affect the currently visible/focused desktop state.
+Web searches and selected page text are sent to external services and Groq.
 
 Personal-browser access, file deletion, webpage video extraction, streaming-platform downloads,
-structured PDF/Word/Excel/PowerPoint parsing, uploads, logins, purchases, messaging, volume control,
-brightness control, and speech input/output are not implemented. Direct downloads currently require
-a public URL that returns the file itself. The emergency stop prevents additional actions, but an
-operating-system call that has already completed cannot be reversed.
+uploads, logins, purchases, messaging, volume control, brightness control, and speech input/output
+are not implemented. Structured document understanding is read-only and currently supports modern
+Open XML Office formats rather than legacy `.doc`, `.xls`, or `.ppt` files. Direct downloads currently
+require a public URL that returns the file itself. The emergency stop prevents additional actions,
+but an operating-system call that has already completed cannot be reversed.
 
 ## Development
 

ROADMAP.md

@@ -6,7 +6,7 @@ Status rule:
 - ✅ = the complete item is implemented and verified.
 - ❌ = the item is not fully complete. Partial implementation still counts as ❌.
 
-Current verified status: **88/101 complete**.
+Current verified status: **89/101 complete**.
 
 ## 1. Core agent foundation
 
@@ -94,7 +94,7 @@ Current verified status: **88/101 complete**.
 64. ✅ Preview and apply exact text edits to existing files.
 65. ✅ Create, copy, move, and rename files or folders.
 66. ❌ Add file deletion with Windows Recycle Bin support.
-67. ❌ Add structured PDF, Word, Excel, and PowerPoint understanding.
+67. ✅ Add structured PDF, Word, Excel, and PowerPoint understanding.
 68. ✅ Compare two approved text files and return their differences.
 69. ✅ Download direct public file URLs with type, size, hash, and destination approval.
 70. ❌ Add malware scanning and quarantine checks for downloads.

interly-priority-ownership-plan.md

@@ -75,27 +75,33 @@ Interly can inspect the visible Windows desktop, capture it, extract text throug
 # Priority 3 — Documents & Safe Filesystem Completion
 
 **Owner: Joint / Convergence Point**  
-**Status: Active — both streams meet here**
+**Status: Active — structured documents complete; filesystem safety remains**
 
 This is the current shared workstream after the parallel implementation passes.
 
 Goal: complete Interly's existing filesystem layer and expand it from text-file access into structured document work.
 
-1. [ ] **#67 — Structured PDF, Word, Excel and PowerPoint understanding**
+1. [x] **#67 — Structured PDF, Word, Excel and PowerPoint understanding**
+   - PDF pages, text, heuristic headings, tables and metadata
+   - Word ordered headings, paragraphs, tables and metadata
+   - Excel sheets, used ranges, headers, rows and formulas
+   - PowerPoint slides, titles, text blocks, tables, speaker notes and metadata
+   - read-only and local-only by default through the existing sensitive file-read approval path
+   - verified in the packaged Windows executable
 2. [ ] **#66 — File deletion through the Windows Recycle Bin**
 3. [ ] **#70 — Malware scanning and quarantine checks for downloads**
 
-### Intended file flow
+### Current file flow
 
-`find → read → understand → compare → create/edit → move/copy → delete safely`
+`find → read → understand ✅ → compare → create/edit → move/copy → delete safely`
 
-### Intended download flow
+### Remaining download flow
 
 `download → inspect → scan → accept / quarantine`
 
-### Completion outcome
+### Current outcome
 
-Interly can work with common structured documents, complete ordinary filesystem management safely, and treat downloaded files as a security-sensitive pipeline.
+Interly can now understand common structured documents instead of treating them as opaque files. Priority 3 remains active until ordinary file deletion uses the Windows Recycle Bin and downloaded files have a malware scanning/quarantine path.
 
 ---
 
@@ -165,7 +171,7 @@ Goal: turn Interly's existing machine and file capabilities into a coherent soft
 **Status: After Priority 3 convergence**
 
 1. [ ] **#78 — Volume and mute control**
-2. [ ] **#79 — Brightness control**
+2. [ ] **#79 — Brightness control where supported**
 3. [ ] **#80 — Speech input**
 4. [ ] **#81 — Speech output and optional wake phrase**
 
@@ -194,7 +200,7 @@ This comes after the major capability and security architecture stabilises so th
 |---|---|---|---|
 | 1 | Execution Governance | **ChatGPT** | Complete ✅ |
 | 2 | Desktop Perception & Interaction | **ChatGPT** | Complete ✅ |
-| 3 | Documents & Safe Filesystem Completion | **Joint** | **Active convergence point** |
+| 3 | Documents & Safe Filesystem Completion | **Joint** | **Active — #67 complete; #66/#70 open** |
 | 4 | Persistent Memory & Reusable Work | **User** | Checkpoint complete; #88 open |
 | 5 | Developer Agent | **User** | Checkpoint complete; #91 and #98 open |
 | 6 | Release & Supply-Chain Security | Deferred | After Priority 3 |
@@ -210,7 +216,7 @@ Completed parallel passes:
 
 Current shared position:
 
-**Both:** `→ Priority 3`
+**Both:** `→ Priority 3 (#67 ✅; #66/#70 next)`
 
 ---
 

packaging/interlink.spec

@@ -7,6 +7,10 @@ rapidocr_data, rapidocr_binaries, rapidocr_hidden = collect_all("rapidocr")
 onnx_data, onnx_binaries, onnx_hidden = collect_all("onnxruntime")
 pillow_data, pillow_binaries, pillow_hidden = collect_all("PIL")
 uia_data, uia_binaries, uia_hidden = collect_all("uiautomation")
+pdf_data, pdf_binaries, pdf_hidden = collect_all("pdfplumber")
+docx_data, docx_binaries, docx_hidden = collect_all("docx")
+excel_data, excel_binaries, excel_hidden = collect_all("openpyxl")
+pptx_data, pptx_binaries, pptx_hidden = collect_all("pptx")
 
 a = Analysis(
     ["entrypoint.py"],
@@ -18,6 +22,10 @@ a = Analysis(
         + onnx_binaries
         + pillow_binaries
         + uia_binaries
+        + pdf_binaries
+        + docx_binaries
+        + excel_binaries
+        + pptx_binaries
     ),
     datas=(
         playwright_data
@@ -26,7 +34,15 @@ a = Analysis(
         + onnx_data
         + pillow_data
         + uia_data
+        + pdf_data
+        + docx_data
+        + excel_data
+        + pptx_data
         + copy_metadata("interly")
+        + copy_metadata("pdfplumber")
+        + copy_metadata("python-docx")
+        + copy_metadata("openpyxl")
+        + copy_metadata("python-pptx")
     ),
     hiddenimports=(
         playwright_hidden
@@ -35,6 +51,10 @@ a = Analysis(
         + onnx_hidden
         + pillow_hidden
         + uia_hidden
+        + pdf_hidden
+        + docx_hidden
+        + excel_hidden
+        + pptx_hidden
     ),
     noarchive=False,
 )

pyproject.toml

@@ -20,12 +20,16 @@ dependencies = [
     "beautifulsoup4==4.15.0",
     "groq==1.6.0",
     "onnxruntime==1.27.0",
+    "openpyxl==3.1.5",
+    "pdfplumber==0.11.7",
     "pillow==12.3.0",
     "playwright==1.62.0",
     "psutil==7.2.2",
     "pynput==1.8.2",
     "pydantic-settings==2.14.2",
+    "python-docx==1.2.0",
     "python-dotenv==1.2.2",
+    "python-pptx==1.0.2",
     "rapidocr==3.9.2",
     "uiautomation==2.0.29",
 ]

requirements.txt

@@ -1,12 +1,16 @@
 beautifulsoup4==4.15.0
 groq==1.6.0
 onnxruntime==1.27.0
+openpyxl==3.1.5
+pdfplumber==0.11.7
 pillow==12.3.0
 playwright==1.62.0
 psutil==7.2.2
 pynput==1.8.2
 pydantic-settings==2.14.2
+python-docx==1.2.0
 python-dotenv==1.2.2
+python-pptx==1.0.2
 rapidocr==3.9.2
 uiautomation==2.0.29
 # Development and verification

src/computer_agent/__main__.py

@@ -10,6 +10,7 @@
 from computer_agent.browser import BROWSER
 from computer_agent.chat import run_chat
 from computer_agent.config import load_settings, save_api_key
+from computer_agent.documents import document_support_status
 from computer_agent.emergency import EmergencyStop
 from computer_agent.models import GroqModel
 from computer_agent.updater import update_interly
@@ -49,6 +50,9 @@ def main() -> None:
     if "--version" in sys.argv[1:]:
         print(f"Interly {__version__}")
         return
+    if "--check-document-support" in sys.argv[1:]:
+        print(document_support_status())
+        return
 
     settings = load_settings()
     if not settings.groq_api_key:
@@ -69,6 +73,7 @@ def reconfigure_groq() -> bool:
             return False
         model.update_api_key(api_key)
         return True
+
     emergency_stop = EmergencyStop()
     if emergency_stop.start():
         print("Emergency stop: press Esc to cancel the current request.")