CorpusLens

CorpusLens is a small C#/.NET tool for building and exploring language corpora from EPUB and text files.

It is CLI-first, testable, and stores analysis data in SQLite so that corpora can be queried after import.

Features

• EPUB and plain-text analysis
• English, Italian, French and German language profiles
• word frequencies, content/function word views, n-grams and next-word statistics
• word detail with previous/next words
• KWIC contexts
• source-book lists for aggregate corpus runs
• desktop chapter browsing with persisted clean-text preview and in-chapter search
• word distribution by book
• word comparison between analysis runs
• compact corpus profiles for quick run validation
• persistent token index for saved analysis runs
• relative difficulty profiles for analysis runs
• collocations with content/function filters
• repeated phrase mining
• import diagnostics for EPUB folders
• desktop reports and exports explorer with safe operating-system opening
• desktop corpus management with language validation and run filtering

Requirements

• .NET 10 SDK
• make available from PowerShell/Git Bash/MSYS2 or similar

Architecture note

CorpusLens keeps analysis and storage logic outside the UI/CLI surfaces. Read-only orchestration for run lists, source books, corpus profiles, health checks and artifact-path resolution lives in CorpusLens.Application/Queries, while corpus creation and listing use dedicated application use cases in CorpusLens.Application/Storage. The CLI and Avalonia desktop app therefore share the same storage boundary. The desktop layer uses feature-specific ViewModels coordinated by MainWindowViewModel; global busy/status state and cancellation are centralized. Opening a report or export is isolated in the desktop system-path launcher and never mutates the generated file.

Project layout

books/
  en/        English EPUB files, ignored by git
  it/        Italian EPUB files, ignored by git
src/         application source
tests/       xUnit tests
docs/        technical notes and roadmap
data/        local SQLite database, generated
artifacts/   reports and extracted text, generated

Generated folders are intentionally not versioned.

Quick start

make check
make setup-books

Put EPUB files in books/en or books/it, then create and analyze a corpus:

make corpus-create-en
make analyze-en

For Italian:

make corpus-create-it
make analyze-it

make check deletes ./data and ./artifacts. Do not run it between two analyses if you want both corpora in the same SQLite database.

To analyze English and Italian into the same database:

make clean
make corpus-create-en
make corpus-create-it
make analyze-en
make analyze-it

Common commands

make stats-runs LIMIT=10
make stats-summary RUN=1
make stats-profile RUN=1 LIMIT=10 PHRASE_LIMIT=10
make stats-health RUN=1
make stats-books RUN=1
make stats-token-index RUN=1
make stats-words RUN=1 LIMIT=25
make stats-content RUN=1 LIMIT=25
make stats-function RUN=1 LIMIT=25

Word-level queries:

make stats-word RUN=1 WORD="alice" LIMIT=25
make stats-word-books RUN=1 WORD="whale" LIMIT=30
make stats-kwic RUN=1 WORD="alice" LIMIT=10 CONTEXT=8
make stats-next RUN=1 WORD="don't" LIMIT=25

Run health and token index diagnostics:

make stats-health RUN=1
make stats-token-index RUN=1

stats health is the compact check. stats token-index is the detailed token-index report.

Token-index-backed queries, when the run is indexed:

make stats-kwic RUN=1 WORD="piazza" LIMIT=10 CONTEXT=8
make stats-collocations-content RUN=1 WORD="piazza" WINDOW=4 LIMIT=30 MIN_COUNT=1
make stats-phrases-content-boundary RUN=1 MIN_N=2 MAX_N=5 MIN_COUNT=3 MIN_CHAPTERS=2 LONGEST_ONLY=--longest-only LIMIT=30
make stats-word-books RUN=1 WORD="piazza" LIMIT=30

The token index is saved when an analysis run is persisted to SQLite. KWIC, collocations, phrase mining, and word-book distribution use it when available and fall back to the chapter-text path for legacy runs.

Compare runs:

make stats-compare-word RUN_A=1 RUN_B=2 WORD="love"
make stats-compare-words-content RUN_A=1 RUN_B=2 LIMIT=30 MIN_COUNT=5
make stats-compare-words-content RUN_A=1 RUN_B=2 LIMIT=30 MIN_COUNT=5 SHARED_ONLY=--shared-only
make stats-compare-words-content RUN_A=1 RUN_B=2 LIMIT=30 MIN_COUNT=5 EXCLUSIVE_ONLY=--exclusive-only

Comparisons are lexical. If two runs use different languages, CorpusLens prints a note and does not translate equivalent concepts.

Language profiles:

make stats-language-profiles
make stats-language-profile LANG=it

Difficulty:

make stats-difficulty RUN=1
make stats-compare-difficulty RUN_A=1 RUN_B=2
make stats-difficulty RUN=1 LONG_WORD_LENGTH=8 VERY_LONG_WORD_LENGTH=12

Difficulty is a relative heuristic based on sentence length, word length, long-word share, content-word share and lexical diversity. By default it uses the run language profile for long-word thresholds. It is useful for comparing similar corpora, not as an absolute reading-grade formula.

Collocations:

make stats-collocations RUN=1 WORD="whale" WINDOW=4 LIMIT=30
make stats-collocations-content RUN=1 WORD="whale" WINDOW=4 LIMIT=30 MIN_COUNT=3
make stats-collocations-function RUN=1 WORD="love" WINDOW=4 LIMIT=30 MIN_COUNT=3

Collocations are ranked with a lightweight Dice score while still showing raw counts. Use MIN_COUNT and MIN_DICE to hide weak low-frequency matches. Indexed runs use persisted token positions for the collocation window.

Phrases:

make stats-phrases RUN=1 MIN_N=2 MAX_N=5 MIN_COUNT=3 LIMIT=30
make stats-phrases-content-boundary RUN=1 MIN_N=2 MAX_N=5 MIN_COUNT=3 MIN_CHAPTERS=2 LIMIT=30
make stats-phrases-content-boundary RUN=1 MIN_N=2 MAX_N=5 MIN_COUNT=3 LONGEST_ONLY=--longest-only LIMIT=30

Import diagnostics:

make inspect-run RUN=1

Direct CLI examples

dotnet run --project src/CorpusLens.Cli -- corpus create "English Literature" --language en --db ./data/corpuslens.db

dotnet run --project src/CorpusLens.Cli -- analyze-epub-folder ./books/en --language en --corpus "English Literature" --db ./data/corpuslens.db --out ./artifacts/en

dotnet run --project src/CorpusLens.Cli -- stats profile 1 --limit 10 --phrase-limit 10 --db ./data/corpuslens.db

dotnet run --project src/CorpusLens.Cli -- stats health 1 --db ./data/corpuslens.db

dotnet run --project src/CorpusLens.Cli -- stats token-index 1 --db ./data/corpuslens.db

dotnet run --project src/CorpusLens.Cli -- stats word-books 1 "whale" --limit 30 --db ./data/corpuslens.db

dotnet run --project src/CorpusLens.Cli -- stats compare-words 1 2 --content-only --min-count 5 --limit 30 --db ./data/corpuslens.db

dotnet run --project src/CorpusLens.Cli -- stats difficulty 1 --db ./data/corpuslens.db

Desktop shell

The Avalonia desktop project is available as an early shell:

make desktop

The desktop app can open an existing corpuslens.db, remember the last database and up to eight recent database paths, list and filter corpora, create a corpus with a validated language, and run a complete EPUB-folder analysis without using the terminal. The analysis panel restores the latest non-sensitive folder preferences, supports recursive scanning, displays stage and file counters, allows cooperative cancellation, opens import diagnostics, refreshes the run list and selects the completed run automatically. Corpus creation and EPUB analysis are explicit persistent writes and require confirmation in the UI. Supported corpus languages are English, Italian, French and German. The app also browses ordered source books and chapters, previews persisted clean text, displays a compact run dashboard, and includes word, n-gram, collocation, phrase, comparison, and reports/exports explorers. Database, explorer and analysis operations run asynchronously so the UI stays responsive. Daily diagnostic logs are available from the top bar.

Windows x64 package

Build a self-contained portable package with:

powershell -NoProfile -ExecutionPolicy Bypass -File .\scripts\publish-win-x64.ps1

or:

make publish-win-x64

The script creates dist/CorpusLens-win-x64-18.15.0.zip and its SHA-256 file. It performs a safety scan and refuses to package SQLite databases, EPUB files, extracted text or local import diagnostics. The self-contained package can run on Windows x64 without the .NET SDK.

Desktop settings and logs are stored outside the application directory:

%LOCALAPPDATA%\CorpusLens\settings.json
%LOCALAPPDATA%\CorpusLens\logs\corpuslens-YYYYMMDD.log

See docs/distribution-windows.md and docs/user-guide-desktop.md.

Development

dotnet restore
dotnet build
dotnet test

or simply:

make check

Documentation

• docs/technical-design.md
• docs/analysis-rules.md
• docs/roadmap.md
• docs/milestone-18-9-desktop-architecture-consolidation.md
• docs/milestone-18-10-chapters-explorer.md
• docs/milestone-18-11-ngram-explorer.md
• docs/milestone-18-12-report-export.md
• docs/milestone-18-13-corpus-management.md
• docs/milestone-18-14-desktop-epub-analysis.md
• docs/milestone-18-15-stabilization-distribution.md
• docs/user-guide-desktop.md
• docs/distribution-windows.md

Desktop word explorer

The Avalonia desktop app can search a word in the selected run and show word summary, next/previous words, KWIC contexts and source-book distribution.

Desktop UI

Run the Avalonia desktop shell with:

make desktop

The desktop UI can open an existing corpuslens.db, restore the last database, reopen recent databases, list and create corpora, analyze EPUB folders into the selected corpus, filter analysis runs by corpus, browse source books and their metadata, inspect ordered chapters and persisted clean text, show a run dashboard, inspect and open generated reports/exports, search words, explore n-grams, explore collocations, explore recurring phrases, and compare runs. Window size and the latest EPUB input/output preferences are stored locally, while recoverable and critical errors are written to daily diagnostic logs.

Desktop corpus management

The corpus panel lists every persisted corpus and an All corpora filter. Selecting a corpus limits the run navigator to its associated runs and shows the corpus id, language, description and timestamps. New corpora can be created with a unique name, an optional description and one of the supported language profiles (en, it, fr, de). Empty names, duplicate names and unsupported language codes are rejected before writing. The confirmation checkbox must be selected for each persistent create operation.

Desktop compare runs

The Avalonia desktop app includes a Compare runs panel for lexical comparison between two analysis runs, including single-word comparison, top word differences, shared/exclusive filters, and difficulty comparison.

Desktop n-gram explorer

The n-gram panel can browse all stored sizes or focus on bigrams, trigrams, 4-grams or 5-grams. Results can be filtered by minimum count, an exact contained word or phrase, and content/function composition. Ordering is available by raw count, frequency per million, document count or text. The read-only results area supports normal text selection and Ctrl+C copying.

Desktop reports and exports

For the selected run, the Reports and exports panel resolves the paths stored in SQLite for report.md, words.csv, ngrams.csv, next_words.csv and extracted_text.txt. It also discovers import_diagnostics.md and import_failures.csv in the resolved output directory when present. A recorded path whose file has been removed is shown as Missing; an empty or optional unrecorded path is shown as Not generated. Only existing files and directories can be opened, and CorpusLens does not edit the exported artifacts.

Desktop EPUB-folder analysis

Select a specific corpus, choose an input folder and output folder, optionally enable recursive scanning, confirm the persistent operation, and click Analyze EPUB folder. The corpus language is used automatically and must match the analysis request. CorpusLens reports the active stage, percentage, processed/imported/skipped files, writes the same artifacts as analyze-epub-folder, saves the run and token index, refreshes the database view, and selects the new run. The latest output folder, import_diagnostics.md and import_failures.csv can be opened directly. Cancellation is cooperative; partial database books and runs are removed, while filesystem artifacts already written may remain in the selected output folder.