diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 28f48d0..fe69d16 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -16,7 +16,6 @@ jobs: uses: actions/setup-node@v4 with: node-version: 22 - cache: npm - name: Install dependencies run: npm install diff --git a/README.md b/README.md index 55b441f..6b8d969 100644 --- a/README.md +++ b/README.md @@ -2,11 +2,11 @@ # GSLHub Software -### Reusable tools for Generative Search, GEO and reproducible AI research +### Reusable software for Generative Search, GEO and reproducible AI research -**Research utilities and independently reusable components developed around GSLHub** +**Framework-independent tools extracted from the GSLHub research infrastructure** -[Platform](https://github.com/gslhub/website) · [Research](https://github.com/gslhub/research) · [Benchmarks](https://github.com/gslhub/benchmarks) +[Platform](https://github.com/gslhub/website) · [Research](https://github.com/gslhub/research) · [Benchmarks](https://github.com/gslhub/benchmarks) · [Docs](https://github.com/gslhub/docs) @@ -14,71 +14,113 @@ ## Purpose -`gslhub/software` is the public-facing home for **reusable research software** that does not need to live inside the main GSLHub platform repository. +`gslhub/software` is the home for **independently reusable research software** developed around **GSLHub — Generative Search Lab Hub**. -The main application remains in [`gslhub/website`](https://github.com/gslhub/website). This repository is intended for focused tools that can be installed, tested or reused independently. +The production research platform remains in [`gslhub/website`](https://github.com/gslhub/website). This repository contains focused tools and libraries that can be installed, tested and reused without requiring the full Next.js/Payload application. -## Candidate tool families +## First package — `@gslhub/metrics-core` -Future releases may include utilities for: +[`packages/metrics-core`](packages/metrics-core) contains deterministic implementations of the first four governed GSLHub visibility metrics: -- benchmark validation; -- metric calculation; -- citation/source normalization; -- research-artifact hashing and manifest generation; -- reproducibility checks; -- dataset export/validation; -- schema validation; -- controlled comparison of benchmark outputs. +| Code | Metric | Core calculation | +|---|---|---| +| **AIR** | Answer Inclusion Rate | target-included executions / eligible executions | +| **CR** | Citation Rate | target-cited executions / eligible executions | +| **MCP** | Mean Citation Position | mean first valid target-citation position | +| **RCR** | Response Consistency Rate | consistent comparisons / eligible baseline comparisons | + +The package is written in **TypeScript**, has **no application-framework or database dependency**, and preserves auditable calculation outputs including exclusions, numerator/denominator data and SHA-256 input/output checksums. -Only tools that are sufficiently independent, documented and safe to reuse should be moved here. +Its normative metric definitions live in [`gslhub/benchmarks`](https://github.com/gslhub/benchmarks); protocols and coding rules live in [`gslhub/research`](https://github.com/gslhub/research). -## Planned structure +### Validation + +The package tests reproduce the public synthetic benchmark fixture: + +```text +AIR = 0.75 +CR = 0.50 +MCP = 2.00 +RCR = 0.75 +``` + +These are **synthetic software-validation values**, not empirical research findings. + +See [`packages/metrics-core/README.md`](packages/metrics-core/README.md) for the API, scope and usage examples. + +## Repository structure ```text software/ -├── packages/ # Reusable libraries -├── tools/ # Standalone research utilities -├── examples/ # Minimal usage examples -├── templates/ # New-tool documentation templates -├── SECURITY.md # Repository-specific security notes if needed +├── packages/ +│ └── metrics-core/ # AIR, CR, MCP and RCR calculation library +├── tools/ # Future standalone research utilities +├── templates/ # New-tool documentation templates +├── .github/workflows/ # Automated validation +├── RELEASE-POLICY.md └── README.md ``` -## Relationship to the platform - -The repositories serve different purposes: +## Relationship to the GSLHub ecosystem -- **`website`** — production research platform and governed application logic; -- **`software`** — small independently reusable tools and libraries; -- **`research`** — methodology and protocols; -- **`benchmarks`** — evaluation specifications; -- **`datasets`** — reviewed data releases. +- **[`website`](https://github.com/gslhub/website)** — production research platform and governed application logic; +- **[`research`](https://github.com/gslhub/research)** — scientific methodology, protocols and codebooks; +- **[`benchmarks`](https://github.com/gslhub/benchmarks)** — benchmark and metric specifications; +- **`software`** — independently reusable implementations and utilities; +- **`datasets`** — reviewed research-data releases when they are ready for publication. -Moving a utility here should not break the audit trail of the platform version that originally used it. +Separating methodology, software and application integration makes it possible to test calculations independently while preserving the audit trail of the platform version that used them. ## Release requirements -A reusable tool should include: +A reusable tool must include: - a clear problem statement; -- supported inputs and outputs; +- explicit supported inputs and outputs; - versioned behavior; -- tests or deterministic validation where applicable; -- installation/usage instructions; -- limitations; +- deterministic tests or validation where applicable; +- installation and usage instructions; +- limitations and out-of-scope behavior; - explicit license; -- security considerations when it handles files, URLs or external services. +- security considerations when relevant; +- traceability to the methodological specification it implements. -## Licensing +See [`RELEASE-POLICY.md`](RELEASE-POLICY.md). -The intended default for original GSLHub research software is **GNU AGPL-3.0-only**, unless a specific package states otherwise. +## Development -Third-party dependencies retain their own licenses. +Requirements: + +- Node.js `>=20.9.0` +- npm `10.x` + +```bash +npm install +npm run typecheck +npm test +npm run build +``` + +The repository uses npm workspaces so additional independent packages can be added under `packages/` without coupling them to the main GSLHub platform. + +## Planned tool families + +Future releases may include: + +- citation/source normalization; +- research-artifact hashing and manifest generation; +- reproducibility checks; +- dataset export/validation; +- schema validation; +- controlled benchmark-output comparison. + +Tools are added only when they are sufficiently independent, documented, tested and safe to reuse. + +## Licensing -## Current status +Original GSLHub software in this repository is released under **GNU AGPL-3.0-only**, unless a specific package states otherwise. -This repository currently defines the publication structure only. No standalone tool is presented as released software until its code, tests and documentation are migrated here deliberately. +Third-party dependencies retain their own licenses. GSLHub brand assets and trademarks are governed separately. --- diff --git a/packages/metrics-core/README.md b/packages/metrics-core/README.md new file mode 100644 index 0000000..c535e5c --- /dev/null +++ b/packages/metrics-core/README.md @@ -0,0 +1,147 @@ +# @gslhub/metrics-core + +Deterministic, framework-independent implementations of the first four governed GSLHub visibility metrics: + +- **AIR** — Answer Inclusion Rate +- **CR** — Citation Rate +- **MCP** — Mean Citation Position +- **RCR** — Response Consistency Rate + +`@gslhub/metrics-core` extracts the calculation layer from the GSLHub research platform so the same metric rules can be reused from scripts, notebooks, validation pipelines or other applications without depending on Payload CMS, MongoDB or Next.js. + +> **Status:** `0.1.0` — initial public-review implementation. + +## Methodology + +The normative metric specifications are maintained in [`gslhub/benchmarks`](https://github.com/gslhub/benchmarks): + +- [`AIR v0.1.0`](https://github.com/gslhub/benchmarks/blob/main/metrics/AIR-v0.1.0.md) +- [`CR v0.1.0`](https://github.com/gslhub/benchmarks/blob/main/metrics/CR-v0.1.0.md) +- [`MCP v0.1.0`](https://github.com/gslhub/benchmarks/blob/main/metrics/MCP-v0.1.0.md) +- [`RCR v0.1.0`](https://github.com/gslhub/benchmarks/blob/main/metrics/RCR-v0.1.0.md) + +Scientific protocols and coding rules live in [`gslhub/research`](https://github.com/gslhub/research). + +The software implementation does **not** replace those methodological documents. A versioned result should identify both the metric-definition version and the calculator version used. + +## What this package does + +The package accepts already-structured observations and: + +- applies the GSLHub `0.1.0` eligibility rules; +- normalizes target type and common domain/URL forms; +- rejects duplicate eligible observations for the same execution; +- reports excluded candidates and their reasons; +- computes numerator/denominator or eligible positions; +- applies the metric-specific rounding rule; +- generates deterministic SHA-256 checksums for normalized inputs and outputs; +- returns an auditable JSON query snapshot describing the calculation conditions. + +## What this package does not do + +It does **not**: + +- execute prompts against AI systems; +- interpret raw generated text; +- decide whether a mention, citation or recommendation exists; +- validate source quality or factual support; +- fetch Payload or database records; +- replace evidence preservation, codebooks or human review. + +Those responsibilities belong to the research protocol and data-governance layers. + +## Installation + +The package is currently developed inside the `gslhub/software` workspace and has not yet been published to the npm registry. + +From a clone of this repository: + +```bash +npm install +npm run build +npm test +``` + +## Basic usage + +```ts +import { calculateAIR } from '@gslhub/metrics-core' + +const result = calculateAIR({ + targetType: 'domain', + targetValue: 'https://www.gslhub.com/', + observations: [ + { + id: 'obs-1', + executionId: 'exec-1', + executionLifecycle: 'completed', + observationLifecycle: 'validated', + reviewStatus: 'accepted', + targetType: 'domain', + targetValue: 'gslhub.com', + mentioned: true, + }, + ], +}) + +console.log(result.numericValue) // 1 +console.log(result.numerator) // 1 +console.log(result.denominator) // 1 +``` + +The same observation shape can carry `cited`, `citationPosition`, `baselineObservationId` and `variationLevel` for CR, MCP and RCR. + +## Public API + +```ts +calculateAIR(input) +calculateCR(input) +calculateMCP(input) +calculateRCR(input) +normalizeTargetType(value) +normalizeTargetValue(value) +``` + +All four calculators are synchronous and have no runtime dependencies outside Node.js built-ins. + +## Metric behavior + +| Metric | Primary outcome | Default precision | Undefined when | +|---|---|---:|---| +| AIR | mentioned executions / eligible executions | 4 | no eligible observations | +| CR | cited executions / eligible executions | 4 | no eligible observations | +| MCP | mean first valid citation position | 2 | no eligible cited position | +| RCR | `none` + `low` comparisons / eligible comparisons | 4 | no eligible comparisons | + +For RCR, the baseline is preserved in the candidate set but does not enter the denominator. + +## Validation fixture + +The automated tests reproduce the public synthetic calculator fixture in [`gslhub/benchmarks`](https://github.com/gslhub/benchmarks/blob/main/fixtures/synthetic-validation.json): + +```text +AIR = 3 / 4 = 0.75 +CR = 2 / 4 = 0.50 +MCP = mean(1, 2, 3) = 2.00 +RCR = 3 / 4 = 0.75 +``` + +These values are **synthetic software-validation data only**. They do not describe any AI system and must not be presented as empirical research findings. + +## Lineage + +The initial implementation was extracted from the governed calculators used by [`gslhub/website`](https://github.com/gslhub/website/tree/main/cms/metrics) and refactored to remove application/database dependencies while preserving the calculation rules. + +The platform remains responsible for retrieving and validating governed records. `metrics-core` is responsible only for deterministic calculation over an explicit input snapshot. + +## Versioning + +Metric behavior is versioned independently from the package release. The initial package implements metric definitions `0.1.0`. + +A change that alters eligibility, numerator/denominator construction, position rules, consistency classification or output semantics requires explicit methodological version review rather than a silent implementation change. + +## License + +GNU Affero General Public License v3.0 only (`AGPL-3.0-only`). + +© 2026 GSLHub / Eduardo Yauri diff --git a/packages/metrics-core/src/mcp.ts b/packages/metrics-core/src/mcp.ts index 4e7de64..8c1d422 100644 --- a/packages/metrics-core/src/mcp.ts +++ b/packages/metrics-core/src/mcp.ts @@ -57,12 +57,12 @@ export const calculateMCP = ({ continue; } - if (!Number.isInteger(observation.citationPosition) || Number(observation.citationPosition) < 1) { + const position = observation.citationPosition; + if (typeof position !== 'number' || !Number.isInteger(position) || position < 1) { excludedCandidates.push(exclusion(observation, 'The evaluated target is cited, but no valid one-based citation position was recorded.')); continue; } - const position = Number(observation.citationPosition); validObservationIds.push(observation.id); validExecutionIds.push(observation.executionId); eligiblePositions.push(position); diff --git a/packages/metrics-core/test/metrics.test.ts b/packages/metrics-core/test/metrics.test.ts index d03e4b8..d9ef12a 100644 --- a/packages/metrics-core/test/metrics.test.ts +++ b/packages/metrics-core/test/metrics.test.ts @@ -27,68 +27,76 @@ const base = (overrides: Partial): MetricObservation => ({ ...overrides, }); -const observations: MetricObservation[] = [ - base({ - id: 'obs-1', observationCode: 'OBS-1', executionId: 'exec-1', executionCode: 'EXEC-1', - mentioned: true, cited: true, citationPosition: 1, - }), - base({ - id: 'obs-2', observationCode: 'OBS-2', executionId: 'exec-2', executionCode: 'EXEC-2', - mentioned: true, cited: false, baselineObservationId: 'obs-1', variationLevel: 'low', - }), - base({ - id: 'obs-3', observationCode: 'OBS-3', executionId: 'exec-3', executionCode: 'EXEC-3', - mentioned: true, cited: true, citationPosition: 3, baselineObservationId: 'obs-1', variationLevel: 'medium', - }), - base({ - id: 'obs-4', observationCode: 'OBS-4', executionId: 'exec-4', executionCode: 'EXEC-4', - mentioned: false, cited: false, baselineObservationId: 'obs-1', variationLevel: 'none', - }), - base({ - id: 'obs-5', observationCode: 'OBS-5', executionId: 'exec-5', executionCode: 'EXEC-5', - mentioned: false, cited: false, baselineObservationId: 'obs-1', variationLevel: 'high', - }), -]; +const metricInput = (observations: MetricObservation[]) => ({ + observations, + targetType: 'domain', + targetValue: 'https://www.gslhub.com/', +}); -const input = { observations, targetType: 'domain', targetValue: 'https://www.gslhub.com/' }; +test('AIR reproduces the public synthetic fixture: 3/4 = 0.75', () => { + const observations = [ + base({ id: 'air-1', executionId: 'air-exec-1', mentioned: true }), + base({ id: 'air-2', executionId: 'air-exec-2', mentioned: true }), + base({ id: 'air-3', executionId: 'air-exec-3', mentioned: true }), + base({ id: 'air-4', executionId: 'air-exec-4', mentioned: false }), + ]; -test('AIR computes answer inclusion proportion', () => { - const result = calculateAIR(input); + const result = calculateAIR(metricInput(observations)); assert.equal(result.metricCode, 'AIR'); assert.equal(result.numerator, 3); - assert.equal(result.denominator, 5); - assert.equal(result.numericValue, 0.6); + assert.equal(result.denominator, 4); + assert.equal(result.numericValue, 0.75); assert.equal(result.excludedCount, 0); assert.equal(result.inputChecksum.length, 64); assert.equal(result.outputChecksum.length, 64); }); -test('CR computes target citation proportion', () => { - const result = calculateCR(input); +test('CR reproduces the public synthetic fixture: 2/4 = 0.50', () => { + const observations = [ + base({ id: 'cr-1', executionId: 'cr-exec-1', cited: true }), + base({ id: 'cr-2', executionId: 'cr-exec-2', cited: true }), + base({ id: 'cr-3', executionId: 'cr-exec-3', cited: false }), + base({ id: 'cr-4', executionId: 'cr-exec-4', cited: false }), + ]; + + const result = calculateCR(metricInput(observations)); assert.equal(result.metricCode, 'CR'); assert.equal(result.numerator, 2); - assert.equal(result.denominator, 5); - assert.equal(result.numericValue, 0.4); + assert.equal(result.denominator, 4); + assert.equal(result.numericValue, 0.5); }); -test('MCP uses only cited observations with valid one-based positions', () => { - const result = calculateMCP(input); +test('MCP reproduces the public synthetic fixture: mean([1,2,3]) = 2.00', () => { + const observations = [ + base({ id: 'mcp-1', executionId: 'mcp-exec-1', cited: true, citationPosition: 1 }), + base({ id: 'mcp-2', executionId: 'mcp-exec-2', cited: true, citationPosition: 2 }), + base({ id: 'mcp-3', executionId: 'mcp-exec-3', cited: true, citationPosition: 3 }), + ]; + + const result = calculateMCP(metricInput(observations)); assert.equal(result.metricCode, 'MCP'); - assert.equal(result.positionSum, 4); - assert.equal(result.denominator, 2); + assert.equal(result.positionSum, 6); + assert.equal(result.denominator, 3); assert.equal(result.numericValue, 2); - assert.deepEqual(result.eligiblePositions, [1, 3]); - assert.equal(result.excludedCount, 3); + assert.deepEqual(result.eligiblePositions, [1, 2, 3]); }); -test('RCR excludes the frozen baseline and scores none/low as consistent', () => { - const result = calculateRCR(input); +test('RCR reproduces the public synthetic fixture: 3/4 = 0.75', () => { + const observations = [ + base({ id: 'rcr-base', executionId: 'rcr-exec-1', variationLevel: 'not-assessed' }), + base({ id: 'rcr-2', executionId: 'rcr-exec-2', baselineObservationId: 'rcr-base', variationLevel: 'none' }), + base({ id: 'rcr-3', executionId: 'rcr-exec-3', baselineObservationId: 'rcr-base', variationLevel: 'low' }), + base({ id: 'rcr-4', executionId: 'rcr-exec-4', baselineObservationId: 'rcr-base', variationLevel: 'low' }), + base({ id: 'rcr-5', executionId: 'rcr-exec-5', baselineObservationId: 'rcr-base', variationLevel: 'high' }), + ]; + + const result = calculateRCR(metricInput(observations)); assert.equal(result.metricCode, 'RCR'); - assert.equal(result.baselineObservationId, 'obs-1'); - assert.equal(result.numerator, 2); + assert.equal(result.baselineObservationId, 'rcr-base'); + assert.equal(result.numerator, 3); assert.equal(result.denominator, 4); - assert.equal(result.numericValue, 0.5); - assert.deepEqual(result.assessedVariationLevels, ['low', 'medium', 'none', 'high']); + assert.equal(result.numericValue, 0.75); + assert.deepEqual(result.assessedVariationLevels, ['none', 'low', 'low', 'high']); }); test('target normalization keeps equivalent domain forms comparable', () => { @@ -100,6 +108,18 @@ test('target normalization keeps equivalent domain forms comparable', () => { assert.equal(result.numericValue, 1); }); +test('ineligible records are excluded and reported rather than silently discarded', () => { + const result = calculateAIR(metricInput([ + base({ id: 'valid', executionId: 'exec-valid', mentioned: true }), + base({ id: 'excluded', executionId: 'exec-excluded', mentioned: true, reviewStatus: 'under-review' }), + ])); + + assert.equal(result.numericValue, 1); + assert.equal(result.denominator, 1); + assert.equal(result.excludedCount, 1); + assert.match(result.excludedCandidates[0]?.reason ?? '', /not accepted/); +}); + test('duplicate eligible observations for one execution are rejected', () => { const duplicate = base({ id: 'obs-duplicate', observationCode: 'OBS-DUP', mentioned: true }); assert.throws(