Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,6 @@ jobs:
uses: actions/setup-node@v4
with:
node-version: 22
cache: npm

- name: Install dependencies
run: npm install
Expand Down
126 changes: 84 additions & 42 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,83 +2,125 @@

# GSLHub Software

### Reusable tools for Generative Search, GEO and reproducible AI research
### Reusable software for Generative Search, GEO and reproducible AI research

**Research utilities and independently reusable components developed around GSLHub**
**Framework-independent tools extracted from the GSLHub research infrastructure**

[Platform](https://github.com/gslhub/website) · [Research](https://github.com/gslhub/research) · [Benchmarks](https://github.com/gslhub/benchmarks)
[Platform](https://github.com/gslhub/website) · [Research](https://github.com/gslhub/research) · [Benchmarks](https://github.com/gslhub/benchmarks) · [Docs](https://github.com/gslhub/docs)

</div>

---

## Purpose

`gslhub/software` is the public-facing home for **reusable research software** that does not need to live inside the main GSLHub platform repository.
`gslhub/software` is the home for **independently reusable research software** developed around **GSLHub — Generative Search Lab Hub**.

The main application remains in [`gslhub/website`](https://github.com/gslhub/website). This repository is intended for focused tools that can be installed, tested or reused independently.
The production research platform remains in [`gslhub/website`](https://github.com/gslhub/website). This repository contains focused tools and libraries that can be installed, tested and reused without requiring the full Next.js/Payload application.

## Candidate tool families
## First package — `@gslhub/metrics-core`

Future releases may include utilities for:
[`packages/metrics-core`](packages/metrics-core) contains deterministic implementations of the first four governed GSLHub visibility metrics:

- benchmark validation;
- metric calculation;
- citation/source normalization;
- research-artifact hashing and manifest generation;
- reproducibility checks;
- dataset export/validation;
- schema validation;
- controlled comparison of benchmark outputs.
| Code | Metric | Core calculation |
|---|---|---|
| **AIR** | Answer Inclusion Rate | target-included executions / eligible executions |
| **CR** | Citation Rate | target-cited executions / eligible executions |
| **MCP** | Mean Citation Position | mean first valid target-citation position |
| **RCR** | Response Consistency Rate | consistent comparisons / eligible baseline comparisons |

The package is written in **TypeScript**, has **no application-framework or database dependency**, and preserves auditable calculation outputs including exclusions, numerator/denominator data and SHA-256 input/output checksums.

Only tools that are sufficiently independent, documented and safe to reuse should be moved here.
Its normative metric definitions live in [`gslhub/benchmarks`](https://github.com/gslhub/benchmarks); protocols and coding rules live in [`gslhub/research`](https://github.com/gslhub/research).

## Planned structure
### Validation

The package tests reproduce the public synthetic benchmark fixture:

```text
AIR = 0.75
CR = 0.50
MCP = 2.00
RCR = 0.75
```

These are **synthetic software-validation values**, not empirical research findings.

See [`packages/metrics-core/README.md`](packages/metrics-core/README.md) for the API, scope and usage examples.

## Repository structure

```text
software/
├── packages/ # Reusable libraries
├── tools/ # Standalone research utilities
├── examples/ # Minimal usage examples
├── templates/ # New-tool documentation templates
├── SECURITY.md # Repository-specific security notes if needed
├── packages/
│ └── metrics-core/ # AIR, CR, MCP and RCR calculation library
├── tools/ # Future standalone research utilities
├── templates/ # New-tool documentation templates
├── .github/workflows/ # Automated validation
├── RELEASE-POLICY.md
└── README.md
```

## Relationship to the platform

The repositories serve different purposes:
## Relationship to the GSLHub ecosystem

- **`website`** — production research platform and governed application logic;
- **`software`** — small independently reusable tools and libraries;
- **`research`** — methodology and protocols;
- **`benchmarks`** — evaluation specifications;
- **`datasets`** — reviewed data releases.
- **[`website`](https://github.com/gslhub/website)** — production research platform and governed application logic;
- **[`research`](https://github.com/gslhub/research)** — scientific methodology, protocols and codebooks;
- **[`benchmarks`](https://github.com/gslhub/benchmarks)** — benchmark and metric specifications;
- **`software`** — independently reusable implementations and utilities;
- **`datasets`** — reviewed research-data releases when they are ready for publication.

Moving a utility here should not break the audit trail of the platform version that originally used it.
Separating methodology, software and application integration makes it possible to test calculations independently while preserving the audit trail of the platform version that used them.

## Release requirements

A reusable tool should include:
A reusable tool must include:

- a clear problem statement;
- supported inputs and outputs;
- explicit supported inputs and outputs;
- versioned behavior;
- tests or deterministic validation where applicable;
- installation/usage instructions;
- limitations;
- deterministic tests or validation where applicable;
- installation and usage instructions;
- limitations and out-of-scope behavior;
- explicit license;
- security considerations when it handles files, URLs or external services.
- security considerations when relevant;
- traceability to the methodological specification it implements.

## Licensing
See [`RELEASE-POLICY.md`](RELEASE-POLICY.md).

The intended default for original GSLHub research software is **GNU AGPL-3.0-only**, unless a specific package states otherwise.
## Development

Third-party dependencies retain their own licenses.
Requirements:

- Node.js `>=20.9.0`
- npm `10.x`

```bash
npm install
npm run typecheck
npm test
npm run build
```

The repository uses npm workspaces so additional independent packages can be added under `packages/` without coupling them to the main GSLHub platform.

## Planned tool families

Future releases may include:

- citation/source normalization;
- research-artifact hashing and manifest generation;
- reproducibility checks;
- dataset export/validation;
- schema validation;
- controlled benchmark-output comparison.

Tools are added only when they are sufficiently independent, documented, tested and safe to reuse.

## Licensing

## Current status
Original GSLHub software in this repository is released under **GNU AGPL-3.0-only**, unless a specific package states otherwise.

This repository currently defines the publication structure only. No standalone tool is presented as released software until its code, tests and documentation are migrated here deliberately.
Third-party dependencies retain their own licenses. GSLHub brand assets and trademarks are governed separately.

---

Expand Down
147 changes: 147 additions & 0 deletions packages/metrics-core/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
# @gslhub/metrics-core

Deterministic, framework-independent implementations of the first four governed GSLHub visibility metrics:

- **AIR** — Answer Inclusion Rate
- **CR** — Citation Rate
- **MCP** — Mean Citation Position
- **RCR** — Response Consistency Rate

`@gslhub/metrics-core` extracts the calculation layer from the GSLHub research platform so the same metric rules can be reused from scripts, notebooks, validation pipelines or other applications without depending on Payload CMS, MongoDB or Next.js.

> **Status:** `0.1.0` — initial public-review implementation.

## Methodology

The normative metric specifications are maintained in [`gslhub/benchmarks`](https://github.com/gslhub/benchmarks):

- [`AIR v0.1.0`](https://github.com/gslhub/benchmarks/blob/main/metrics/AIR-v0.1.0.md)
- [`CR v0.1.0`](https://github.com/gslhub/benchmarks/blob/main/metrics/CR-v0.1.0.md)
- [`MCP v0.1.0`](https://github.com/gslhub/benchmarks/blob/main/metrics/MCP-v0.1.0.md)
- [`RCR v0.1.0`](https://github.com/gslhub/benchmarks/blob/main/metrics/RCR-v0.1.0.md)

Scientific protocols and coding rules live in [`gslhub/research`](https://github.com/gslhub/research).

The software implementation does **not** replace those methodological documents. A versioned result should identify both the metric-definition version and the calculator version used.

## What this package does

The package accepts already-structured observations and:

- applies the GSLHub `0.1.0` eligibility rules;
- normalizes target type and common domain/URL forms;
- rejects duplicate eligible observations for the same execution;
- reports excluded candidates and their reasons;
- computes numerator/denominator or eligible positions;
- applies the metric-specific rounding rule;
- generates deterministic SHA-256 checksums for normalized inputs and outputs;
- returns an auditable JSON query snapshot describing the calculation conditions.

## What this package does not do

It does **not**:

- execute prompts against AI systems;
- interpret raw generated text;
- decide whether a mention, citation or recommendation exists;
- validate source quality or factual support;
- fetch Payload or database records;
- replace evidence preservation, codebooks or human review.

Those responsibilities belong to the research protocol and data-governance layers.

## Installation

The package is currently developed inside the `gslhub/software` workspace and has not yet been published to the npm registry.

From a clone of this repository:

```bash
npm install
npm run build
npm test
```

## Basic usage

```ts
import { calculateAIR } from '@gslhub/metrics-core'

const result = calculateAIR({
targetType: 'domain',
targetValue: 'https://www.gslhub.com/',
observations: [
{
id: 'obs-1',
executionId: 'exec-1',
executionLifecycle: 'completed',
observationLifecycle: 'validated',
reviewStatus: 'accepted',
targetType: 'domain',
targetValue: 'gslhub.com',
mentioned: true,
},
],
})

console.log(result.numericValue) // 1
console.log(result.numerator) // 1
console.log(result.denominator) // 1
```

The same observation shape can carry `cited`, `citationPosition`, `baselineObservationId` and `variationLevel` for CR, MCP and RCR.

## Public API

```ts
calculateAIR(input)
calculateCR(input)
calculateMCP(input)
calculateRCR(input)
normalizeTargetType(value)
normalizeTargetValue(value)
```

All four calculators are synchronous and have no runtime dependencies outside Node.js built-ins.

## Metric behavior

| Metric | Primary outcome | Default precision | Undefined when |
|---|---|---:|---|
| AIR | mentioned executions / eligible executions | 4 | no eligible observations |
| CR | cited executions / eligible executions | 4 | no eligible observations |
| MCP | mean first valid citation position | 2 | no eligible cited position |
| RCR | `none` + `low` comparisons / eligible comparisons | 4 | no eligible comparisons |

For RCR, the baseline is preserved in the candidate set but does not enter the denominator.

## Validation fixture

The automated tests reproduce the public synthetic calculator fixture in [`gslhub/benchmarks`](https://github.com/gslhub/benchmarks/blob/main/fixtures/synthetic-validation.json):

```text
AIR = 3 / 4 = 0.75
CR = 2 / 4 = 0.50
MCP = mean(1, 2, 3) = 2.00
RCR = 3 / 4 = 0.75
```

These values are **synthetic software-validation data only**. They do not describe any AI system and must not be presented as empirical research findings.

## Lineage

The initial implementation was extracted from the governed calculators used by [`gslhub/website`](https://github.com/gslhub/website/tree/main/cms/metrics) and refactored to remove application/database dependencies while preserving the calculation rules.

The platform remains responsible for retrieving and validating governed records. `metrics-core` is responsible only for deterministic calculation over an explicit input snapshot.

## Versioning

Metric behavior is versioned independently from the package release. The initial package implements metric definitions `0.1.0`.

A change that alters eligibility, numerator/denominator construction, position rules, consistency classification or output semantics requires explicit methodological version review rather than a silent implementation change.

## License

GNU Affero General Public License v3.0 only (`AGPL-3.0-only`).

© 2026 GSLHub / Eduardo Yauri
4 changes: 2 additions & 2 deletions packages/metrics-core/src/mcp.ts
Original file line number Diff line number Diff line change
Expand Up @@ -57,12 +57,12 @@ export const calculateMCP = ({
continue;
}

if (!Number.isInteger(observation.citationPosition) || Number(observation.citationPosition) < 1) {
const position = observation.citationPosition;
if (typeof position !== 'number' || !Number.isInteger(position) || position < 1) {
excludedCandidates.push(exclusion(observation, 'The evaluated target is cited, but no valid one-based citation position was recorded.'));
continue;
}

const position = Number(observation.citationPosition);
validObservationIds.push(observation.id);
validExecutionIds.push(observation.executionId);
eligiblePositions.push(position);
Expand Down
Loading
Loading