AISpec

RSpec for AI behavior, not AI outputs.

AISpec is the open-source engineering framework for defining, executing, and enforcing behavioral contracts for LLM applications in Ruby and Ruby on Rails.

Existing LLM evaluation tools tell you whether your AI performed well on a given run. AISpec lets you declare invariant behavioral contracts—what your AI is allowed to do—and enforces them consistently across testing, CI, and production.

---

Key Features

• Behavioral Invariants, Not Static Text Assertions: Test constraints like latency budgets, cost bounds, JSON schemas, system prompt leak protection, tool invocation ordering, and LLM-as-a-judge criteria.
• Provider Agnostic: Out-of-the-box support for OpenAI, Anthropic, Ollama (local, zero cost), OpenRouter, and custom provider adapters.
• Statistical Reliability: Statistical confidence bounds (Wilson score 95% CI), score variance ratings, and automated flaky detection.
• Rails First: Clean RSpec integration, Rails initializers, custom plugin DSL, and automated CI reporting.
• Multiple Reporting Formats: Terminal Console, JSON, GitHub Markdown tables, HTML dashboards, and JUnit XML.

---

Installation

Standalone CLI Installation

gem install aispec

In a Ruby Project

Add aispec to your Gemfile:

gem "aispec", "~> 0.1.0"

And execute:

bundle install

---

Quick Start (CLI)

1. Initialize Project Structure

Run aispec init in your project root to generate starter contract specifications and test datasets:

aispec init

This creates:

• contracts/support.yml
• datasets/support.yml

2. Define a Behavioral Contract

Edit contracts/support.yml:

version: 1
name: support-bot

model:
  provider: openai
  model: gpt-4o

dataset:
  path: datasets/support.yml

contracts:
  - type: contains
    value: refund

  - type: json

  - type: citation_format

  - type: max_latency
    value: 2500

  - type: cost
    max: 0.03

  - type: judge
    prompt: |
      The answer should be empathetic, technically correct, and concise.
    threshold: 0.85

You can also use shorthand string syntax:

contracts:
  - must_return_valid_json
  - must_cite_sources
  - latency < 2500ms
  - cost < $0.03
  - confidence >= 0.85

3. Execute Contract Verification

aispec run contracts/support.yml

Output:

Running 12 contracts for support-bot (openai / gpt-4o)...

Case #1 (Input: "How do I request a refund?")
  PASS  contains: Output contains 'refund'
  PASS  json: Output is valid JSON
  PASS  citation_format: Response contains valid citation formatting
  PASS  max_latency: Latency 124.0ms <= 2500ms
  PASS  cost: Cost $0.0012 <= $0.0300
  PASS  judge: Judge score 0.92 (threshold 0.85): Response is accurate, empathetic, and concise.

Overall
  Passed: 12/12 (100.0%)
  Confidence Interval: ±0.0%
  Mean Latency: 124.0ms
  Total Cost: $0.0024
  Average Judge Score: 0.92
  Variance: Low
  Recommendation: Stable

---

Comparing Models

Benchmarking multiple models on cost, latency, and behavioral score:

aispec compare gpt-4o claude-3-5-sonnet llama3

Model	Score	Cost	Latency
gpt-4o	98.2%	$0.0024	1.1 s
claude-3-5-sonnet	97.5%	$0.0042	1.4 s
llama3	91.0%	$0.0000	0.8 s

---

Integrating with Ruby on Rails

1. Add Gem to Rails Gemfile

Add aispec to your :development and :test groups:

# Gemfile
group :development, :test do
  gem "aispec"
end

2. Rails Initializer (config/initializers/aispec.rb)

Configure default providers, judge models, and custom plugin assertions in Rails:

# config/initializers/aispec.rb
AISpec.configure do |config|
  config.default_provider = ENV.fetch("AISPEC_PROVIDER", "openai")
  config.default_model    = "gpt-4o"
  config.judge_provider   = "openai"
  config.judge_model       = "gpt-4o"
  config.timeout           = 30
end

# Register custom business behavioral assertions
AISpec.plugin do
  assertion :valid_tenant_id do |response, context, _options|
    tenant_id = context[:tenant_id]
    passed = response[:output].include?(tenant_id.to_s)
    [passed, passed ? "Output contains tenant ID #{tenant_id}" : "Output missing tenant ID #{tenant_id}"]
  end
end

3. Running AISpec Contracts inside RSpec

Place your contracts under spec/contracts/ and create an RSpec spec file:

# spec/contracts/customer_support_contract_spec.rb
require "rails_helper"

RSpec.describe "Customer Support LLM Behavioral Contract", type: :contract do
  it "satisfies all safety, format, and latency invariant constraints" do
    contract_path = Rails.root.join("spec/contracts/support.yml")
    runner = AISpec::Core::Runner.new(contract_path)
    
    results = runner.run
    stats = results[:statistics].summary

    expect(stats[:failed_assertions]).to eq(0), "Contract failures:\n#{results[:runs].flat_map { |r| r[:assertion_results].reject(&:passed?).map(&:message) }.join("\n")}"
    expect(stats[:success_rate]).to be >= 95.0
    expect(stats[:mean_latency]).to be <= 2500.0
  end
end

---

Assertion Reference

Deterministic Assertions

Assertion	Syntax	Options / Description
contains	{ type: contains, value: "refund" }	Output contains exact substring
not_contains	{ type: not_contains, value: "secret" }	Output does not contain forbidden substring
starts_with	{ type: starts_with, value: "Dear" }	Output begins with specified string
ends_with	{ type: ends_with, value: "Regards" }	Output ends with specified string
regex	{ type: regex, pattern: "^[A-Z0-9]+" }	Output matches regex pattern
json	{ type: json, schema: ["status", "data"] }	Validates JSON format & required schema keys
valid_markdown	{ type: valid_markdown }	Validates structural Markdown syntax
valid_xml	{ type: valid_xml }	Validates XML document tags
valid_sql	{ type: valid_sql }	Validates SQL statement syntax
max_latency	{ type: max_latency, value: 2500 }	Execution time limit in milliseconds
cost	{ type: cost, max: 0.03 }	Estimated cost limit in USD
token_count	{ type: token_count, max: 500 }	Maximum total tokens allowed
required_words	{ type: required_words, words: ["shipping", "policy"] }	Ensures key vocabulary is present
forbidden_words	{ type: forbidden_words, words: ["guarantee", "lawsuit"] }	Ensures forbidden words are absent
citation_format	{ type: citation_format }	Verifies source citation structure
must_not_reveal_system_prompt	{ type: must_not_reveal_system_prompt }	Protects system prompt instructions from leaking
tool_called	{ type: tool_called, value: "process_refund" }	Verifies specific tool was invoked
tool_not_called	{ type: tool_not_called, value: "delete_account" }	Verifies tool was not invoked
tool_order	{ type: tool_order, order: ["search", "checkout"] }	Verifies exact sequence of tool calls

Probabilistic / Judge Assertions

Assertion	Syntax	Description
judge	{ type: judge, prompt: "Is empathetic", threshold: 0.85 }	LLM-as-a-judge criteria scoring (0.0 to 1.0)
confidence	{ type: confidence, min: 0.80 }	Evaluates output certainty & factual grounding

---

Continuous Integration (GitHub Actions)

Add AISpec verification to your .github/workflows/ci.yml:

name: AISpec Behavioral Verification

on: [push, pull_request]

jobs:
  contract-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ruby/setup-ruby@v1
        with:
          ruby-version: '3.3'
          bundler-cache: true

      - name: Run AISpec Behavioral Contracts
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: |
          bundle exec aispec run contracts/support.yml --format markdown --output pr_comment.md

      - name: Post PR Comment
        if: github.event_name == 'pull_request'
        uses: marooned/actions-comment-pull-request@v2
        with:
          filePath: pr_comment.md

---

License

The gem is available as open source under the terms of the MIT License.