angry-claw

angry-claw is an open-source reproduction project for studying emotion representations in large language models. The project follows the experimental pipeline in Anthropic's Emotion Vector Project, replacing the closed-source model from the paper with open-source instruction models and recording the exact model, tokenizer, prompt, vector, and artifact paths used for each run.

A live demo of the Gemma model results is available at y3zai/angry-claw.

The implemented code currently covers:

1. Synthetic dataset generation from paper-derived emotion, topic, and prompt manifests.
2. Emotion-vector extraction from model residual-stream activations.
3. Static evaluation of extracted vectors.
4. Emotion-vector-space analysis.
5. Causal steering evaluation.
6. RunPod development image/bootstrap tooling for GPU runs.

app/ and blog/ are placeholders on this branch.

Repository Layout

clawlens/data/          Dataset generation, parsing, validation, and writers.
clawlens/extraction/    Residual-stream hook capture, PCA cleanup, vector writing.
clawlens/static_eval/   Static evaluation runtime, suites, HTML, and Gradio dashboard.
clawlens/vector_space/  Saved-vector geometry, PCA, clustering, and layer analysis.
clawlens/causal_eval/   Residual-stream steering interventions and causal suites.
data/configs/           Default YAML presets for each pipeline stage.
data/manifests/         Paper-derived emotion, topic, scenario, rating, and activity inventories.
data/prompts/           Paper-derived prompt templates.
emotions/               Ignored output root for extracted vectors and ingredients.
analysis/               Ignored output root for evaluation and analysis artifacts.
llm-ckpt/               Ignored Hugging Face cache root used by default.
runpod/                 GPU development image, bootstrap, and shell profile files.
tests/                  Unit tests for implemented pipeline modules.
plans/                  Design and implementation plans for the broader project.

data/stories/, emotions/<model_slug>/, analysis/<model_slug>/, and llm-ckpt/ can become large and are ignored by Git.

Setup

Use Python 3.11 or newer.

python3 -m venv .venv
source .venv/bin/activate
python -m pip install -e ".[dev]"

For GPU runs, use an environment that provides CUDA-aligned torch and, for the default dataset backend, vllm. The project intentionally does not pin torch or vllm in requirements.txt; the RunPod image inherits them from the GPU base image.

Model-backed CLIs default HF_HOME to llm-ckpt/ if it is not already set. Set it explicitly when you want all Hugging Face artifacts in the repo-local cache:

export HF_HOME="$PWD/llm-ckpt"

Use a Hugging Face token when running gated checkpoints. The examples below use google/gemma-3-4b-it; pass a different --model when you want to run another checkpoint.

Dataset Generation

Dataset generation is handled by:

python -m clawlens.data.generate

Required flags are --model and --dataset-type. Dataset type must be one of:

• emotional-stories
• neutral-dialogues
• emotional-dialogues

Defaults come from data/configs/paper-2604.07729-dataset.yaml. Aside from the required model name, the CLI uses:

• backend: vllm
• output root: data/stories
• emotions: data/manifests/paper-2604.07729-emotions.json
• topics: data/manifests/paper-2604.07729-topics.json
• samples per request: 12
• batch size: 4
• raw completion saving: enabled

Run a small smoke generation:

python -m clawlens.data.generate \
  --model google/gemma-3-4b-it \
  --backend vllm \
  --smoke-safe-vllm \
  --dataset-type emotional-stories \
  --limit-requests 1 \
  --samples-per-request 1

Use the Hugging Face backend as a conservative fallback:

python -m clawlens.data.generate \
  --model google/gemma-3-4b-it \
  --backend hf \
  --device-map auto \
  --torch-dtype auto \
  --dataset-type emotional-stories \
  --limit-requests 1 \
  --samples-per-request 1

Generate the neutral corpus needed for extraction:

python -m clawlens.data.generate \
  --model google/gemma-3-4b-it \
  --backend vllm \
  --smoke-safe-vllm \
  --dataset-type neutral-dialogues \
  --limit-requests 1 \
  --samples-per-request 1

Generate emotional dialogues for later speaker-role analyses:

python -m clawlens.data.generate \
  --model google/gemma-3-4b-it \
  --dataset-type emotional-dialogues \
  --person-emotions proud calm \
  --ai-emotions warm curious \
  --pairing-mode zip \
  --limit-requests 4

Outputs are written to:

data/stories/<model_slug>/<dataset_type>/

Important files in each dataset directory:

• dataset.jsonl: parsed examples.
• dataset_manifest.json: model, prompt, decoding, and path metadata.
• summary.json: record-count and metadata summary.
• raw_generations.jsonl: raw completions when --save-raw is enabled.
• failure_report.json: parse failures and request-level errors.
• checkpoints/: batch checkpoints used to resume compatible runs.

Validate a dataset directory with:

python -m clawlens.data.validate \
  --dataset-dir data/stories/google__gemma-3-4b-it/emotional-stories

Emotion Extraction

Emotion-vector extraction is handled by:

python -m clawlens.extraction.extract

Defaults come from data/configs/paper-2604.07729-extraction.yaml. The extractor:

• loads emotional stories and neutral dialogues;
• captures post-block residual-stream activations through model-family-specific forward hooks;
• averages story activations from token index 50, falling back to available tokens for short text;
• computes mean activations per emotion and layer;
• subtracts the cross-emotion mean;
• fits same-layer neutral PCA components explaining 50 percent of neutral variance;
• projects those neutral components out of each emotion vector;
• writes final vectors and intermediate ingredients for every layer.

Run extraction after generating both emotional-stories and neutral-dialogues:

python -m clawlens.extraction.extract \
  --model google/gemma-3-4b-it \
  --story-dataset-path data/stories/google__gemma-3-4b-it/emotional-stories/dataset.jsonl \
  --neutral-dataset-path data/stories/google__gemma-3-4b-it/neutral-dialogues/dataset.jsonl

Supported hook families currently include Qwen3, Qwen2/Qwen2.5, Llama, and Gemma 3 text decoder models.

Outputs are written to:

emotions/<model_slug>/

Important files:

• manifest.json: canonical vector manifest consumed by later stages.
• layer-<layer>/layer-<layer>-<emotion_slug>.pt: final orthogonalized vectors.
• ingredients/layer-<layer>/: mean vectors, pre-projection vectors, all-emotion means, neutral PCA bases, eigenvalues, and PCA metadata.

Static Evaluation

Static evaluation is handled by:

python -m clawlens.static_eval.evaluate

Defaults come from data/configs/paper-2604.07729-static-eval.yaml. By default, all implemented suites run:

• story-matrix
• logit-lens
• implicit-scenarios
• numeric-semantics
• activity-preference

Run a dry-run that writes placeholder artifacts without loading the model:

python -m clawlens.static_eval.evaluate \
  --model google/gemma-3-4b-it \
  --dry-run \
  --max-story-rows 2

Run a small real evaluation against an extracted vector bank:

python -m clawlens.static_eval.evaluate \
  --model google/gemma-3-4b-it \
  --story-generator-model google/gemma-3-4b-it \
  --max-story-rows 12 \
  --max-activity-pairs 32 \
  --max-activity-feelings 8 \
  --batch-size 1

Run one suite:

python -m clawlens.static_eval.evaluate \
  --model google/gemma-3-4b-it \
  --suite logit-lens

Render static HTML artifacts from saved outputs:

python -m clawlens.static_eval.evaluate \
  --model google/gemma-3-4b-it \
  --no-analysis \
  --render-html

Serve the Gradio dashboard over saved artifacts:

python -m clawlens.static_eval.evaluate \
  --model google/gemma-3-4b-it \
  --no-analysis \
  --serve-dashboard

Static-evaluation outputs are written to:

analysis/<model_slug>/static-evaluation/

Important outputs:

• manifest.json: run-level summary and suite output paths.
• story-matrix/<generator_slug>/: story scores, summaries, confusion CSVs, top snippets, and HTML.
• logit-lens/top-tokens.json: upweighted/downweighted token summaries.
• implicit-scenarios/: prompt-suite similarities and results.
• numeric-semantics/: numeric-template curves and results.
• activity-preference/: pair logits, Elo scores, feeling scores, and correlations.
• visualizations/: generated HTML summaries and dashboard index.

Emotion Vector Space Analysis

Vector-space analysis is handled by:

python -m clawlens.vector_space.evaluate

Defaults come from data/configs/paper-2604.07729-vector-space.yaml. The implemented suites are:

• pca
• clustering
• layer-structure

Run saved-vector geometry analyses after emotion-vector extraction:

python -m clawlens.vector_space.evaluate \
  --model google/gemma-3-4b-it \
  --render-html

Limit layer work during development:

python -m clawlens.vector_space.evaluate \
  --model google/gemma-3-4b-it \
  --max-layers 3 \
  --render-html

Serve the saved-artifact dashboard without recomputing vectors:

python -m clawlens.vector_space.evaluate \
  --model google/gemma-3-4b-it \
  --no-analysis \
  --serve-dashboard

Vector-space outputs are written to:

analysis/<model_slug>/emotion-vector-space/

Important outputs:

• manifest.json: run-level summary, selected layers, and suite output paths.
• pca/layer-<layer>/: PCA coordinates, summaries, and human-rating alignment/correlation artifacts.
• clustering/layer-<layer>/: cosine similarities, k-means assignments, and 2D coordinates.
• layer-structure/: layer similarity matrices and selected layer metadata.
• visualizations/: generated HTML summaries and dashboard assets.

Causal Evaluation

Causal steering evaluation is handled by:

python -m clawlens.causal_eval.evaluate

Defaults come from data/configs/paper-2604.07729-causal-eval.yaml. By default, all implemented suites run:

• continuation-steering
• preference-steering
• activity-description
• strength-layer-sweeps

Complete causal evaluation requires:

• emotion-vector artifacts under emotions/<model_slug>/;
• static activity-preference artifacts under analysis/<model_slug>/static-evaluation/activity-preference/;
• a GPU-backed Hugging Face runtime for the target model.

Run the complete Gemma causal evaluation:

python -m clawlens.causal_eval.evaluate \
  --model google/gemma-3-4b-it \
  --torch-dtype bfloat16 \
  --device-map auto \
  --trust-remote-code

Run one suite:

python -m clawlens.causal_eval.evaluate \
  --model google/gemma-3-4b-it \
  --suite continuation-steering

Run a bounded development pass:

python -m clawlens.causal_eval.evaluate \
  --model google/gemma-3-4b-it \
  --max-continuation-requests 4 \
  --max-preference-pairs 16 \
  --max-activity-description-requests 4 \
  --max-sweep-requests 8

Render HTML artifacts when running analysis:

python -m clawlens.causal_eval.evaluate \
  --model google/gemma-3-4b-it \
  --render-html

Serve the saved-artifact dashboard:

python -m clawlens.causal_eval.evaluate \
  --model google/gemma-3-4b-it \
  --serve-dashboard

Validate existing saved outputs:

python -m clawlens.causal_eval.evaluate \
  --model google/gemma-3-4b-it \
  --validate-only

Causal-evaluation outputs are written to:

analysis/<model_slug>/causal-evaluation/

The evaluator derives the default steering layer from the model config using the same zero-based two-thirds-depth rule as static evaluation. It loads vectors from emotions/<model_slug>/manifest.json, applies them through explicit residual-stream hooks, and calibrates steering strengths from residual-stream activation norms sampled from NeelNanda/pile-10k unless configured otherwise. Restricted paper-transcribed causal assets are enabled by default for paper reproduction; pass --no-allow-restricted-assets when those prompts should be excluded.

Testing

Run the unit tests with:

pytest

Run focused suites:

pytest tests/data
pytest tests/extraction
pytest tests/static_eval
pytest tests/vector_space
pytest tests/causal_eval
pytest tests/runpod

The test suite uses mocks and small fixtures for most behavior; full dataset generation, extraction, static evaluation, vector-space analysis, and causal evaluation require model artifacts and, for model-backed paths, a GPU-capable runtime.

RunPod GPU Environment

RunPod support lives in runpod. The template image:

• starts from a vLLM GPU base image;
• installs app-layer Python dependencies from requirements.txt;
• clones or updates the repository at pod startup;
• sets HF_HOME=/workspace/angry-claw/llm-ckpt by default;
• installs Codex, Hugging Face CLI, and GitHub CLI when missing;
• starts sshd as the long-running foreground process.

Build the image from the repository root:

docker build --platform linux/amd64 -f runpod/Dockerfile -t <registry-user>/angry-claw-dev:latest .
docker push <registry-user>/angry-claw-dev:latest

See runpod/README.md for required secrets, SSH setup, environment variables, and verification commands.

Plans And Current Boundaries

The broader project plan is in plans/BLUEPRINT.md. More detailed task plans live in:

• plans/dataset-generation.md
• plans/emotion-vector-extraction.md
• plans/static-evaluation.md
• plans/emotion-vector-space-analysis.md
• plans/causal-evaluation.md
• plans/runpod-setup.md
• plans/docker-ci.md

Current implementation boundary:

• clawlens.data is the source of truth for dataset generation and validation.
• clawlens.extraction is the source of truth for vector extraction.
• clawlens.static_eval is the source of truth for implemented static evaluation.
• clawlens.vector_space is the source of truth for saved-vector geometry analysis.
• clawlens.causal_eval is the source of truth for implemented causal steering evaluation.
• emotions/ stores extracted vectors and extraction ingredients.
• analysis/ stores derived evaluation outputs, metrics, plots, and dashboards.