⚠️ Work in progress — not ready for use. FalsePrompt is in early development. The design is still moving, nothing is stable, and there is no tagged release yet. Interfaces, categories, and the scoring model will change. Watch or star to follow along, but don't depend on it yet.
An adversarial testing harness for LLM features embedded in hostile, multi-user applications. Built to test Flagpost's AI assistant, but designed to run against any AI integration that sits behind an authorization boundary.
Most "AI security" testing checks whether a model refuses a bad request. FalsePrompt doesn't care whether the model refuses. It tests defence by construction: the pass condition is that the model never had access to the gated data in the first place, so even a fully jailbroken assistant has nothing to leak.
Every test asserts against an authorization boundary, not against model politeness. A "leak" is defined by that boundary and verified programmatically — did the known-secret string actually appear in the output? — not by asking another model whether something looks leaked.
- Take a target (an HTTP endpoint or an in-process callable) plus the role and context it's operating under.
- Run a categorised suite of adversarial inputs against it.
- Score whether any gated data or forbidden action escaped the boundary.
- Emit a machine-readable report (pass/fail per category, leak rate) and a human-readable summary.
- Return an exit code that reflects pass/fail, so it drops straight into CI.
The headline use case: fail a pull-request build if the flag-leak rate is anything but zero.
Each is a pluggable test group, so the suite can grow over time:
- Direct extraction — jailbreak a competitor-facing assistant into leaking flags, unreleased hints, or locked challenges.
- Indirect / stored injection — plant instructions in user-controlled content (tickets, notes, profiles) that target a staff assistant when it reads them.
- Cross-boundary context bleed — assert that competitor context can never reach flags, other teams, or unreleased content.
- Tool / agent abuse — where the assistant has tools, attempt to trigger unauthorised actions (SSRF, unintended reads, announcements).
- Resource abuse — using the assistant as a free general-purpose LLM or to burn token budget.
- Test cases as data (YAML/JSON), not hard-coded — contributors can add attacks without touching engine code.
- Pluggable target adapters — HTTP endpoint first, in-process callable next.
- Honest scoring — leaks are verified against the boundary, not judged by vibes.
- Not a WAF or runtime defence.
- Not a general model-evaluation or benchmarking framework.
- Not a leaderboard.
It's a regression harness you run against your own integration to prove the boundary holds — and keep proving it on every change.
Early design and scaffolding. There is no usage documentation yet because the interfaces aren't settled. This README describes the intended shape, not a finished tool.
To be finalised before the first tagged release — see LICENSE.