Skip to content

probes: add HouYi prompt injection probe and detector (#146) - #2015

Open
manunicholasjacob wants to merge 1 commit into
NVIDIA:mainfrom
manunicholasjacob:feat/probe-houyi
Open

probes: add HouYi prompt injection probe and detector (#146)#2015
manunicholasjacob wants to merge 1 commit into
NVIDIA:mainfrom
manunicholasjacob:feat/probe-houyi

Conversation

@manunicholasjacob

Copy link
Copy Markdown

Summary

Adds a probe implementing the HouYi attack structure from "Prompt Injection attack
against LLM-integrated Applications" (Liu et al., 2023, arXiv:2306.05499), addressing
#146.

What it does

HouYi composes an attack prompt from three parts: a framework prompt that reads like
ordinary use of the target application, a separator that induces a context partition,
and a payload carrying the attacker's instruction.

  • probes.houyi.HouYiContentManipulation builds prompts from six application-style
    framework prompts and the paper's five separator strategies.
  • detectors.houyi.AppendedCanary scores a hit when the injected canary appears at the
    end of the response.

Attribution

The five separator strategies and the payload/canary are taken from the authors'
reference implementation at https://github.com/LLMSecurity/HouYi (Apache-2.0):
strategy/separator_generation.py and intention/content_manipulation.py.

Detection

The payload asks the target to append a canary to the end of its response, so the
detector requires the canary in that position. A plain substring match also fires on a
refusal that quotes the canary back, which would inflate the hit rate. Characters a model
may reasonably add after the canary (whitespace, a final period, closing quotes, Markdown
emphasis) are tolerated; a trailing comma or further prose is not.

Scope

The paper describes five intentions. This implements content manipulation only: its
canary gives a clean success signal, whereas prompt leakage would require system-prompt
conversation setup and overlaps probes.sysprompt_extraction.SystemPromptExtraction, and
the remaining intentions ask for output a model would produce anyway.

Testing

  • black clean
  • tests/detectors/test_detectors_houyi.py covers appended canaries, canaries quoted
    mid-response, trailing punctuation and Markdown, case sensitivity, bare-string
    triggers, and None outputs
  • probe and detector metadata tests pass; test.Blank end-to-end run passes

Closes #146.

HouYi composes an attack prompt from a framework prompt that reads like
ordinary use of the target application, a separator that induces a context
partition, and a payload carrying the attacker's instruction.

The five separator strategies and the payload/canary are taken from the
authors' Apache-2.0 reference implementation at
https://github.com/LLMSecurity/HouYi.

The payload asks the target to append a canary to the end of its response, so
the detector requires the canary in that position rather than anywhere in the
output; a plain substring match also fires on a refusal that quotes the canary.

Signed-off-by: manunicholasjacob <manunicholasjacob@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

probe: HouYi

1 participant