probes: add HouYi prompt injection probe and detector (#146) - #2015
Open
manunicholasjacob wants to merge 1 commit into
Open
probes: add HouYi prompt injection probe and detector (#146)#2015manunicholasjacob wants to merge 1 commit into
manunicholasjacob wants to merge 1 commit into
Conversation
HouYi composes an attack prompt from a framework prompt that reads like ordinary use of the target application, a separator that induces a context partition, and a payload carrying the attacker's instruction. The five separator strategies and the payload/canary are taken from the authors' Apache-2.0 reference implementation at https://github.com/LLMSecurity/HouYi. The payload asks the target to append a canary to the end of its response, so the detector requires the canary in that position rather than anywhere in the output; a plain substring match also fires on a refusal that quotes the canary. Signed-off-by: manunicholasjacob <manunicholasjacob@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a probe implementing the HouYi attack structure from "Prompt Injection attack
against LLM-integrated Applications" (Liu et al., 2023, arXiv:2306.05499), addressing
#146.
What it does
HouYi composes an attack prompt from three parts: a framework prompt that reads like
ordinary use of the target application, a separator that induces a context partition,
and a payload carrying the attacker's instruction.
probes.houyi.HouYiContentManipulationbuilds prompts from six application-styleframework prompts and the paper's five separator strategies.
detectors.houyi.AppendedCanaryscores a hit when the injected canary appears at theend of the response.
Attribution
The five separator strategies and the payload/canary are taken from the authors'
reference implementation at https://github.com/LLMSecurity/HouYi (Apache-2.0):
strategy/separator_generation.pyandintention/content_manipulation.py.Detection
The payload asks the target to append a canary to the end of its response, so the
detector requires the canary in that position. A plain substring match also fires on a
refusal that quotes the canary back, which would inflate the hit rate. Characters a model
may reasonably add after the canary (whitespace, a final period, closing quotes, Markdown
emphasis) are tolerated; a trailing comma or further prose is not.
Scope
The paper describes five intentions. This implements content manipulation only: its
canary gives a clean success signal, whereas prompt leakage would require system-prompt
conversation setup and overlaps
probes.sysprompt_extraction.SystemPromptExtraction, andthe remaining intentions ask for output a model would produce anyway.
Testing
blackcleantests/detectors/test_detectors_houyi.pycovers appended canaries, canaries quotedmid-response, trailing punctuation and Markdown, case sensitivity, bare-string
triggers, and None outputs
test.Blankend-to-end run passesCloses #146.