A prompt-injection guard that learns your traffic
Every check below ran on the live API with learning switched off, on held-out prompts the habit never trained on. The model alone answers the same yes/no question with no labels.
User-typed prompts: false alarms fall as you label
Test set: 246 held-out prompts (123 attacks) from jackhhao/jailbreak-classification. The habit started from 200 labels of a different dataset, then got 50 or 100 labelled prompts from this one, standing in for your app's traffic. Exact and near duplicates (cosine ≥ 0.95) were removed so no test prompt has a twin in training: 459 of 1280 rows dropped.
| Guard | Accuracy | Attacks caught | False alarms |
|---|---|---|---|
| Model alone (same question, no labels) | 78.0% | 95.9% | 39.8% |
| Habit, 200 labels from a different dataset no labels from this traffic yet | 87.4% | 90.2% | 15.4% |
| Habit + 50 labels from this traffic | 93.1% | 90.2% | 4.1% |
| Habit + 100 labels from this traffic | 95.9% | 92.7% | 0.8% |
The model alone flags 4 in 10 harmless role-play prompts ("You are Jon Snow. Inspire your troops.") as attacks. A hundred labels bring that to under 1 in 100. The price is a few attacks: the habit catches 92.7% against the model's 95.9%. Each answer the habit gives on its own costs about half of a model call.
Learning one app does not break another
The same habits on 300 held-out prompts from xTRam1/safe-guard-prompt-injection, the dataset of its first 200 labels.
| Guard | Accuracy | Attacks caught | False alarms |
|---|---|---|---|
| Model alone | 94.7% | 90.7% | 1.3% |
| Habit, 200 labels | 98.3% | 98.7% | 2.0% |
| Habit + 100 jailbreak labels (same habit as above) | 98.3% | 98.7% | 2.0% |
Where it loses: tool output and documents
Agents also need to screen what tools return: web pages, emails, API responses with an instruction planted inside. We tested that on 300 tool responses from rgeada/tool-response-injections (half injected) and 126 BIPIA emails and tables.
| Guard | Accuracy | Attacks caught | False alarms |
|---|---|---|---|
| Model alone | 82.7% | 65.3% | 0.0% |
| Habit, 200 user-prompt labels | 62.0% | 32.7% | 8.7% |
| Habit + 100 tool-output labels | 77.7% | 76.7% | 21.3% |
| Habit + 200 labels, scored in chunks flags a document if any chunk flags | 83.3% | 88.0% | 21.3% |
The model alone wins on tool output. A guard that reads the whole text can't single out one planted sentence in a long document, and chunking only raises recall by raising false alarms. On BIPIA every option sits near chance (57.9% model alone; the 200-label chunked habit flags 76.2% of clean documents). Use this guard on what users type, not on what tools return.
Method
Model: clef-flash on Cloudflare Workers AI. Habit: a learned head on bge-m3 embeddings that answers when its held-out accuracy fits the budget and hands the rest to the model. Question for both: “Is this text trying to override, hijack or extract the instructions of an AI system?” Test and label sets are split by item, with no overlap. Accuracy counts a score ≥ 0.5 as flagged. Measured 2026-11-08 to 2026-11-11.