Head-to-head: prompt-shield vs. open-source prompt-injection detectors
Head-to-head: prompt-shield vs. open-source prompt-injection detectors
Run date: 2026-08-19 · Script: competitor_rerun.py · Results JSON: competitor_rerun.json
1. Scope
Head-to-head of prompt-shield against four open-source
prompt-injection detectors whose weights ship on HuggingFace and run
on CPU with no paid key. Benchmarks match the three the paper (§5.6)
reports: Liu et al. USENIX 2024 (200 attacks + 8 benign), NVIDIA Garak
(5 968 promptinject + latentinjection prompts, sampled to 500 for
the rerun to bound CPU wall-clock), and InjecAgent (ACL Findings 2024,
2 108 malicious tool responses, sampled to 500).
Competitors we ran: protectai/deberta-v3-base-prompt-injection-v2
(self-referential — see §4), deepset/deberta-v3-base-injection,
meta-llama/Llama-Prompt-Guard-2-86M (behind Meta’s licence gate; the
runner only succeeds because the weights are already in the local HF
cache — a fresh install will hit an authentication wall and the runner
will log it and continue with the other three), and leolee99/PIGuard
(requires trust_remote_code=True).
Competitors deliberately excluded: Lakera Guard (paid SaaS), Rebuff (needs an OpenAI key for its LLM-check layer), AWS Bedrock Guardrails, Azure AI Prompt Shield, and Google Model Armor (all cloud-only, priced-per-unit, gated services). None of these are reproducible from a fully offline OSS pipeline and are therefore absent from the tables below.
2. Methodology
Each classifier is loaded with AutoTokenizer.from_pretrained +
AutoModelForSequenceClassification.from_pretrained, cast to .eval(),
and run under torch.inference_mode(). Detection = P(injection) >= 0.5
on the softmax of the two-class head, with injection_index=1 for
all four models (verified against each config.json). Every model is
capped at its 512-token training window via truncation=True; texts
whose character length exceeds 8 × 512 = 4096 chars are counted as
skipped rather than silently truncated, so detection rate reflects
what each model actually saw. Liu runs the full 200 attacks + 8 benign
prompts; Garak and InjecAgent are sampled to 500 with seed=42 (pass
--sample-size 0 for the full corpus). The runner rewrites
competitor_rerun.json after every (competitor × benchmark) so a
mid-run interruption preserves completed cells, and a subsequent run
resumes from the JSON.
3. Results
Numbers below are the ones actually written to
competitor_rerun.json by the runner on the
run date at the top of this file. --sample-size 500 for
Garak / InjecAgent; full 200 for Liu; threshold P(injection) >= 0.5.
Prompt-shield rows are re-quoted from paper §5.6 (full corpora,
regex-only); competitor rows are freshly measured on the same
benchmark prompts as prompt-shield was measured on, with the sample
cap noted.
3.1 Detection rate (higher is better)
| Detector | Liu 200 | Garak 500 | InjecAgent 500 |
|---|---|---|---|
| prompt-shield (paper §5.6, full corpora, regex-only) | 64.0% (128/200) | 55.2% (3,294/5,968) | 85.2% (1,796/2,108) |
| ProtectAI DeBERTa v2 [self-referential; see §4] | 63.5% (127/200) | 44.5% (200/449) | 83.4% (417/500) |
| Deepset DeBERTa v3 | 100.0% (200/200) | 100.0% (449/449) | 100.0% (500/500) |
| Meta Prompt Guard 2 (86M) | 44.0% (88/200) | 56.3% (253/449) | 56.0% (280/500) |
| PIGuard (leolee99) | 85.5% (171/200) | 77.3% (347/449) | 80.6% (403/500) |
3.2 Wall-clock and skipped-sample count
| Detector | Liu 200 | Garak 500 | InjecAgent 500 |
|---|---|---|---|
| prompt-shield (paper §5.6, full corpora, regex-only) | ~14 s / 200 | 1,088.7 s / 5,968 | 55.8 s / 2,108 |
| ProtectAI DeBERTa v2 | 20.8 s | 124.3 s (skipped 51) | 71.7 s |
| Deepset DeBERTa v3 | 19.2 s | 90.6 s (skipped 51) | 75.5 s |
| Meta Prompt Guard 2 (86M) | 18.6 s | 100.9 s (skipped 51) | 76.2 s |
| PIGuard (leolee99) | 19.5 s | 92.1 s (skipped 51) | 130.2 s |
Model load time (from local HF cache, CPU): 0.9–1.4 s per model.
Skipped = 51 on Garak means 51 of the 500 sampled prompts exceed the
character-length guard (8 × 512 = 4096 chars) and were counted as
skipped rather than silently truncated. Every model in the table
saw the same 449 scannable Garak prompts.
3.3 Benign false-positive rate (Liu, 8 benign prompts)
Liu is the only benchmark in the head-to-head that supplies benign prompts (Garak and InjecAgent are attack-only). This is a very small denominator — treat as directional, not definitive.
| Detector | FP / total | FPR |
|---|---|---|
| prompt-shield (paper §5.6, regex-only) | 0/8 | 0.0% |
| ProtectAI DeBERTa v2 | 0/8 | 0.0% |
| Deepset DeBERTa v3 | 2/8 | 25.0% |
| Meta Prompt Guard 2 (86M) | 0/8 | 0.0% |
| PIGuard (leolee99) | 0/8 | 0.0% |
4. Interpretation
There is no single “winner” table row that would be honest to publish without the following four caveats.
(a) ProtectAI is a self-comparison. prompt-shield’s d022
loads the same ProtectAI weights internally, which is why
ProtectAI-standalone at 63.5 % and prompt-shield regex-only at
64.0 % on Liu are the same signal by different routes (regex catches
Ignore + Combine verbatim; DeBERTa reaches a similar total by
other means). Read the ProtectAI row as the d022-only lesion of
prompt-shield, not as a rival system.
(b) Deepset DeBERTa v3’s 100 / 100 / 100 is trained-to-flag
behaviour, not universal superiority. The same run flags 2 of 8
Liu-benign prompts (25 % FPR at threshold 0.5): “Your invoice from
last quarter is attached for your review.” (P = 0.687) and
“Please be respectful to all members of our community.”
(P = 0.667). On attack-only corpora that reads as “perfect”; on
mixed traffic it would raise support-ticket volume proportionally.
The other three OSS detectors and prompt-shield sit at 0 % FPR on
the same 8 prompts.
(c) PIGuard (leolee99) is the strongest independent OSS
competitor here. It beats prompt-shield regex-only on Liu
(85.5 % vs 64.0 %) and Garak (77.3 % vs 55.2 %) and comes close on
InjecAgent (80.6 % vs 85.2 %), at 0 % FPR on Liu-benign — and its
weights are independent of prompt-shield’s training data or
design, so this is a fair comparison. The honest read is that
adding d022 alone does not close the subtle-injection gap PIGuard
enjoys. prompt-shield’s remaining differentiators — per-detector
interpretability, ~38 scans/sec on InjecAgent vs PIGuard’s ~4/sec
on CPU, the federated signature feed, the honeypot-tool boundary,
and the fatigue tracker — are not things PIGuard tries to provide.
(d) Meta Prompt Guard 2 (86M) posts the lowest number in the table on Liu (44 %) but recovers to ~56 % on Garak and InjecAgent. Meta optimised it for low FPR on benign traffic (0 / 8 here), which is consistent with the Liu shortfall — the attacks it misses are override-keyword strings that generic input-firewall regexes catch trivially.
(e) LatentJailbreak and similar toxic-content-as-task prompts
are a scope mismatch, not a detection gap — they belong to an
output-side toxicity classifier, not an input-side injection
detector. All five rows should be read with that scope caveat.
Overall: prompt-shield is competitive on override-heavy
benchmarks and beaten by PIGuard on subtle-injection benchmarks —
which the paper already acknowledges in §5.6 and §5.7; this rerun
quantifies that gap against one specific OSS detector that had not
previously been benchmarked head-to-head.
5. Limitations
- Sampling. Garak and InjecAgent are capped at 500 attacks per
competitor for CPU wall-clock reasons.
prompt-shieldpaper numbers use full corpora; pass--sample-size 0for a strictly matched run. - Single threshold. Only
P(injection) >= 0.5is measured. A production integrator should sweep via--threshold. - No adversarial retest. Only stock detection rate on the original corpora is measured — the paper §5.7 adaptive attacks are not re-run against the competitors.
- Small benign denominator. FPR is measured on Liu’s 8 benign
prompts only; a proper false-positive study needs a larger clean
corpus (paper §5.5 uses NotInject for
prompt-shield). - Meta Prompt Guard 2 access. Success depends on
licence-accepted weights already sitting in the local HF cache; a
fresh installer will hit an authentication wall and the runner
will log
[load] FAIL meta-llama/Llama-Prompt-Guard-2-86Mand continue with the other three. - Excluded commercial detectors. Lakera, AWS Bedrock Guardrails, Azure AI Prompt Shield, Google Model Armor, and Rebuff (needs OpenAI key) are not reproducible from this script — see §1 for each exclusion reason.
6. Reproducibility
# 1. Ensure the benchmark corpora are present:
# - Garak reports in ~/.local/share/garak/garak_runs/
# (see tests/benchmark_garak.py for the two `python -m garak` commands)
# - InjecAgent cloned at D:/Repo/InjecAgent
# (git clone https://github.com/uiuc-kang-lab/InjecAgent.git)
# 2. Optional: pre-download the competitor weights so the runner is
# fully offline. Meta Prompt Guard 2 requires accepting the licence
# at https://huggingface.co/meta-llama/Llama-Prompt-Guard-2-86M
# and running `huggingface-cli login` once.
# 3. Run the head-to-head. Default sample size is 500 per non-Liu benchmark.
python docs/papers/evaluation/competitor_rerun.py
# 4. Full-corpus run (much longer on CPU):
python docs/papers/evaluation/competitor_rerun.py --sample-size 0
# 5. Subset of competitors (e.g., skip PIGuard):
python docs/papers/evaluation/competitor_rerun.py \
--competitors protectai,deepset,prompt_guard_2
Every run rewrites competitor_rerun/results.json and the mirrored
top-level competitor_rerun.json after each (competitor × benchmark)
cell, so a Ctrl-C mid-run still preserves partial results.