Garak Assess

security evaluation governance red-team llm-security nvidia assess sec may-2026 garak

May 2026

Overview

Garak is an LLM vulnerability scanner probing for jailbreaks, data leakage, toxicity, and misinformation patterns (Garak).

Assess as a baseline probe suite before release. Combine with domain-specific evals; garak scores are not sufficient alone for shipping decisions.

Adoption Signals

Active project with NVIDIA ownership: about 9,400 GitHub stars and 1,300 forks, Apache 2.0 licence, and releases v0.15.0 (May), v0.16.0 (August) and v0.17.0 (September 2026) (Garak GitHub, garak releases).
NVIDIA commits to contributing to garak "in the long term" and keeping it Apache 2.0 (garak FAQ).
Embedded in a vendor product: NVIDIA NeMo Auditor, in early access, runs garak probes and detectors and outputs JSONL, HTML and hit-log reports (NVIDIA NeMo Auditor).
Compliance mapping: v0.17.0 added tags that group results by EU AI Act risk categories, and report digests can be grouped by OWASP LLM Top 10 (garak v0.17.0 release, garak FAQ).
Documented method: the arXiv preprint "garak: A Framework for Security Probing Large Language Models" (Derczynski et al., 2024) describes the architecture of probes, detectors and generators (arXiv).

Risks

Misreading scores. The FAQ states the scores have no scientific validity as a normalised scale and that a pass rate alone says little; the HTML report includes Z-scores against a calibration set of reference models for context (garak FAQ).
Detector noise. Most detectors look for keywords or use a classifier to judge replies, so hits need human review before they become findings or block a release (garak FAQ).
Coverage gaps. Garak probes a model endpoint; for RAG pipelines it scans only a few indirect prompt-injection cases (garak FAQ), and agent testing is limited to the agent breaker probe for tools exposed by the target, added in v0.15.0 (garak v0.15.0 release). Authorisation flaws, business-logic abuse and domain-specific harms need separate tests.
Runtime and cost. Running every probe can take hours; the maintainers recommend the fast config and parallel attempts, and hosted-model scans consume paid tokens (garak FAQ).
Static prompts age. Well-known jailbreak prompts such as DAN variants are widely trained against, so passing them overstates robustness to new attacks.

Pros & Cons

Advantages

Broad, free probe library under Apache 2.0: jailbreaks, encoding-based prompt injection, data leakage, package hallucination, XSS and markdown exfiltration, run against OpenAI, Bedrock, Hugging Face, NIM, Ollama or any REST endpoint.
Backed long term by NVIDIA, actively released (v0.17.0 in September 2026) and used as the engine of NVIDIA's NeMo Auditor, so probe coverage keeps growing.
Produces auditable evidence: JSONL reports with every prompt, response and score, plus report digests grouped by OWASP LLM Top 10 and, since v0.17.0, by EU AI Act risk categories.

Disadvantages

Scores are not a benchmark: the maintainers state results have no normalised scale, cannot be compared across probes, and drift as probes get harder, so pass rates older than six months should not be relied on.
Detectors mostly use keyword matching or classifiers, which can misjudge replies in both directions, so hits need manual triage.
Focuses on the model or endpoint rather than the whole application; beyond the agent breaker probe for tool abuse, authorisation flaws and business-logic abuse need other tests, and a full run can take hours.

Recommendation

Keep garak in Assess as a baseline scanner for LLM endpoints before release and after model or prompt changes. Start with a small, relevant probe set (prompt injection, encoding, data leakage, package hallucination for code assistants) using the fast config against a staging endpoint, store the JSONL reports as evidence, and triage hits manually before tracking trends.

Do not use garak pass rates as a release gate on their own or compare them across probes or garak versions. Combine it with domain-specific evals, PyRIT or manual red-teaming for multi-turn and agentic attacks, and application-level security tests. Teams that need scans as audit evidence for the EU AI Act or OWASP LLM Top 10 can use the built-in mappings, but should document which probes were run and why.