Manual-Only AI Audits Pauzeren
Overview
Manual-only AI audits (slide decks and checklist walkthroughs without continuous controls, telemetry, or automated tests) fail to keep pace with weekly model and agent changes.
Hold as the sole assurance mechanism. Pair periodic audits with CI evals, guardrails, SBOM, and operational metrics.
Adoption Signals
- Measured drift in hosted models: a Stanford/UC Berkeley study found GPT-4's accuracy on identifying prime numbers fell from 84% to 51% between the March and June 2023 versions, and the authors call for continuous monitoring (arXiv 2307.09009).
- Short deprecation windows: OpenAI announced on 14 April 2025 that gpt-4.5-preview would shut down on 14 July 2025, and lists more deprecation notices across 2025 and 2026, including the Assistants API on 26 August 2026 (OpenAI deprecations).
- Regulation asks for continuous evidence: Article 12 of the EU AI Act requires high-risk systems to allow "automatic recording of events (logs) over the lifetime of the system" (Article 12), and Article 72 requires providers to "actively and systematically collect, document and analyse" performance data (Article 72).
- Frameworks are built around ongoing measurement: the NIST AI RMF organises work into Govern, Map, Measure, and Manage across the lifecycle, and NIST AI 600-1 (July 2024) adds generative-AI-specific actions (NIST AI RMF).
- The OWASP Top 10 for LLM Applications 2025 lists risks such as prompt injection, excessive agency, and unbounded consumption that depend on prompts, tools, and traffic, and can only be checked by testing the running system (OWASP GenAI).
Risks
- Findings are stale on arrival. Model snapshots, system prompts, retrieval indexes, and tool permissions change between audits, so a review certifies a configuration that is no longer in production.
- Silent regressions. Provider-side model changes and forced migrations (arXiv 2307.09009, OpenAI deprecations) are not caught without automated regression evals tied to model and prompt versions.
- Evidence gap for regulators. Without automatic logs and monitoring data, providers cannot meet the Article 12 and Article 72 duties that apply to high-risk systems from 2 December 2027 for Annex III use cases (Hunton).
- False assurance on security. A checklist can confirm that a prompt-injection policy exists, but not whether the deployed agent resists injected instructions or over-uses its tools.
- Review becomes a bottleneck. When every change waits for a committee, teams tend to batch changes or bypass the process, which reduces coverage further.
Pros & Cons
Advantages
- It fits processes that compliance, internal audit, and external auditors already run: committee reviews, checklists, sign-offs, and an annual cycle, with no new tooling budget.
- It produces the policies, approvals, and minutes that boards and certification audits ask for, and human reviewers are needed anyway to judge context-specific harms that automated tests do not capture.
- It can start before engineering teams have evaluation pipelines, tracing, or red-teaming tools, so it looks like the fastest way to show that AI is "under control".
Disadvantages
- Hosted model behaviour changes between versions: the same GPT-4 service dropped from 84% to 51% accuracy on one task between March and June 2023, so a sign-off describes a system that may no longer exist.
- Provider deprecations run faster than audit cycles; OpenAI announced the shutdown of gpt-4.5-preview in April 2025 for July 2025, a three-month window.
- The EU AI Act requires automatic logging over a high-risk system's lifetime and active, systematic post-market monitoring, evidence that point-in-time reviews cannot produce.
Recommendation
Hold manual-only audits as the sole assurance mechanism for AI systems. Human review stays necessary, but it should judge evidence that machines produce continuously, not replace that evidence. Do not start new AI governance programmes that rely on slide decks, questionnaires, and annual sign-offs alone.
If audits are currently manual, add the technical layer in this order: an inventory of AI systems with pinned model versions; an evaluation suite that runs in CI on every model, prompt, or tool change; scheduled red-teaming runs with tools such as PyRIT or garak; and production tracing with retained logs. Then change the audit itself: reviewers sample eval results, red-team findings, incidents, and change logs for the period, and a new model version or a new tool permission triggers a re-review automatically instead of waiting for the next cycle.
Sources
Advies
Van radar naar implementatie
U heeft technologieën beoordeeld — wij helpen u ze veilig en meetbaar in productiesystemen te brengen.
