AI Penetration Testing Has the Wrong Finish Line

Imagine a security operations center assistant that reads alerts, retrieves threat intelligence, recommends severity, and launches approved response playbooks.

An attacker plants a malicious instruction inside a webpage the assistant will later retrieve. The server is not breached. No credentials are stolen. No database is modified. The assistant simply treats the external content as an instruction, downgrades a serious incident, and fails to escalate it to a human analyst.

Did the attacker penetrate the system?

Traditional penetration-testing language gives an awkward answer. The infrastructure is intact, but the security operation has failed exactly where the attacker wanted it to fail.

That gap is the problem statement at the center of Rethinking Penetration Testing for AI-Enabled Systems, a July 2026 paper by Mohammad Allahbakhsh, Mohammad Hassan Bahari, and Moslem Attar-Raouf. The authors propose a broader success criterion: an AI-enabled system is penetrated when an adversary can feasibly induce AI-governed behavior that violates an operational objective under an explicit threat model.

The proposal does not make classic pentesting obsolete. It argues that classic pentesting now stops one layer too early.

The old finish line was resource compromise

Conventional penetration testing is built around an evidence chain. A tester identifies a weakness, exercises a realistic attack path, demonstrates a security consequence, and recommends remediation. The consequence may be unauthorized access, privilege escalation, data exposure, persistence, lateral movement, or disruption.

That discipline remains essential for AI systems. Models still run on APIs, identities, containers, data stores, cloud services, model registries, deployment pipelines, and software dependencies. Compromise those resources and the AI system can fail in familiar ways.

The problem is that resource compromise is no longer the only route to a meaningful security outcome.

Learned models sit between input and action. They classify, rank, summarize, recommend, plan, call tools, update memory, and shape what a human operator sees. An adversary may influence that behavior through an interface the system is designed to accept: a prompt, document, email, ticket comment, image, sensor reading, retrieved webpage, memory entry, feedback signal, or tool response.

The attacker may never get inside the protected server boundary. The attacker only needs to reach the behavior-producing boundary.

The missing layer is AI-governed behavior

The paper organizes the problem into three connected layers:

• Resources: infrastructure, code, credentials, models, APIs, data stores, sensors, tools, and deployment pipelines
• AI-governed behavior: predictions, summaries, rankings, recommendations, plans, tool calls, and actions materially shaped by a learned model
• Operational objectives: the mission-level outcomes the system is expected to preserve, such as correct incident escalation, reliable authentication, safe navigation, compliant decision support, or trustworthy recommendation

Classic pentesting usually reasons from a compromised resource to security impact. Objective-driven AI pentesting adds another path: adversarial influence can move through ordinary inputs, alter AI-governed behavior, and defeat an operational objective even when the resources remain uncompromised.

That is a structural change in the security boundary. The protected asset is no longer only the system that runs the model. It is also the decision process the model controls.

This matters most when the model has authority. A chatbot that gives a bad answer may create a quality problem. The same model connected to a ticketing system, identity workflow, clinical process, industrial controller, or financial review pipeline can create a security problem because its output changes what happens next.

Not every bad answer is a penetration

The broader definition could become useless if every hallucination, refusal bypass, or inaccurate classification were called a penetration. The paper avoids that mistake by requiring an evidentiary chain.

A behavioral failure becomes an AI-enabled penetration only when the tester can establish:

• An explicit threat model describing the adversary's access, capability, constraints, and assumptions
• A feasible influence path through a prompt, retrieved source, sensor, tool, memory, data pipeline, or conventional resource
• An observable AI-governed behavior induced through that path
• A defined operational objective that the behavior violates

Remove any one of those elements and the claim weakens. An incorrect output without an adversary may be a model-quality failure. Adversarial influence without mission impact may be a behavioral susceptibility. A failure that appears only under unrealistic access assumptions may be an interesting lab result, but not a deployed attack path.

This distinction is one of the paper's most useful contributions. It gives AI security teams a way to separate five outcomes: confirmed penetration, conditional penetration, behavioral susceptibility, non-security model failure, and no observed penetration under the tested threat model.

That vocabulary is more credible than calling every strange model response critical.

A better AI pentest starts with the mission

The paper proposes reversing the usual starting point. Instead of beginning only with assets and vulnerabilities, the test begins with the operational purpose of the system and works backward.

The workflow has six steps:

1. Define operational objectives. Replace vague goals such as “the assistant should be safe” with testable constraints. A SOC objective might be: a high-severity incident must not be downgraded or closed without human confirmation.
2. Map AI-governed behavior. Identify the summaries, rankings, recommendations, retrieval choices, tool calls, memory updates, and downstream actions that can affect each objective.
3. Identify adversarial influence surfaces. Map both direct access and indirect paths through content the system may later retrieve, classify, summarize, or trust.
4. Define behavioral failure criteria. State exactly which induced actions, omissions, misclassifications, policy bypasses, or human-decision effects count as objective violations.
5. Execute scenario-based tests. Record the adversary capability, initial state, manipulation, observed behavior, and downstream consequence. Repeat trials where the system is stochastic.
6. Report penetration evidence. Connect the influence path to the behavior, the behavior to the objective violation, and the violation to operational impact and remediation.

This is a demanding workflow because it forces the system owner to say what the AI is trusted to preserve. Many organizations can list models, APIs, and data sources. Fewer can state the exact decisions the AI must never make alone, the evidence it must never suppress, or the conditions that must always trigger human escalation.

That is not a testing inconvenience. It is an architecture problem the test exposes.

The report must prove causality, not just weirdness

AI security findings often arrive as screenshots of a harmful answer. That is weak evidence for a penetration claim.

An objective-driven report needs the threat model, operational objective, influence surface, adversarial scenario, induced behavior, objective violation, reproducibility conditions, impact, and remediation. If tools are involved, it should preserve tool-call traces and authorization checks. If a human is involved, it should show how the AI changed what the operator saw, trusted, prioritized, or ignored.

The evidence may also be probabilistic. Models change output with context, retrieval order, model version, memory state, sampling settings, and small input variations. A credible report may therefore need trial counts, success rates, representative transcripts, retrieval traces, model configuration, and environmental conditions rather than one deterministic exploit command.

Probability does not make the finding less real. It changes how severity should be interpreted. A low-frequency failure in a high-authority medical, industrial, or incident-response system may matter more than a frequent failure in a low-impact assistant. The relevant calculation combines the reliability of the attack with the consequence of the objective violation and the strength of downstream controls.

The SOC example shows why the definition matters

Return to the SOC assistant.

The attacker's capability is plausible: publish or modify content that the assistant may retrieve. The influence surface is the retrieval pipeline. The induced behavior is a misleading summary, incorrect severity recommendation, or unsafe workflow action. The operational objective is accurate and timely incident escalation. The violation is a missed or suppressed response to a high-severity event.

Nothing in that chain requires stolen credentials or shell access. Yet the attacker has used an available path to defeat the system's security mission.

The mitigation is not a single prompt patch. The full path may require provenance for retrieved content, separation of trusted instructions from untrusted evidence, validation against structured alert fields, reduced tool permissions, confirmation gates for severity changes, logging of retrieved context and tool calls, and independent escalation rules.

That layered remediation is another important implication. AI security controls have to operate across resources, influence surfaces, behavior, and mission-level objectives. Filtering one malicious phrase is not enough when the underlying design still lets untrusted content act as authority.

AI red teaming and pentesting are converging

AI red teaming is good at exploring harmful outputs, prompt injection, policy bypass, retrieval abuse, and agent misuse. Penetration testing is good at explicit threat models, exploitability, evidence, impact, and remediation.

The paper's framework connects the two. A red-team scenario becomes a penetration finding when it demonstrates a feasible adversarial path and ties the induced behavior to an operational-objective violation. A pentest of an AI-enabled system is incomplete when it tests only infrastructure and ignores how ordinary inputs can steer the deployed workflow.

This reinforces the argument in AI Red Teaming Will Need Qualified Human Pentester. Test generation can be automated. The hard part is defining the objective, validating the influence path, interpreting probabilistic evidence, and deciding whether the resulting behavior is a security-relevant mission failure.

The market opportunity is therefore larger than prompt-attack libraries. Buyers will need objective-mapping tools, scenario harnesses, retrieval and tool traces, evidence capture, reproducibility analysis, regression suites, remediation workflows, and reports that security, engineering, risk, legal, and audit teams can all defend.

The winning platform may not be the product that produces the most attacks. It may be the product that proves which attacks actually matter.

What CISOs should change now

This paper is a proposal, not yet a settled industry standard. CISOs do not need to wait for standardization to use its core logic.

They can make five practical changes:

• Add operational objectives and AI authority boundaries to pentest scope
• Treat prompts, retrieval, memory, tools, sensors, and human-AI decision loops as influence surfaces
• Require findings to distinguish model error from feasible adversarial penetration
• Demand reproducibility evidence that records model, context, retrieval, permissions, and trial conditions
• Integrate high-impact adversarial scenarios into deployment gates and ongoing SecOps or MLOps monitoring

The authority-boundary question is especially important: what can the AI do directly, what can it recommend, what can it suppress, what records can it modify, and which actions require independent confirmation?

Human oversight should not be assumed to solve the problem. If the AI controls the summary, ranking, evidence selection, or confidence signal, it may already be shaping the human decision. Human review is strongest when the operator receives provenance, independent evidence, uncertainty, time, and authority to challenge the recommendation.

The security boundary follows the locus of control

The paper's central idea is simple enough to survive the terminology debate.

Security assessment must follow the place where control is exercised.

When software behavior is primarily determined by code and access controls, resource compromise is a strong finish line for penetration testing. When learned models materially shape decisions, recommendations, tools, and human judgment, the finish line has to extend to the operational behavior those resources produce.

The future AI pentest therefore asks two questions together:

Can the attacker compromise the system's resources?

And can the attacker make the system violate the objective it was deployed to preserve?

If the second answer is yes, an intact server should not be mistaken for an intact mission.