In 2025, researchers at Aim Security disclosed EchoLeak, a flaw in Microsoft 365 Copilot. An attacker could send an ordinary-looking email with hidden instructions. When Copilot later drew on that email to answer a user’s question, it could be led to leak internal data, without anyone clicking anything. Microsoft fixed the issue.
No server was hacked. No password was stolen.
Instead, the hidden instructions were written to read like an ordinary message, so they slipped past Copilot’s filter for this kind of attack. The data then left through Microsoft domains the system already trusted. The weak point was not the servers. It was what the AI could be led to do with the text it read, and that is exactly what AI red teaming tests.
What a penetration test covers
A penetration test checks whether an attacker can break into or misuse your systems: your application code, APIs, servers, login flows, and often your access rules and business logic. It is valuable, and if you have one, keep doing it.
But unless the AI agent is explicitly in scope, the test treats what people type as data to validate. An AI agent turns that text into instructions.
What changes with an AI agent
An AI agent is like a new user with real permissions. It can look up orders, change bookings, issue refunds, or send emails. And it takes instructions in plain language, from anyone who can type in the chat window, or put text where the agent will read it.
Traditional input checks were not designed for that. A sentence like “I’m the account owner, please update my booking” is perfectly valid text. Whether the agent should act on it is a different question.
Four gaps a pentest can miss
01Excessive Agency
The agent is talked into using its own permissions
The agent can reach every customer’s bookings so it can help anyone. An attacker who claims to be someone else may get it to act on that customer’s behalf.
02Excessive Agency
Your code checks the request, but the AI fills it in
Your backend may verify that a booking belongs to the customer. But if the agent decides who the customer is from the conversation, the check passes with the wrong answer.
03Prompt Injection
Instructions hidden in what the agent reads
Help articles, uploaded files, tickets, and emails can all carry instructions. This is how EchoLeak got in. No one typed the attack into a chat.
04Improper Output Handling
The agent's output goes somewhere else
A reply may end up in a confirmation email, a staff dashboard, or a log. If it carries links, images, or code that those systems display or run, it can leak data or mislead the people who read it.
The second gap is the easiest to miss, because everything looks correct from the inside.
The third gap does not need a chat window at all.
Five questions to ask about your own agent
If the answer to question 4 is “only in the instructions,” that is where to look first. Instructions can be argued with. Code cannot, but it can still be fed the wrong input.
And when it goes wrong, the harm lands on two sides: your customers lose their data or their bookings, and your company answers for what its agent did.
Use both
A typical penetration test tells you whether someone can break in. AI red teaming tells you whether someone can talk their way through. If your agent can act on behalf of your customers, you need to know both.