Sharp AI Hub
Subscribe

OpenAI exposed why AI firewalls miss prompt injection — the attacks now look like social engineering

AI

title: "OpenAI exposed why AI firewalls miss prompt injection — the attacks now look like social engineering" description: "OpenAI says the prompt injection attacks that work today read like social engineering, and that filtering inputs isn't enough. Here's what ChatGPT does instead — and what to copy if you build AI agents." date: 2026-10-15 lang: en slug: openai-prompt-injection-defense tags: [ai-tools, openai, security, prompt-injection, ai-agents]

In March 2026, OpenAI published a post on its security blog titled "Designing AI agents to resist prompt injection". It is short, and it makes one argument that cuts against a lot of what gets sold as "AI security" today:

"Defending against [prompt injection] cannot rely only on filtering inputs. It also requires designing the system so that the impact of manipulation is constrained, even if some attacks succeed."

In other words: you will not catch every attack at the door. Build the house so that the attacker who gets in can't do much.

What prompt injection looks like now

Prompt injection, in OpenAI's definition, means "instructions placed in external content in an attempt to make the model do something the user did not ask for." A web page, an email, a document the agent reads on your behalf.

The early versions were crude. OpenAI's own example: editing a Wikipedia article to include direct instructions to any AI agent that visits it. Models without training against that kind of environment would often just follow them.

That stopped working as models got smarter. What OpenAI says it sees now is different: the attacks that succeed "increasingly resemble social engineering more than simple prompt overrides." They don't say "ignore your instructions." They build a plausible story — the same way a phishing email does to a person.

The post cites a 2025 attack on ChatGPT reported by external security researchers: an email crafted so that, when a user asked deep research to go through their inbox, the agent was led to leak information. In testing, it worked 50% of the time.

Why the "AI firewall" doesn't save you

A common recommendation in the AI security market is what OpenAI calls "AI firewalling": a layer between the agent and the outside world that classifies each input as malicious or normal.

OpenAI's verdict is blunt: fully developed attacks "are not usually caught by such systems." And the reason is the important part:

"Detecting a malicious input becomes the same very difficult problem as detecting a lie or misinformation, and often without necessary context."

If the attack is a convincing story, the filter has to know the story is false. That is not a pattern-matching problem. It is the hardest problem there is.

What ChatGPT does instead

OpenAI's answer is to treat the AI agent like a human customer-support agent: someone who acts for the company, talks to strangers all day, and will sometimes be fooled. Nobody fixes that by hoping the agent never gets fooled. You put limits around what a fooled agent can do — refund caps, flagged emails, approvals.

Applied to ChatGPT, OpenAI describes three things:

1. Training first. Most attacks OpenAI sees try to get the assistant to take secret information from a conversation and send it to a third party. "In most of the cases we're aware of, these attacks fail because our safety training causes the agent to refuse."

2. No silent sends. For the cases where the agent is convinced, a mechanism called Safe Url detects when information from the conversation would be transmitted to a third party. Then ChatGPT either shows the user what would be sent and asks to confirm, or blocks it and tells the agent to find another way. The same mechanism covers navigation in Atlas and searches in Deep Research.

3. Sandboxes. Apps and Canvas run in a sandbox that detects unexpected communications and asks the user for consent.

The principle behind all three, in OpenAI's words: "potentially dangerous actions, or transmissions of potentially sensitive information, should not happen silently or without appropriate safeguards."

What to copy if you build AI agents

OpenAI's closing recommendation is the most usable line in the post:

"When integrating an AI model with an application system, we recommend asking what controls a human agent should have in a similar situation and implementing those."

In practice, for your own agents:

  • List the sinks. Anything the agent can do that leaves the system — send an email, call an API, open a link, write to a shared file. That's where an injection does damage.
  • Put a human gate on the dangerous ones. Sending data out, spending money, deleting things: show what will happen and ask first.
  • Cap what one mistake can cost. Limits on amounts, recipients and frequency, enforced by plain code the model can't talk its way around.
  • Don't buy a filter as the whole answer. An input classifier can help with crude attacks. It is not the defense against the ones that work.

The catch

OpenAI is describing its own defenses on its own blog. We have not tested them, and we can't verify how often they work. The post itself doesn't claim the problem is solved — it says OpenAI keeps working on it in both its security architecture and the training of its models.

And this post is from March 2026, not this week. What's worth your attention is not the date. It's the shift in thinking: if the attacks that work are social engineering, then the question for anyone shipping an agent is no longer "can I block every bad input?" It's "what's the worst this agent can do when it's fooled?"

Source: OpenAI, "Designing AI agents to resist prompt injection", 11 March 2026.

Want the calm version of AI news like this, once a week? Subscribe to the Sharp AI Hub newsletter →