Prompt injection: data leakage risks of AI assistants

What is prompt injection?

Prompt injection is a cyberattack against large language models (LLMs): the attacker disguises malicious inputs as legitimate ones [10]. It targets services that rely on AI [11]. Its goal is to push the model to ignore its original instructions [13] and follow the attacker's instructions instead [27].

The three elements of the definition

• An input that appears legitimate: the trapped text looks like an ordinary request or content [10].
• An effect on the instructions: the model abandons the instructions that framed its behavior [27].
• A beneficiary: the attacker's orders are now followed, rather than those of the designer or the legitimate user [27].

One source presents it as a way to hijack an AI without exploiting a flaw in the classic sense [28]. For a CISO, this changes the nature of the problem: the manipulation goes through the model's normal channel of use, not through a code vulnerability that a patch would close.

Two common analogies, to be used with caution

Two comparisons recur in vendor literature:

• SQL injection: Flutteris describes prompt injection as "the AI equivalent of SQL injection" [27].
• Social engineering: OpenAI classifies it as a social engineering attack specific to conversational AI [16].

Both sources are commercial actors. Their comparisons are pedagogical: they help situate the risk relative to known attacks, but they do not demonstrate a technical equivalence. The first emphasizes the crafted input that hijacks a system, the second the persuasion of an interlocutor. Prompt injection borrows a little from both without being confused with either.

It remains to understand why a model lets itself be hijacked this way, and through what paths the malicious input reaches it: these are the subjects of the following sections.

Why do LLMs let themselves be manipulated?

LLMs can be manipulated because they do not separate the instructions they are given from the data they read: the model treats all inputs as one continuous prompt [34].

A boundary that fades

When an agent receives "analyze this contract," "summarize this email," or "rephrase this paragraph," it executes the instruction. But to accomplish the task, it reads a document, and the content of that document does not have, for it, a status distinct from the initial instruction [34]. A text slipped into the document can therefore be taken as an order. AI thus becomes an attack vector because it erases the boundary between data and instructions [21].

The consequences are visible in the indirect case: the AI then executes the hidden orders as if they came from the legitimate user [36]. An analysis by Palo Alto Networks on coding assistants points the same way: the model does not reliably distinguish system instructions from user requests [33].

An attack that exploits no flaw

Prompt injection hijacks an AI without exploiting a flaw in the traditional sense [28]. It does not rely on an implementation error. It exploits the model's normal operation to turn it against its designers [24]. The model does what it was designed to do: follow the instructions it reads.

The comparisons proposed by the sources reflect this particularity. Some liken the attack to SQL injection, since the attacker pushes the model to abandon its original instructions in favor of their own [27]. Others see it as a social engineering attack specific to conversational AI [16]. The second parallel is the more telling: no protection is forced, the model is persuaded.

This root cause weighs more heavily as assistants open up to internal data and the web. This extension of their capabilities introduces a form of risk that is still poorly framed [28]. The more the agent reads sources it does not control, the more opportunities there are to slip in an instruction. Hence the idea, defended by some sources, that security rests on reducing the attack surface, least privilege, and controls at every step rather than on a single countermeasure [39].

What is the difference between direct and indirect injection?

Direct injection is typed by the attacker into the request they send to the model. Indirect injection is hidden in a source the assistant processes, and the legitimate user often does not notice it. One source defines it as the insertion of malicious instructions into a request or a data source in order to influence the model's behavior [23].

Where the instructions hide

• Direct: the instruction arrives through the conversation channel, in the form of a request.
• Indirect: the instruction passes through the data the AI processes, often without the user's knowledge [18]. It can be a document, a web page, or another data source.

The second case works because the AI treats all inputs as one continuous prompt. It does not separate the instructions the user gave it ("analyze this contract," "summarize this email") from the data it is reading [34].

A website that traps an assistant

A website can be enough to hijack an assistant. An article about Claude describes this scenario: the page content carries hidden orders, and the AI executes them as if they came from the legitimate user. The possible consequences are data leaks, unauthorized actions, or biased responses [36]. The user only asked to consult or summarize a page. The attacker, for their part, never had access to the conversation.

A point of attention for agents

The academic sources in this file treat indirect injection as a distinct risk. An article on delegating intent to an AI agent cites prompt injection, "including indirect," as the first risk, and data leakage as the second [8]. A book on responsibility when machines act also explicitly addresses indirect prompt injection [5]. For an agent that reads external sources and then acts, the attack surface is therefore not limited to what the user types: it includes everything the agent opens, reads, or queries.

How does a prompt injection cause a data leak?

A prompt injection causes a data leak through three documented pathways: unwanted actions by the agent, the model's outputs, and exposure of the system prompt. These leaks caused by an attacker must be distinguished from those arising from everyday use.

Three leak pathways caused by the attack

• Unwanted agent actions. According to Bruno Lussato, quoted by franceinfo, a hacker can craft prompts that push the AI agent to act against its role: data leakage, code execution, or generation of malicious content [26]. Sensitive data leakage also figures among the main risks listed, along with partial takeover of systems and reputational harm [25].
• Leak through model outputs. Among the frequent threats, one source cites "from prompt injection to leakage through outputs" [2]. What the model responds with can therefore become the output channel for information.
• Leak of the system prompt. It is technically related to prompt injection and jailbreaking [30]. API keys and credentials are among the exposed elements [30]. It can be inferred that a system prompt should not contain secrets.

Leaks through everyday use: a different mechanism

In this second case, no attacker is involved. Confidential information is passed involuntarily to third-party services, through the AI interfaces employees use [20]. The cause is a usage behavior, not a manipulation of the model.

The two families converge on the question of data entered into prompts. An analysis of Shadow AI recommends asking what data a tool incorporates into its prompts [3]. Information sent to an uncontrolled service is exposed, whether or not an injection takes place.

This distinction matters for a CISO: the two leaks do not call for the same measures. For one of the sources, with this type of assistant, "the real security issue is governance" [2].

What other damage do agents, memory, and code cause?

The risk of prompt injection is not limited to data leakage: as soon as an assistant has tools, memory, or access to code, a hijacked instruction can trigger actions and not just disclosures.

Unintended actions and propagation

According to Bruno Lussato, quoted by franceinfo, a hacker can craft prompts that cause unwanted agent actions: data leakage, code execution, or generation of malicious content [26]. IBM also describes a propagation scenario: the malicious prompt asks the assistant to forward it to other contacts [10]. The attack then does not remain confined to the first victim.

Memory poisoning

Hidden instructions can be quietly injected into the memory of mainstream assistants: Microsoft Copilot, ChatGPT, Claude, Perplexity, and Grok [29]. The target is no longer an isolated conversation but the assistant's memory.

Hijacked coding assistants

Palo Alto Networks' Unit 42 team found that these assistants can be hijacked by users or malicious actors, via chat, autocomplete, or unit test generation. The consequences observed are the insertion of backdoors, the leak of sensitive information, and the creation of harmful content [33]. Secure Code Warrior reports, for its part, a coding agent tricked into writing code vulnerable to SQL injection and installing system tools [17].

What these sources are worth

These cases come from a mainstream media outlet (franceinfo) and from commercial actors: IBM, Palo Alto Networks, Secure Code Warrior. The article reporting the memory poisoning is also from a commercial site, and the available excerpt does not allow the authors of the study it cites to be identified [29]. None of these items is a primary study reviewed here. They illustrate plausible and documented scenarios, but they neither measure their frequency nor their success rate. A CISO will therefore use them to draw up a list of scenarios to test, not to quantify a risk.

What does the OWASP say, and what are the GDPR obligations?

Prompt injection is presented by several sources as the number 1 vulnerability according to the OWASP. According to one commercial source, a data leak it causes falls under the same GDPR regime as a classic hack.

Its place in the OWASP framework

The OWASP is described as an international reference organization in cybersecurity. A specialized site states that it ranks prompt injection as vulnerability number 1 [24]. Another cybersecurity provider speaks of the "number one danger for LLMs according to the OWASP" [37]. These two sources are professional content, not the framework itself. For an authoritative ranking, one must consult the OWASP document directly.

An article on the "OWASP Top 10 LLM 2026" mentions, for its part, a focus on agentic risks, "Unbounded Consumption," prompt injection, and OWASP ASI [40]. The file does not allow the 2026 ranking to be detailed, nor to say what place prompt injection occupies in it. We therefore do not assert it.

GDPR obligations in case of a leak

According to core.security, a commercial source, a data controller who suffers a customer data leak caused by prompt injection faces the same legal obligations as with a classic hack [32]. The source cites:

• notification to the CNIL within 72 hours;
• financial penalties that can reach 4% of worldwide annual turnover [32].

According to this source, the attack technique therefore does not change the classification of the incident. For a CISO, prompt injection is a possible cause of a data breach, to be integrated into incident management procedures on the same footing as an intrusion.

The file contains neither the text of the GDPR nor any publication from the CNIL. These elements are reported by a commercial actor, and the DPO will need to confirm them against the official texts.

How can the risk be reduced in practice?

No single countermeasure is enough: defense in depth combines reducing the attack surface, least privilege, and controls at every step. [39] Each layer limits the damage that the others have not prevented.

Levers the file allows us to support

• Limit what the agent can reach. Prompt injection is less of a concern if the agent cannot access sensitive data. [15] Databricks, a data platform vendor, therefore recommends restricting the agent's capabilities, in particular its access to sensitive systems and private data. [15]
• Place the human and rights in the right spot. A specialized source cites human validation and granular access controls, particularly in the financial sector. [35] The idea is that a sensitive action does not rest on the model's decision alone.
• Secure before deploying. According to Flowpi, a commercial vendor, chatbots and copilots must be protected before going into production. Without safeguards, prompt injection and data leaks threaten the brand, customers, and return on investment. [22]
• Treat the issue as a governance matter. For an enterprise assistant, "the real security issue is governance." This implies a data security posture dedicated to AI. [2]

A limit to acknowledge

These measures reduce the risk without eliminating it. A test involving 2,000 hackers against an AI assistant concludes that prompt injection remains a real security problem. [35] The sources in the file do not present any infallible technical measure, and those that detail measures are mostly vendors or consultants. For a CISO, the practical consequence is this: the architecture must be designed on the assumption that an injection will eventually get through, and one must ask what the agent could then read, execute, or transmit.

How does Oupi frame its agent's actions?

Oupi frames its agent's actions according to two principles: sensitive acts are subject to human validation, and the agent's execution is isolated. This section describes these principles without detailing their implementation.

Two principles

• Human validation of sensitive acts. The agent does not carry a consequential action through to completion on its own. A person validates it before it is carried out.
• Isolated execution. The agent works within a separate perimeter, which limits what a hijacked action could reach.

The link with general recommendations

These two principles align with the measures recommended elsewhere in this file.

• Limit the agent's capabilities. Databricks, a data platform vendor, states that prompt injection is less of a concern if the agent cannot access sensitive data. It recommends restricting access to sensitive systems and private data [15]. Isolated execution goes in this direction: it reduces what a manipulated agent can touch.
• Maintain human oversight. An article by Roboto, a commercial actor, cites human validation, combined with granular access controls, among the responses adopted in the financial sector [35]. Validation of sensitive acts follows this logic: an injected instruction is not enough on its own to trigger a critical action.
• Reduce the attack surface. According to a technical guide on securing LLM applications, security rests on reducing the surface, least privilege, and controls at every step, not on a single countermeasure [39]. Human validation and isolation are two of these controls.

The limits of this approach

These measures reduce the risk; they do not eliminate it. A test conducted with 2,000 participants on an AI assistant concludes that prompt injection remains a real security problem [35]. Human validation also presupposes that the person understands what they are approving. It complements the other protections (output control, detection, monitoring) and does not replace them, in line with the defense-in-depth approach [39].

Oupi's trust commitments are presented on the Confiance page.