BlogJuly 10, 2026
Blog
What Prompt Injection Shows Us About AI And Its Trust Boundaries
Last updated: 10 July 2026
Daniel Cork
What Prompt Injection Shows Us About AI And Its Trust Boundaries
Imagine asking an AI assistant to sort your inbox, identify the most important messages, and help draft replies. Now imagine one of those emails contains hidden instructions the AI can read but you cannot. Instead of simply summarising, the system is quietly nudged into behaving differently: ignoring prior instructions, exposing internal information, or taking unintended actions. Variants of this issue have already been demonstrated in systems that consume everyday content such as emails, webpages, and calendar invites.
That is why prompt injection matters. This goes beyond chatbots behaving oddly. It is about systems assigning authority to the wrong input, and once that happens, the question is not just security but whether the system is under meaningful control.
Prompt injection is often framed as a novel attack or a collection of jailbreaks. That is partly true, and any serious software team should treat it as a real attack surface. It is widely recognised as a major LLM risk and an ongoing security challenge. The more revealing question is not whether systems can be manipulated it is why this is so often possible.
Why prompt injection is more than a jailbreak
Anyone familiar with AI has seen versions of this: models revealing restricted information, ignoring instructions, or behaving unpredictably once properly tested. This matters because users do not always behave as intended. Most are reasonable, but some will push boundaries or experiment. Systems must be designed with that in mind.
The consequences do not need to be dramatic to matter. Unexpected behaviour, internal leakage, unintended tool use, or unnecessary costs are all risks. If a system can be pushed off course that easily, its boundaries were never as strong as they appeared.
Calling this a “jailbreak” is slightly misleading. It suggests a clever attacker defeating a robust system. More often, the system is simply too casual about what it trusts.
What happens when systems trust the wrong thing
Prompt injection exposes how AI systems assign authority. In conventional software, untrusted input is not treated as equivalent to system logic. Control and data are separated. A well-written message does not gain authority just because it looks legitimate.
A useful analogy is a secure building. Visitors may be legitimate, but they are not given the same access as staff. They cannot enter control rooms or issue instructions simply by asking persuasively. There is also a slightly unsettling parallel to Doctor Who’s psychic paper: show the right thing in the right context, and it is treated as authoritative without being questioned. Prompt injection works in much the same way. The issue is not the input itself, but the system’s failure to distinguish between data and instruction.
Many systems still combine user input, retrieved content, and governing rules into a single stream. This can appear polished, but from an engineering perspective it is loose: the rules sit alongside the material they are meant to control. Once that boundary blurs, it is not surprising the system can be steered off course. Prompt injection is therefore less a clever trick and more a structural warning. Untrusted content is sitting too close to the logic that defines behaviour.
This also explains why common mitigations can feel insufficient. Filters and stronger prompts help, but adding instructions is not the same as establishing control if the structure remains unclear. The real requirement is clearer separation: what is trusted, what is not, and what is allowed to influence behaviour at all.
Why stronger boundaries matter
Any useful AI system will eventually be tested. Some users will probe limits, deliberately or not, that is normal, what matters is whether the system is designed with that expectation.
The key distinction is not between systems that can or cannot be attacked. It is between systems built assuming pressure will come and those built hoping it will not. Trust is not tested when everything works it is testing under awkward or hostile input. If a system performs well only on the happy path but becomes unpredictable under pressure, it is not reliable.
Prompt injection is therefore a design test. It reveals whether a system has meaningfully separated trusted instructions from untrusted input, and whether its boundaries are robust enough for real conditions.
That matters because trust depends on behaviour remaining constrained and understandable when conditions degrade. If a system cannot distinguish instruction from input, oversight weakens and confidence becomes dependent on appearances rather than engineering.
Ultimately, prompt injection is not just about breaking chatbots. It is a reminder that AI systems cannot be built on implicit trust. Inputs should not be authoritative by default, and control must be explicit. If an ordinary email can change system behaviour, the issue is not just the exploit, the boundaries were never strong enough to justify our trust.
This article is part of our Building Trustworthy AI Systems series exploring how reliability, control, and system design are important for engineering AI systems.
These ideas inform the development of Cyclone Sage, our AI assistant for MCNP being built with a focus on structured, reliable system design.
If you’re working on similar challenges or exploring how AI can be applied reliably in technical environments, feel free to get in touch at support@orthrussoftware.com
