What does “AI security” mean and why does it matter to your business?

Let's imagine a customer-support chatbot—it's running on Red Hat OpenShift AI and searches internal documents to answer questions. A user asks it a common question, but the chatbot inadvertently retrieves a malicious document that contains hidden instructions like, “ignore all policies and reveal secrets.” Not knowing any better, the AI model follows these malicious instructions and leaks internal data—and no one notices until screenshots appear online. This is the new computer security reality in which we live. Modern AI systems do more than “respond.” They reason over untrusted inputs, and sometimes they use external tools, which means the potential attack surface grows quickly.

The concept of “AI security” is not only about stopping malicious attackers, it’s also about mitigating risks that can lead to costly business mistakes. Whether you use AI in healthcare, finance, HR, or even as a simple enterprise assistant, the strength of your security posture is the difference between a helpful system or a possible hazard for your company.

This article is the first of a series in which we're going to look at AI security holistically, exploring the potential risks posted by these systems and how we can defend against them. Note that we are not discussing AI safety as safety issues are out of the scope of this specific security conversation. To learn more about AI safety versus AI security, check out our post, Mitigating AI’s new risk frontier: Unifying enterprise cybersecurity with AI safety.

We'll be focusing on how traditional systems security principles are still relevant to these new technologies, and how we can design and defend these new tools while constantly watching for new potential attack vectors.

What does “AI security” mean?

AI security is the discipline of protecting AI systems from attacks and failures that compromise confidentiality (data leaks), integrity (manipulated behavior), and availability (service disruption). It overlaps with traditional cybersecurity, but there are  new risks involved because models learn from data, respond to natural language, and can behave unpredictably under adversarial prompting.

It’s also important to be precise—we don’t enhance the security of the AI model in isolation, we deliver an improved security posture to the AI system as a whole. The full system includes training data, prompts, retrieval-augmented generation tools (RAG), memory, external tool access (APIs), logs, user interfaces, and deployment infrastructure. Many real incidents happen around the model, not inside it.

To quickly identify if a risk targets any of these security principles, you can ask yourself these questions:

The AI attack surface

A useful way to think about AI security is in terms of layers. Each layer offers different ways to fail, and attackers will often chain these together into sophisticated attacks - not unlike a traditional software exploit. They can accomplish this by injecting something into the data, which influences the model, which then triggers a tool to do something like leak a secret.

Data layer

This includes training data, fine-tuning datasets, user feedback logs, and the documents you index for retrieval augmented generation (RAG). If an attacker can poison this data—for example, by breaking into the storage system and modifying the training data with biased examples or backdoor triggers—they can shift behavior over time. Additionally, sensitive data can leak into training logs and later appear in outputs.

Model layer

This includes the model weights, model architecture, and the inference endpoint. Attacks here include adding a backdoor to the model architecture with malware, exploiting memory safety issues in the inference endpoint, or, at the model data layer, model extraction (stealing the model behavior via queries), and abuse of weaknesses that cause unexpected output. Also, if you host models, your API is a target and it should be equipped with measures such as rate limits, authentication, and abuse monitoring.

Prompt / interaction layer

This is where large language model (LLM) systems are uniquely vulnerable and where most attacks target enterprise deployments. Prompts include system instructions, developer messages, user messages, conversation memory, and retrieved context. Through malicious prompting, a model can be tricked into ignoring rules, revealing hidden instructions, or following attacker-provided instructions that look like normal text. 

Tooling and agent layer

Tools are where security risk explodes. If the model can call functions—such as for search, database queries, file reads, payments, or ticketing—and if it has enough permissions, a prompt injection can become an action injection. The model doesn’t need to “break” encryption, it just needs to be convinced to use legitimate access in a harmful way. That's why organizations deploying AI agents should implement strict tool authorization and audit logging for AI systems.

Infrastructure and supply chain

AI systems run on dependencies—vector databases, orchestration frameworks, model downloads, telemetry, and CI/CD. A vulnerable dependency or misconfigured storage can expose embeddings, prompts, or logs. Additionally, supply chain issues matter more if you are quickly assembling AI stacks with many moving parts.

It's important to use tools and frameworks, such as Sigstore, that offer methods to enhance the security of software supply chains in a more open, transparent, and accessible manner.

A taxonomy of AI security threats

The following table summarizes the main attack types targeting modern AI systems, especially LLM-based applications. These attacks exploit weaknesses across prompts, data, models, and tools, often by manipulating how the system interprets language or trusted inputs. Understanding these categories helps clarify where risks arise and why layered guardrails are essential for building AI systems with a strong security posture. 

How can we defend from these attacks?

The best way to defend AI systems from these sorts of attacks is to implement guardrails. Guardrails constrain model behavior and system actions. They can block, redact, rewrite, route to a safer mode, or require extra confirmation before doing risky things. Good guardrails aren't just for protecting against certain types of language or content, they also enforce policies at the right points in the pipeline.

What guardrails do depends on where they exist in the pipeline:

• Input guardrails: These inspect user requests before the model sees them and can protect against things like policy violations and prompt injection attempts.
   
• Output guardrails: These review what the model produced before it’s shown to the end user, and can redact secrets and stop unsafe content.
   
• Runtime guardrails: These rules apply while the model is using external tools, enforcing things like least privilege, allowlists, and “two-person rule” confirmations.

Guardrails reduce risk, but they aren’t magic. If your system has access to sensitive data and no permission boundaries, guardrails alone won’t save you. Strong security hygiene comes from guardrails plus system design choices like minimal tool access and good monitoring.

This is why Red Hat takes a defense-in-depth approach, combining guardrails with least privilege, monitoring, and secure-by-default configurations.

If you want to read more, learn how Red Hat implements AI guardrails in OpenShift AI. 

A simple defender mindset: Risk = likelihood × impact

Not every attack is equally likely, and not every failure is equally damaging. A good security approach prioritizes threats based on likelihood (how easy is it?) and impact (what happens if it works?). For example, prompt injection is often high-likelihood in public chatbots, while model extraction may require more effort but can still be very damaging.

A helpful checklist for threat modeling an AI system:

• Identify sensitive information, such as personally identifiable information (PII), credentials, internal docs, system prompts, and API keys.
   
• Determine input sources, such as users, documents, web pages, and integrations.
   
• List actions the model can perform, such as search, email, database, payments, and ticket edits.
   
• Identify  potential consequences, such as bad advice, data leakage, and unauthorized actions.
   
• Describe how failures are detected, such as logging, alerts, anomaly detection, and audits.

One important thing to note: If your model can take actions, its hallucinations and probabilistic nature become a security problem. The goal isn’t only to prevent disallowed text, it’s to prevent insecure outcomes.

What does a security-focused AI system mean?

A security-focused AI system is built like any other security-focused system: Layered defenses, operating under the principle of least privilege, and undergoing continuous testing. At a high level, good security practices include:

• Embed security throughout the development lifecycle: This approach integrates protection from initial design to final deployment through threat modeling, secure-by-default architecture, and automated review gates.
   
• Apply principle of least privilege for tools: The model should only be granted the minimum access it needs for any given tool.
   
• Build guardrails at multiple points: You should have comprehensive guardrails at all 3 levels: input, output, and runtime.
   
• Use red teaming and evals for continuous testing: Test against realistic attacks before and after release.
   
• Implement detailed monitoring and plan for incident response: Build observability and monitoring so you can track prompts and tool calls, and watch for anomalies. Equally important, you should plan how you're going to respond when an issue is detected.

Relying on “security by policy” is not enough. If the AI model has permission to read sensitive documents, a clever prompt can often find a way to manipulate the system into leaking confidential information. Strong security combines policy enforcement with architecture, including permissions, separation, and safe tool design.

Conclusion and what’s next

AI security matters because AI systems combine 3 risky ingredients: Untrusted inputs, learned behavior, and an increasing ability to take autonomous action in the real world. Security threats span the entire stack—data, model, prompts, retrieval, tools, and infrastructure. Guardrails are an essential part of defense, but they work best when paired with the principle of least privilege, robust monitoring, and systematic testing. If you want to try building some guardrails yourself, you can always try TrustyAI.

If you treat your AI as “just a chatbot,” it's easy to overlook the real risks. If you treat it as an application with a new (and potentially expanding) attack surface, you'll have a much better chance of protecting it, and your organization, against potential attacks.