What is explainable AI?

Explainable AI (XAI) is a set of techniques applied during the machine learning (ML) lifecycle, with the goal of making AI outputs more understandable and transparent to humans. Ideally, XAI answers questions like:

• Why did the model do that?
• Why not something else?
• When was the model successful?
• When did the model fail?
• When can I trust the model’s output?
• How can I correct an error?

Explainable AI should be able to demonstrate its competencies and understandings; explain its past actions, ongoing processes and upcoming steps; and cite any relevant information on which its actions are based. In short, explainable AI encourages AI systems to “show their work.”

Explore Red Hat AI

Businesses are increasingly relying on AI systems to make decisions. For example, in healthcare, AI may be used for image analysis or medical diagnosis. In financial services, AI may be used to approve loans and automate investments.

These decisions affect and can pose risks to people, environments, and systems. Transparency and accountability are essential to creating trust between humans and AI systems. Meanwhile, a lack of understanding can lead to confusion, errors, and sometimes legal consequences.

By prioritizing transparency and explainability, you can build AI that’s not only technically advanced, but also safe, fair, and aligned with human values ​​and needs. 

Interpretability vs. explainability

In the context of XAI, explainability and interpretability are often used interchangeably, which can lead to confusion. 

Interpretability refers to the degree to which a human can understand a model’s internal logic. Interpretability pertains to the state of a model and exists on a spectrum. A model with high interpretability has features that are inherently understandable—meaning a non-expert can comprehend the relationship between inputs and outputs. A model with low interpretability has inner workings that are too complex for a human to understand.

Explainability describes the process of generating a justification or explanation. Explainability is achieved through a set of techniques (XAI techniques) applied to a complex model in order to reveal how and why it made a specific decision. When a model’s logic is too complex to interpret firsthand, XAI techniques can help you better understand why the model behaved the way it did. 

When high interpretability provides sufficient transparency, external explainability is generally not needed. Low interpretability—a lack of inherent transparency—creates a need for external explainability to establish trust and understanding in a model.

Machine learning is the core technology of AI applications. It’s a process where a computer uses an algorithm to learn from data and create a model. As the machine (computer) uses data and an algorithm to learn, a data scientist supports the creation of a model by choosing 1 or several training techniques. 

How to train your model

To create a machine learning model, you need 3 things: 

1. Data

This can be numbers, text, audio files, video clips, and/or images. The amount of data you need depends on the complexity of the problem you’re trying to solve, the quality of the data, and the complexity of the algorithm you choose. 

If you choose a simple algorithm, like linear regression (with the goal of finding a straight line within data on a scatter plot), you might only need a few dozen data points. If you choose a complex algorithm, like neural networks, you may need millions or billions of data points. 

2. An algorithm

The algorithm is a recipe or formula the computer uses as it learns. Algorithms must be well defined and have a clear stopping point. The main goal of an ML algorithm is to find patterns in data so the machine can make decisions without being explicitly programmed for each task.

Some algorithms, like decision trees, are designed to produce a traceable and straightforward output. Think of decision trees like a flow chart—easy to understand and correct, if necessary. 

Next, consider the random forest algorithm. This involves training hundreds of decision trees, then asking each tree to “vote,” which creates the final output. Because no human can trace the logic of hundreds of flowcharts, the algorithm becomes almost impossible to understand.

3. A training technique

This refers to a technique a data scientist uses when designing, implementing, and fine-tuning the computer’s learning process. This learning process, called training, uses methods such as:

• Supervised learning: You give the model a dataset where all the input data is labeled with the correct answer. Its job is to study the relationship between the input data and its correct label. Supervised learning can help predict what will happen.
  • For example, you give the model 100,000 photos of horses and bears, and correctly label each photo “horse” or “bear.” The model will learn the patterns and can eventually label a new photo correctly.
• Unsupervised learning: You give the model a dataset without any labeled data. Its job is to find patterns and associations within the data on its own. Unsupervised learning can help uncover existing patterns.
  • For example, you give the model data about customer shopping behavior. The model might discover that shoppers who purchase dog food have a 60% probability of buying walking shoes.
• Reinforcement learning: You give the model a goal and a set of rules, but no labeled data. Its job is to learn by interacting and getting “rewarded” or “penalized” for its actions. Reinforcement learning can help make suggestions about what actions to take next.
  • For example, a model might learn chess by playing millions of games. It's “rewarded” for moves that lead to a win and “penalized” for moves that lead to a loss. The process helps the model “learn” how to play chess.

Once the machine takes the data and applies the algorithm and training technique, a model is born. 

A note on neural networks

Neural network is another type of machine learning algorithm. Inspired by the human brain, it passes data through different layers of interconnected nodes called neurons. As the data goes through each layer, it’s assigned different “weights,” which determine how it passes through the next layer (and the next, and the next) before becoming a final output. 

Neural networks tend to have an input layer and an output layer. Sometimes they have hidden layers. Hidden layers can make models less transparent. This is especially true when hidden layers are large or a model has many hidden layers. When a neural network has multiple hidden layers, it can be classified as a deep neural network, and the methodology used to train it is called deep learning. 

But how is a hidden layer created? 

A hidden layer isn’t an instance of a machine creating a mind of its own. Hidden layers happen when a data scientist directs a machine to make its own connections within predesigned layers. That's when its final, learned logic becomes too complex for humans to understand.

A black box refers to an AI model that’s too complex to understand and/or doesn’t show its work. This creates a scenario where no one—including the data scientists and engineers who created the algorithm—is able to explain exactly how the model arrived at a specific output.

For example, GPT4, one of the neural networks that powers ChatGPT, performs more than 3 trillion mathematical calculations to generate a single word. If you wanted to check that math by hand and could do 1 calculation per second, it would take you 95 years to replicate the calculations needed to generate 1 word. In cases like this, you can confirm the output is correct, but checking the work is nearly impossible. 

The lack of interpretability in black-box models can create harmful consequences when used for high-stakes decision making, especially in high-risk industries like healthcare, transportation, security, military, legal, aerospace, criminal justice, or finance. 

You can think of explainable AI as a method for looking inside black boxes. 

Are all black box models harmful?

The mysterious nature of black-box systems is not inherently harmful. But these systems can pose significant risk when used in high-stakes situations. Black-box AI systems can result in:

Bias and discrimination. When AI systems are trained on data that’s inherently biased, the patterns will likely repeat themselves. Consider a hiring tool trained on a company’s past 20 years of “successful” executive hires: If this pool of executives was mostly men, the system may learn to penalize résumés that include female names.

Lack of accountability. When black-box systems make an error in judgement, there’s no way to trace the system’s logic pattern. This could pose legal complexity in terms of deciding who (or what) is at fault should someone be harmed, e.g., if a medical device or autonomous vehicle using a black-box system caused a misdiagnosis or accident.

When developers are unable to understand the internal workings of an AI system, it’s much harder to fix or improve it. When the logic is hidden, it's harder to build trust with a system.

Black box vs. white box

The opposite of a black box is a white box. Also called a glass box, a white box refers to a model with internal workings that are transparent. This means a human is able to trace the entire decision-making process from input to output. Think back to the concept of interpretability: A white box is an interpretable model, while a black box requires explainability. 

Why create a black-box model when white-box models exist? It all comes down to power and performance. White-box models are easier to interpret because their inner workings are less complicated. As a result, they tend to be smaller, with less computational capacity and power. 

When a white-box model isn’t powerful or accurate enough to create the desired output, data scientists may turn to a black-box solution, e.g., when training a model on subject matter that’s complex, nuanced, and intricate, like generative AI. 

Explainable AI helps users understand how AI systems come up with their outputs. This can lead to benefits such as:

More trust. To successfully implement technologies like agentic AI, there needs to be trust between algorithms and humans. The main goal of explainable AI is to help users trust the results of their AI applications. 

Less risk. Explainable AI allows for better evaluation of models so you can make a more informed decision on which model best suits your needs. 

Better collaboration. Having an explainable model aids in cross-functional team dynamics. Consider the relationship between data scientists and clinicians during the rollout of a machine learning model in a hospital setting: The model predicts that a patient has a high risk of sepsis, and XAI shows the top factors leading to this prediction are elevated heart rate, low blood pressure, and low oxygen saturation. The clinician can understand and validate this, confirming the model is relying on medically sound advice. 

Faster troubleshooting. When data scientists are able to understand the logic of a model as it creates an output, it’s easier and faster to develop, debug, and make decisions as a human in the loop. It’s important to note that XAI doesn’t make computers go faster—in fact, it creates more computational overhead—but it does save humans time. 

Better regulatory compliance. Explainable AI can help companies comply with regulations and privacy laws. This includes the European Union’s General Data Protection Regulation (GDPR), which grants individuals the right to know the logic involved in automated decision making, as well as the “significance and the envisaged [imagined] consequences of such processing for the data subject.” In this case, XAI can help explain decisions by detailing which data points and factors influenced a particular outcome. 

The California Transparency in Frontier Artificial Intelligence Act (TFAIA) requires developers to publish transparency reports on their model’s risks and risk mitigation measures they’ve put in place. XAI helps organizations more easily identify those risks. 

Reduced model drift. Model performance can drift (degrade over time as new data comes in). XAI helps you analyze models and generate alerts for when models drift and output deviates from what’s intended. 

The field of explainable AI is continuously growing to meet the pace of AI as it quickly advances. Implementing XAI is necessary, but the solutions aren’t perfect. Limitations to keep in mind include:

Technical complexity of output. Current methods of XAI are highly technical and geared toward ML experts, not everyday users. Addressing this issue requires either more technical education and training or creating explanations to complex issues in layperson’s terms. 

High computational cost. Running XAI techniques is expensive. This is because explanation algorithms must perform extensive calculations to understand why a model created a certain output. This might mean running the model thousands of times just to understand how a single prediction was made. 

Security risks. By opening up the black box to understand a model, you run the risk of exposing how to fool the model. This can create vulnerabilities that allow malicious actors to game the system and reverse-engineer a way to breach security. 

Understanding vs. trust 

Understanding how a process works isn’t the same as trusting the process. You can understand the mechanics of a skateboard but not trust it enough to ride it. 

The topic of trust begs the question: Could it be harmful to put too much trust in AI? Skepticism is healthy, and placing too much trust in a system can expose you to potential mistakes when you don’t expect them. At the same time, if you don’t have enough trust in the systems, you can miss out on the benefits they can provide. 

Explainable AI aims to help users adjust their trust levels. When XAI provides helpful context, users can decide for themselves how much to trust the system.

Implementing XAI requires committing to more transparency in the entire machine learning lifecycle—from initial design to monitoring. There’s no single way to explain the outputs of an ML or AI algorithm. Your approach will depend on how your model is built and who your end users are. Here are some factors to consider:

Global vs. local XAI models: What level of explanation is needed?

• Global explanations provide a high-level understanding of the general patterns your model uses to make decisions. For example, in a model that predicts loan acceptance, a global explanation might say: “The model prefers to accept candidates with a high credit score.”
• Local explanations detail the factors that influenced a single decision by the model. For example, a local explanation of that same loan acceptance model might be: “John Smith was denied a loan on Nov. 19 because his credit score was 520, and his income was below the threshold of $35,000.”

Direct vs. post-hoc XAI models: How was your model designed to provide explanations?

• Direct models produce transparent and traceable results from the start, just like white-box models.
• Post-hoc models weren’t originally designed to be interpretable as they’re black-box models. But you can gain insight into how they work by applying algorithms after training is complete. These algorithms help analyze the output and produce an explanation:
  • LIME (Local Interpretable Model-agnostic Explanations) is a technique that manipulates input data to create a series of slightly different artificial data. LIME runs that artificial data through the model and observes the output, creating interpretable “surrogate” models to help explain the black-box model’s original prediction.
  • SHAP (SHapley Additive exPlanations) is a method based on cooperative game theory that calculates the contribution of each input variable and considers all possible variable combinations. It provides a unified view of how each variable contributes to the model's output and which variables are driving predictions.

Data vs. model XAI models: What type of explanation do you need?

• Data models provide an explanation based on how input data influences prediction.
• Model XAI models provide an explanation based on the inner workings of your model.

Responsible AI practices such as ethical guidelines, transparency, and bias mitigation are essential to promoting trustworthy AI systems that benefit society. 

Red Hat® AI is a platform that accelerates AI innovation and reduces the operational cost of developing and delivering AI solutions across hybrid cloud environments. 

Red Hat AI uses open source technology to meet the challenges of wide-scale enterprise AI. This means providing our customers with support and tools to implement explainability, AI governance, trustworthiness, AI guardrails, bias and drift detection, and versioning.

To deliver these capabilities, Red Hat AI integrates tools that allow users to monitor and manage the entire AI model lifecycle. This functionality is powered in part by TrustyAI, an open source toolkit of responsible AI technology that Red Hat contributes to.