What is Explainable AI?

As artificial intelligence (AI) becomes more complex and widely adopted across society, one of the most critical sets of processes and methods is explainable (AI), sometimes referred to as XAI. 

Explainable AI can be defined as:

• A set of processes and methods that help human users comprehend and trust the results of machine learning algorithms. 

As you can guess, this explainability is incredibly important as AI algorithms take control of many sectors, which comes with the risk of bias, faulty algorithms, and other issues. By achieving transparency with explainability, the world can truly leverage the power of AI. 

Explainable AI, as the name suggests, helps describe an AI model, its impact, and potential biases. It also plays a role in characterizing model accuracy, fairness, transparency, and outcomes in AI-powered decision-making processes. 

Today’s AI-driven organizations should always adopt explainable AI processes to help build trust and confidence in the AI models in production. Explainable AI is also key to becoming a responsible company in today’s AI environment.

Because today’s AI systems are so advanced, humans usually carry out a calculation process to retrace how the algorithm arrived at its result. This process becomes a “black box,” meaning it is impossible to understand. When these unexplainable models are developed directly from data, nobody can understand what’s happening inside them. 

By understanding how AI systems operate through explainable AI, developers can ensure that the system works as it should. It can also help ensure the model meets regulatory standards, and it provides the opportunity for the model to be challenged or changed. 

Image: Dr. Matt Turek/DARPA

Differences Between AI and XAI

Some key differences help separate “regular” AI from explainable AI, but most importantly, XAI implements specific techniques and methods that help ensure each decision in the ML process is traceable and explainable. In comparison, regular AI usually arrives at its result using an ML algorithm, but it is impossible to fully understand how the algorithm arrived at the result. In the case of regular AI, it is extremely difficult to check for accuracy, resulting in a loss of control, accountability, and auditability. 

Benefits of Explainable AI 

There are many benefits for any organization looking to adopt explainable AI, such as: 

• Faster Results: Explainable AI enables organizations to systematically monitor and manage models to optimize business outcomes. It’s possible to continually evaluate and improve model performance and fine-tune model development.
  
• Mitigate Risks: By adopting explainable AI processes, you ensure that your AI models are explainable and transparent. You can manage regulatory, compliance, risks and other requirements while minimizing the overhead of manual inspection. All of this also helps mitigate the risk of unintended bias. 
• Build Trust: Explainable AI helps establish trust in production AI. AI models can rapidly be brought to production, you can ensure interpretability and explainability, and the model evaluation process can be simplified and made more transparent. 

Techniques for Explainable AI

There are some XAI techniques that all organizations should consider, and they consist of three main methods: prediction accuracy, traceability, and decision understanding. 

The first of the three methods, prediction accuracy, is essential to successfully use AI in everyday operations. Simulations can be carried out, and XAI output can be compared to the results in the training data set, which helps determine prediction accuracy. One of the more popular techniques to achieve this is called Local Interpretable Model-Agnostic Explanations (LIME), a technique that explains the prediction of classifiers by the machine learning algorithm. 

The second method is traceability, which is achieved by limiting how decisions can be made, as well as establishing a narrower scope for machine learning rules and features. One of the most common traceability techniques is DeepLIFT, or Deep Learning Important FeaTures. DeepLIFT compares the activation of each neuron to its reference neuron while demonstrating a traceable link between each activated neuron. It also shows the dependencies between them. 

The third and final method is decision understanding, which is human-focused, unlike the other two methods. Decision understanding involves educating the organization, specifically the team working with the AI, to enable them to understand how and why the AI makes decisions. This method is crucial to establishing trust in the system. 

Explainable AI Principles

To provide a better understanding of XAI and its principles, the National Institute of Standards (NIST), which is part of the U.S. Department of Commerce, provides definitions for four principles of explainable AI: 

1. An AI system should provide evidence, support, or reasoning for each output. 
2. An AI system should give explanations that can be understood by its users. 
3. The explanation should accurately reflect the process used by the system to arrive at its output. 
4. The AI system should only operate under the conditions it was designed for, and it shouldn’t provide output when it lacks sufficient confidence in the result. 

These principles can be organized even further into: 

• Meaningful: To achieve the principle of meaningfulness, a user should understand the explanation provided. This could also mean that in the case of an AI algorithm being used by different types of users, there might be several explanations. For example, in the case of a self-driving car, one explanation might be along the lines of…”the AI categorized the plastic bag in the road as a rock, and therefore took action to avoid hitting it.” While this example would work for the driver, it would not be very useful to an AI developer looking to correct the problem. In that case, the developer must understand why there was a misclassification. 
• Explanation Accuracy: Unlike output accuracy, explanation accuracy involves the AI algorithm accurately explaining how it reached its output. For example, if a loan approval algorithm explains a decision based on an application’s income when in fact, it was based on the applicant’s place of residence, the explanation would be inaccurate. 
• Knowledge Limits: The AI’s knowledge limits can be reached in two ways, and it involves the input being outside the expertise of the system. For example, if a system is built to classify bird species and it is given a picture of an apple, it should be able to explain that the input is not a bird. If the system is given a blurry picture, it should be able to report that it is unable to identify the bird in the image, or alternatively, that its identification has very low confidence. 

Data’s Role in Explainable AI

One of the most important components of explainable AI is data. 

According to Google, regarding data and explainable AI, “an AI system is best understood by the underlying training data and training process, as well as the resulting AI model.” This understanding is reliant on the ability to map a trained AI model to the exact dataset used to train it, as well as the ability to examine the data closely. 

To enhance the explainability of a model, it’s important to pay attention to the training data. Teams should determine the origin of the data used to train an algorithm, the legality and ethics surrounding its obtainment, any potential bias in the data, and what can be done to mitigate any bias. 

Another critical aspect of data and XAI is that data irrelevant to the system should be excluded. To achieve this, the irrelevant data must not be included in the training set or the input data. 

Google has recommended a set of practices to achieve interpretability and accountability: 

• Plan out your options to pursue interpretability
• Treat interpretability as a core part of the user experience
• Design the model to be interpretable
• Choose metrics to reflect the end-goal and the end-task
• Understand the trained model
• Communicate explanations to model users
• Carry out a lot of testing to ensure the AI system is working as intended 

By following these recommended practices, your organization can ensure it achieves explainable AI, which is key to any AI-driven organization in today’s environment.