They say that familiarity breeds contempt. However, in an AI context, it's more likely to breed trust and acceptance. When voice recognition systems were first developed, it seemed like science fiction. Nowadays, our children are unfazed by talking to smart gadgets and not in the least freaked out by the fact that the gadget can correctly identify them.

In a work context, we might be more cautious when new AI-based systems are introduced, but if they prove themselves to be reliable we'll eventually learn to stop worrying and love AI, as Dr. Strangelove might have said.

However, in the early days -- and particularly if anything goes wrong -- humans would be more comfortable if AI systems were better at explaining themselves. As a PwC report titled "Explainable AI Driving business value through greater understanding" describes, explainable AI (XAI) "helps to build trust by strengthening the stability, predictability and repeatability of interpretable models. When stakeholders see a stable set of results, this helps to strengthen confidence over time. Once that faith has been established, end users will find it easier to trust other applications that they may not have seen."

The PwC paper argues there are significant benefits to be gained from increasing the explainability of AI. In addition to building trust, XAI can help to improve performance, as a better understanding of how a model works enables users to better fine tune it. A further benefit is enhanced control -- understanding more about a system behavior provides greater visibility over unknown vulnerabilities and flaws.

The issue of AI trust is not confined to solutions bought from third parties but applies equally to AI systems developed in house. For example, an AI center of excellence might develop an automated solution for network alarm correlation, but if the operations team has no idea how it works they will be reluctant to let it loose on 'their' network in an autonomous, closed-loop mode.

As the figure below shows, there is often a trade-off between the accuracy of an AI technique and its explainability. For example, deep learning can handle millions of parameters and have high prediction accuracy, but low explainability. Conversely, simpler techniques such as regression algorithms and classification rules might be more explainable but have lower accuracy.

Figure 1: DARPA's Explainable AI, DARPA XAI Proposers Day, Aug. 2016.

How to make your AI explainable
Making sure your domain experts are involved in the AI modelling process is one way to engender explainability. According to the Institute for Ethical AI & ML, "it is possible to introduce explainability even in very complex models by introducing domain knowledge. Deep learning models are able to identify and abstract complex patterns that humans may not be able to see in data. However, there are many situations where introducing a-priori expert domain knowledge into the features, or abstracting key patterns identified in the deep learning models as actual features, it would be possible to break down the model into subsequent, more explainable pieces."

General techniques for generating explanations of AI systems include:

Two notable efforts to create XAI include DARPA-XAI and LIME: