Explainable AI — All you need to know. The what, how, why of explainable AI

AI blackbox
Lawtomated employee, Baloo, being classified by an AI image classifer.

What is explainable AI?

  1. Produce more explainable models, i.e. we understand how and why the system achieves its outcome given an input.
  2. Enable human users to understand, appropriately trust and effective manage AI systems.
DARPA Explainable AI Diagram
Source: here

Why is explainable AI helpful?

Why is explainable AI necessary? Helicopters vs Guns

Adversarial Attack Gun vs Helicopter
Source: here. © Wired

Assassination by pixel

Trolling Teslas

Rik & Morty

Rick & Morty Robot QR Code
© Adult Swim

Explainability depends on complexity

  • More complex AI systems are more accurate but less explainable
  • Less complex models are less accurate but more explainable
Complexity vs Explainability of AI

What does complexity mean?

Explainability of two AI models

The inequality of explainability

Machine learning techniques and their explainability
Source: here

Balance explainability vs complexity, guided by criticality

Impact vs Explainability for Explainable AI
Explainable AI Key Concerns
Criticality, Explainability and Complexity Mapping

How to evaluate a system’s necessary explainability?

Explainable AI Gap Analysis
  1. Work with the vendor (if a bought solution) or your own IT team (if a bought or built solution) to understand the extent to which the proposed / existing system allows users to explain how and why an output is produced (“System’s Explainability”).
  2. Perform a gap analysis of the System’s Explainability vs. Necessary Explainability (previous slide).
  3. If System’s Explainability < Necessary Explainability = negative gap. You should consider customising the solution (if possible) or buying / building something more explainable.
  4. If System’s Explainability > Necessary Explainability = positive gap. You could consider increasing the complexity of the model to improve its accuracy to the extent it does not reduce Explainability below the Necessary Explainability.
  5. In either scenario you will need to work with your IT team (to the extent system is built or bought) and potentially also a vendor (to the extent the system is bought) to understand to what extent it is possible to tweak an existing system, or design a new one, to patch the identified gaps. Likewise you may need to re-engineer the surrounding people and process to reduce or eliminate such gaps.
  6. The above should be carried out at the outset of any build / buy discussion and throughout the system’s implementation to ensure no gap exists at the outset nor emerges over time.
Explainable AI mapped to criticality of decision and required explainability
Source: here

But not everyone is agreed. Explainable AI may be unnecessary

Arguments for: testing = reliability = trustability

Arguments against: reliability is not trustability, and we need more than trust

Examples of where explainable AI would have helped

Washington Post - UK Exam Algorithm Scandal
Source: here
  1. Teachers were asked to supply for each pupil: (a) an estimated grade + (b) a ranking compared with every other pupil in their class.
  2. These were put into an algorithm that factored in the school’s performances in each subject over the previous 3 years.

The explainable AI no one is talking about

How to do explainable AI

Explainable AI Techniques
Source: here

The law of explainability

  1. The affected individual must have provided explicit consent, which has a high threshold under the GDPR.
  2. The automated decision making is necessary for the performance of the contract, e.g. a credit check against a maintained set of databases.
  3. Authorised by law.
  1. The fact of automated decision making;
  2. The significance of the automated decision making; and
  3. How the automated decision making operates, which has been described as a “right to explainability”.



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store