Explainable AI:

AI-powered systems have become so sophisticated that almost no human intervention is required for their design and deployment. However, there is a need to understand how decisions are made by AI methods especially when the decision derived affects human lives (as in medicine, law, or defense). Currently, there are three core notions of explainable AI. They are opaque systems, interpretable systems, and comprehensible systems.

Opaque Systems

These systems offer no insight into its algorithmic mechanisms. This means the mechanisms mapping inputs to outputs are invisible to the user. Essentially, it is viewed as an Oracle that makes predications without indicating how and why the predications are made. Opaque systems appear when closed-source Ai is licensed by an organization. The licensor doesn’t want to reveal the workings of its proprietary AI. Systems that reply on “black box” approaches (inspection of the algorithm or implementation doesn’t give insight into the system’s actual reasoning from inputs to corresponding outputs) are also classified as opaque systems.

Interpretable Systems

These systems allow users to mathematically analyze its algorithmic mechanisms. This means a user can see, study, and understand how inputs are mathematically mapped to outputs. This implies model transparency and requires a level of understanding the technical details of mapping. 

Comprehensible Systems

These systems emit symbols enabling user-driven explanations of how a conclusion is reached.  The symbols permit the user to relate properties of the inputs to their outputs. The user is responsible for compiling and comprehending the symbols and using his/her own knowledge and reasoning about them. The form of knowledge required by the user is often an implicit cognitive “intuition” about how the input, the symbols, and the output relate to each other.

As AI technologies are applied more and more in areas of life where decisions can have significant impacts and consequences, the pressure to develop AI methods whose results can be understood by different users will grow. Research in explainable AI us advancing.

 

For more Tidbits & Thoughts, please click here.