Necessity of Explainable Models in Digital Health

Quantitative health researchers have traditionally used simple parametric models such as general(ized) linear models to conduct statistical inference. However, the recent advent of increasing computing power, modern data collection techniques, and improved data storage options has encouraged a current rise in applications of machine learning technologies in health research. Machine learning models handle feature interactions and non-linear effects automatically. Models such as support vector machines, gradient boosting, random forests, or neural networks often outperform simpler and more explainable models on many prediction tasks. Machine learning has advantages such as the ability to extract structured knowledge from extensive data making few formal assumptions. In other words, such algorithmic methods aim to uncover general principles underlying a series of observations without explicit instructions and let the data to “speak for themselves”. Machine-learning community has widely adopted and cultivated practices that can be used to develop predictive models that can extrapolate from one set of data to another set of data by making useful predictions for new cases. In the context of health research and as a crucial avenue to hold up to the promises of precision medicine, there is a necessity for implementation of statistical learning procedures that explicitly model and test extrapolation to individuals who have not yet been seen.

However, there is an increasing demand for transparency from the various stakeholders in health data science. Machine learning models are being used to make important decisions in critical contexts. Hereby there is a major risk of development and implementation of decision-making tools that do not allow obtaining detailed explanations of their criteria are not justifiable nor explainable. In precision medicine, where health professionals require far more information from the model than a simple class allocation to support their decision (e.g. diagnosis), it is crucial to implement interpretable machine learning algorithms to support the output of the model. Healthcare system leverages machine learning for high-stakes decision making applications that deeply impact society members. In this context, health authorities should be cautious of black box machine learning models that lack transparency and accountability to explain their predictions in a way that humans can understand. Currently, the European Union’s General Data Protection Regulation underline that ‘The data subject shall have the right not to be subject to a decision based solely on automated processing, including profiling, which produces legal effects concerning him or her or similarly significantly affects him or her’ (Article 22 of GDPR regulations) referring to ‘right to an explanation’. Here the term ‘explanation’ refers to an understanding of how a model works and is equal to interpretability, comprehensibility, and transparency. To highlight the advantages of interpretable machine learning models the following points should be considered: 1) interpretable machine learning models ensure impartiality in decision-making by facilitating detection and consequently correction of biases, 2) by highlighting potential adversarial perturbations that could substantially change the prediction interpretability facilitates the provision of model robustness, 3) interpretability can approximate if an underlying causality exists in the model and act as a potential warranty that only meaningful variables infer the output.

The current generation of explainable AI systems, (XAI) aims towards techniques that focuses on the production of more interpretable models while maintaining a high level of learning performance. This enables experts to understand and hence trust/manage the emerging generation of artificially intelligent partners. Although linear and parametric models present numeric values concerning the contribution of each feature in the prediction process, this is not the case for non-linear machine learning models. More complex non-linear models for the most part do not have intelligible parameters, which makes it more to extract such intelligible knowledge from these models. Along these lines and with the intent of extracting some information from the prediction procedure, experts designed model-agnostic techniques for post-hoc interpretability with the possibility to be plugged to any model. Such methods necessarily try to simplify the relationships between certain features and the target and to achieve something tractable and of reduced complexity by strategies such as marginalizing the performance over other features, systematically manipulating features, or even visualizing the features to ease the interpretation of the model’s behavior. Since the model-agnostic interpretability separates interpretation from the model, It has more flexibility compared to model-specific approaches. Prominent techniques for global feature effects include the permutation feature importance (PFI) which uses the difference between some baseline performance measure and the same performance measure obtained after permuting the values of a particular feature in the training data, partial dependence plot (PDP) which shows if the marginal effect of a given feature on the target is linear, monotonic or more complex, as well as individual conditional expectation (ICE) that focuses on how the instance’s prediction changes when a feature changes. Some experts and policy makers in the domain of health are still wary about the use of machine learning because of the legal reasons, risk aversion, or lack of insight into the underlying task due to complexity and ambiguity of the prediction process. The implementation of interpretability pipelines will address these issues and make machine learning attractive to health organisations and policy makers who demand some transparency. If such efforts do not succeed, it is possible that black box models will continue to be permitted in the context of health research and intervention where their implementation is not safe. Likewise, sub-optimal interpretability pipeline might lead to legal conflict with regulations such as ‘right to explanation’. The current context of development in legal and technological frameworks highlights the necessity of shifting the focus from the basic assumption of high-performance to explainable AI in heath data science.