Scientific Exploration and Explainable Artificial Intelligence

Beisbart C.

Computer simulations are often claimed to be opaque and thus to lack transparency. But what exactly is the opacity of simulations? This paper aims to answer that question by proposing an explication of opacity. Such an explication is needed, I argue, because the pioneering definition of opacity by P. Humphreys and a recent elaboration by Durán and Formanek are too narrow. While it is true that simulations are opaque in that they include too many computations and thus cannot be checked by hand, this doesn’t exhaust what we might want to call the opacity of simulations. I thus make a fresh start with the natural idea that the opacity of a method is its disposition to resist knowledge and understanding. I draw on recent work on understanding and elaborate the idea by a systematic investigation into what type of knowledge and what type of understanding are required if opacity is to be avoided and why the required sort of understanding, in particular, is difficult to achieve. My proposal is that a method is opaque to the degree that it’s difficult for humans to know and to understand why its outcomes arise. This proposal allows for a comparison between different methods regarding opacity. It further refers to a kind of epistemic access that is important in scientific work with simulations.

Erasmus A., Brunet T.D., Fisher E.

We argue that artificial networks are explainable and offer a novel theory of interpretability. Two sets of conceptual questions are prominent in theoretical engagements with artificial neural networks, especially in the context of medical artificial intelligence: (1) Are networks explainable, and if so, what does it mean to explain the output of a network? And (2) what does it mean for a network to be interpretable? We argue that accounts of “explanation” tailored specifically to neural networks have ineffectively reinvented the wheel. In response to (1), we show how four familiar accounts of explanation apply to neural networks as they would to any scientific phenomenon. We diagnose the confusion about explaining neural networks within the machine learning literature as an equivocation on “explainability,” “understandability” and “interpretability.” To remedy this, we distinguish between these notions, and answer (2) by offering a theory and typology of interpretation in machine learning. Interpretation is something one does to an explanation with the aim of producing another, more understandable, explanation. As with explanation, there are various concepts and methods involved in interpretation: Total or Partial, Global or Local, and Approximative or Isomorphic. Our account of “interpretability” is consistent with uses in the machine learning literature, in keeping with the philosophy of explanation and understanding, and pays special attention to medical artificial intelligence systems.

Confalonieri R., Coba L., Wagner B., Besold T.R.

Boon M.

Zednik C.

Many of the computing systems programmed using Machine Learning are opaque: it is difficult to know why they do what they do or how they work. Explainable Artificial Intelligence aims to develop analytic techniques that render opaque computing systems transparent, but lacks a normative framework with which to evaluate these techniques’ explanatory successes. The aim of the present discussion is to develop such a framework, paying particular attention to different stakeholders’ distinct explanatory requirements. Building on an analysis of “opacity” from philosophy of science, this framework is modeled after accounts of explanation in cognitive science. The framework distinguishes between the explanation-seeking questions that are likely to be asked by different stakeholders, and specifies the general ways in which these questions should be answered so as to allow these stakeholders to perform their roles in the Machine Learning ecosystem. By applying the normative framework to recently developed techniques such as input heatmapping, feature-detector visualization, and diagnostic classification, it is possible to determine whether and to what extent techniques from Explainable Artificial Intelligence can be used to render opaque computing systems transparent and, thus, whether they can be used to solve the Black Box Problem.

Massimi M.

I analyze the exploratory function of two main modeling practices: targetless fictional models and hypothetical perspectival models. In both cases, I argue, modelers invite us to imagine or conceive something about the target system, which is known to be either nonexistent (fictional models) or just hypothetical (in perspectival models). I clarify the kind of imagining or conceiving involved in each modeling practice, and I show how each—in its own right—delivers important modal knowledge. I illustrate these two kinds of exploratory models with Maxwell’s ether model and supersymmetric particle models at the Large Hadron Collider.

Schmidt J., Marques M.R., Botti S., Marques M.A.

One of the most exciting tools that have entered the material science toolbox in recent years is machine learning. This collection of statistical methods has already proved to be capable of considerably speeding up both fundamental and applied research. At present, we are witnessing an explosion of works that develop and apply machine learning to solid-state systems. We provide a comprehensive overview and analysis of the most recent research in this topic. As a starting point, we introduce machine learning principles, algorithms, descriptors, and databases in materials science. We continue with the description of different machine learning approaches for the discovery of stable materials and the prediction of their crystal structure. Then we discuss research in numerous quantitative structure–property relationships and various approaches for the replacement of first-principle methods by machine learning. We review how active learning and surrogate-based optimization can be applied to improve the rational design process and related examples of applications. Two major questions are always the interpretability of and the physical understanding gained from machine learning models. We consider therefore the different facets of interpretability and their importance in materials science. Finally, we propose solutions and future research paths for various challenges in computational materials science.

Sullivan E.

Hohman F., Kahng M., Pienta R., Chau D.H.

Deep learning has recently seen rapid development and received significant attention due to its state-of-the-art performance on previously-thought hard problems. However, because of the internal complexity and nonlinear structure of deep neural networks, the underlying decision making processes for why these models are achieving such performance are challenging and sometimes mystifying to interpret. As deep learning spreads across domains, it is of paramount importance that we equip users of deep learning with tools for understanding when a model works correctly, when it fails, and ultimately how to improve its performance. Standardized toolkits for building neural networks have helped democratize deep learning; visual analytics systems have now been developed to support model explanation, interpretation, debugging, and improvement. We present a survey of the role of visual analytics in deep learning research, which highlights its short yet impactful history and thoroughly summarizes the state-of-the-art using a human-centered interrogative framework, focusing on the Five W's and How (Why, Who, What, How, When, and Where). We conclude by highlighting research directions and open research problems. This survey helps researchers and practitioners in both visual analytics and deep learning to quickly learn key aspects of this young and rapidly growing body of research, whose impact spans a diverse range of domains.

Rudin C.

Black box machine learning models are currently being used for high-stakes decision making throughout society, causing problems in healthcare, criminal justice and other domains. Some people hope that creating methods for explaining these black box models will alleviate some of the problems, but trying to explain black box models, rather than creating models that are interpretable in the first place, is likely to perpetuate bad practice and can potentially cause great harm to society. The way forward is to design models that are inherently interpretable. This Perspective clarifies the chasm between explaining black boxes and using inherently interpretable models, outlines several key reasons why explainable black boxes should be avoided in high-stakes decisions, identifies challenges to interpretable machine learning, and provides several example applications where interpretable models could potentially replace black box models in criminal justice, healthcare and computer vision. There has been a recent rise of interest in developing methods for ‘explainable AI’, where models are created to explain how a first ‘black box’ machine learning model arrives at a specific decision. It can be argued that instead efforts should be directed at building inherently interpretable models in the first place, in particular where they are applied in applications that directly affect human lives, such as in healthcare and criminal justice.

Dattilo A., Vanderburg A., Shallue C.J., Mayo A.W., Berlind P., Bieryla A., Calkins M.L., Esquerdo G.A., Everett M.E., Howell S.B., Latham D.W., Scott N.J., Yu L.

For years, scientists have used data from NASA's Kepler Space Telescope to look for and discover thousands of transiting exoplanets. In its extended K2 mission, Kepler observed stars in various regions of sky all across the ecliptic plane, and therefore in different galactic environments. Astronomers want to learn how the population of exoplanets are different in these different environments. However, this requires an automatic and unbiased way to identify the exoplanets in these regions and rule out false positive signals that mimic transiting planet signals. We present a method for classifying these exoplanet signals using deep learning, a class of machine learning algorithms that have become popular in fields ranging from medical science to linguistics. We modified a neural network previously used to identify exoplanets in the Kepler field to be able to identify exoplanets in different K2 campaigns, which range in galactic environments. We train a convolutional neural network, called AstroNet-K2, to predict whether a given possible exoplanet signal is really caused by an exoplanet or a false positive. AstroNet-K2 is highly successful at classifying exoplanets and false positives, with accuracy of 98% on our test set. It is especially efficient at identifying and culling false positives, but for now, still needs human supervision to create a complete and reliable planet candidate sample. We use AstroNet-K2 to identify and validate two previously unknown exoplanets. Our method is a step towards automatically identifying new exoplanets in K2 data and learning how exoplanet populations depend on their galactic birthplace.

Cichy R.M., Kaiser D.

Artificial deep neural networks (DNNs) initially inspired by the brain enable computers to solve cognitive tasks at which humans excel. In the absence of explanations for such cognitive phenomena, in turn cognitive scientists have started using DNNs as models to investigate biological cognition and its neural basis, creating heated debate. Here, we reflect on the case from the perspective of philosophy of science. After putting DNNs as scientific models into context, we discuss how DNNs can fruitfully contribute to cognitive science. We claim that beyond their power to provide predictions and explanations of cognitive phenomena, DNNs have the potential to contribute to an often overlooked but ubiquitous and fundamental use of scientific models: exploration.

Lapuschkin S., Wäldchen S., Binder A., Montavon G., Samek W., Müller K.

Current learning machines have successfully solved hard application problems, reaching high accuracy and displaying seemingly intelligent behavior. Here we apply recent techniques for explaining decisions of state-of-the-art learning machines and analyze various tasks from computer vision and arcade games. This showcases a spectrum of problem-solving behaviors ranging from naive and short-sighted, to well-informed and strategic. We observe that standard performance evaluation metrics can be oblivious to distinguishing these diverse problem solving behaviors. Furthermore, we propose our semi-automated Spectral Relevance Analysis that provides a practically effective way of characterizing and validating the behavior of nonlinear learning machines. This helps to assess whether a learned model indeed delivers reliably for the problem that it was conceived for. Furthermore, our work intends to add a voice of caution to the ongoing excitement about machine intelligence and pledges to evaluate and judge some of these recent successes in a more nuanced manner. Nonlinear machine learning methods have good predictive ability but the lack of transparency of the algorithms can limit their use. Here the authors investigate how these methods approach learning in order to assess the dependability of their decision making.

Durán J.M., Formanek N.

Several philosophical issues in connection with computer simulations rely on the assumption that results of simulations are trustworthy. Examples of these include the debate on the experimental role of computer simulations (Parker in Synthese 169(3):483–496, 2009; Morrison in Philos Stud 143(1):33–57, 2009), the nature of computer data (Barberousse and Vorms, in: Durán, Arnold (eds) Computer simulations and the changing face of scientific experimentation, Cambridge Scholars Publishing, Barcelona, 2013; Humphreys, in: Durán, Arnold (eds) Computer simulations and the changing face of scientific experimentation, Cambridge Scholars Publishing, Barcelona, 2013), and the explanatory power of computer simulations (Krohs in Int Stud Philos Sci 22(3):277–292, 2008; Durán in Int Stud Philos Sci 31(1):27–45, 2017). The aim of this article is to show that these authors are right in assuming that results of computer simulations are to be trusted when computer simulations are reliable processes. After a short reconstruction of the problem of epistemic opacity, the article elaborates extensively on computational reliabilism, a specified form of process reliabilism with computer simulations located at the center. The article ends with a discussion of four sources for computational reliabilism, namely, verification and validation, robustness analysis for computer simulations, a history of (un)successful implementations, and the role of expert knowledge in simulations.

Zhavoronkov A.

Rosenberger J., Tlatlik J., Rump N., Münstermann S.

Bjerring J.C., Mainz J., Munch L.

Abstract It has often been argued that we face a trade-off between accuracy and opacity in deep learning models. The idea is that we can only harness the accuracy of deep learning models by simultaneously accepting that the grounds for the models’ decision-making are epistemically opaque to us. In this paper, we ask the following question: what are the prospects of making deep learning models transparent without compromising on their accuracy? We argue that the answer to this question depends on which kind of opacity we have in mind. If we focus on the standard notion of opacity, which tracks the internal complexities of deep learning models, we argue that existing explainable AI (XAI) techniques show us that the prospects look relatively good. But, as it has recently been argued in the literature, there is another notion of opacity that concerns factors external to the model. We argue that there are at least two types of external opacity—link opacity and structure opacity—and that existing XAI techniques can to some extent help us reduce the former but not the latter.

Esders M., Schnake T., Lederer J., Kabylda A., Montavon G., Tkatchenko A., Müller K.

Watson D.S., Mökander J., Floridi L.

AbstractSeveral competing narratives drive the contemporary AI ethics discourse. At the two extremes are sociotechnical dogmatism, which holds that society is full of inefficiencies and imperfections that can only be solved by better technology; and sociotechnical skepticism, which highlights the unacceptable risks AI systems pose. While both narratives have their merits, they are ultimately reductive and limiting. As a constructive synthesis, we introduce and defend sociotechnical pragmatism—a narrative that emphasizes the central role of context and human agency in designing and evaluating emerging technologies. In doing so, we offer two novel contributions. First, we demonstrate how ethical and epistemological considerations are intertwined in the AI ethics discourse by tracing the dialectical interplay between dogmatic and skeptical narratives across disciplines. Second, we show through examples how sociotechnical pragmatism does more to promote fair and transparent AI than dogmatic or skeptical alternatives. By spelling out the assumptions that underpin sociotechnical pragmatism, we articulate a robust stance for policymakers and scholars who seek to enable societies to reap the benefits of AI while managing the associated risks through feasible, effective, and proportionate governance.

Aljawawdeh H., Fawareh H., Al Refai M., Alshraideh F., Salman R., Dabaa’t S.A., Alazzeh H., Al-Shdaifat H., Khouj M.

Semnani P., Bogojeski M., Bley F., Zhang Z., Wu Q., Kneib T., Herrmann J., Weisser C., Patcas F., Müller K.

Saoud L.S., Sultan A., Elmezain M., Heshmat M., Seneviratne L., Hussain I.

Dudek N.K., Chakhvadze M., Kobakhidze S., Kantidze O., Yuriy G.

Mujawar S., Sayed M., Chaware A.

Páez A.

AbstractIn the natural and social sciences, it is common to use toy models—extremely simple and highly idealized representations—to understand complex phenomena. Some of the simple surrogate models used to understand opaque machine learning (ML) models, such as rule lists and sparse decision trees, bear some resemblance to scientific toy models. They allow non-experts to understand how an opaque ML model works globally via a much simpler model that highlights the most relevant features of the input space and their effect on the output. The obvious difference is that the common target of a toy and a full-scale model in the sciences is some phenomenon in the world, while the target of a surrogate model is another model. This essential difference makes toy surrogate models (TSMs) a new object of study for theories of understanding, one that is not easily accommodated under current analyses. This paper provides an account of what it means to understand an opaque ML model globally with the aid of such simple models.

Eberle O., Büttner J., el-Hajj H., Montavon G., Müller K., Valleriani M.

Understanding the evolution and dissemination of human knowledge over time faces challenges due to the abundance of historical materials and limited specialist resources. However, the digitization of historical archives presents an opportunity for AI-supported analysis. This study advances historical analysis by using an atomization-recomposition method that relies on unsupervised machine learning and explainable AI techniques. Focusing on the “Sacrobosco Collection,” consisting of 359 early modern printed editions of astronomy textbooks from European universities (1472–1650), totaling 76,000 pages, our analysis uncovers temporal and geographic patterns in knowledge transformation. We highlight the relevant role of astronomy textbooks in shaping a unified mathematical culture, driven by competition among educational institutions and market dynamics. This approach deepens our understanding by grounding insights in historical context, integrating with traditional methodologies. Case studies illustrate how communities embraced scientific advancements, reshaping astronomic and geographical views and exploring scientific roots amidst a changing world.

Shetty S., Schneider P., Stebel K., David Hamer P., Kylling A., Koren Berntsen T.

Satellite observations from instruments such as the TROPOspheric Monitoring Instrument (TROPOMI) show significant potential for monitoring the spatiotemporal variability of NO2, however they typically provide vertically integrated measurements over the tropospheric column. In this study, we introduce a machine learning approach entitled 'S-MESH' (Satellite and ML-based Estimation of Surface air quality at High resolution) that allows for estimating daily surface NO2 concentrations over Europe at 1 km spatial resolution based on eXtreme gradient boost (XGBoost) model using primarily observation-based datasets over the period 2019–2021. Spatiotemporal datasets used by the model include TROPOMI NO2 tropospheric vertical column density, night light radiance from the Visible Infrared Imaging Radiometer Suite (VIIRS), Normalized Difference Vegetation Index from the Moderate Resolution Imaging Spectroradiometer (MODIS), observations of air quality monitoring stations from the European Environment Agency database and modeled meteorological parameters such as planetary boundary layer height, wind velocity, temperature. The overall model evaluation shows a mean absolute error of 7.77 μg/m3, a median bias of 0.6 μg/m3 and a Spearman rank correlation of 0.66. The model performance is found to be influenced by NO2 concentration levels, with the most reliable predictions at concentration levels of 10–40 μg/m3 with a bias of

Shafik W.

The rapid integration of artificial intelligence (AI) into seas, ocean, and marine resource health has opened new opportunities for smart ocean health automation, transforming marine diagnosis, treatment, and the overall ocean ecosystem, thus contributing to Sustainable Development Goal 14 of ensuring sustainable and conserved use of seas, ocean, and other marine resources. However, the widespread adoption of AI algorithms in several domains, like human and ocean healthcare, comes with challenges, particularly regarding the transparency and explainability of these techniques. This study explores the explainable AI (XAI) concept and its crucial role in ocean healthcare automation. Discuss the significance of XAI, various techniques for achieving explainability, and their potential applications in ocean health. XAI can enhance accountability and facilitate better decision-making by enabling ocean, sea, and marine health professionals and the general population to understand and trust AI-driven ocean decisions. Moreover, the ethical considerations and challenges associated with implementing XAI in ocean healthcare settings, including privacy, bias, and regulatory implications, are addressed. Highlighted future directions in XAI research for smart ocean health and emphasized the implications for ocean health providers and policymakers. By embracing XAI, the ocean health industry can unlock the full potential of AI while ensuring transparency, fairness, and improved patient outcomes. It is revealed that ocean healthcare is most comparable to human healthcare. Therefore, it is paramount to have a collective responsibility to ensure the proper utilization and health of the ocean, marine resources, and seas.

Genin K., Grote T., Wolfers T.

AbstractAs a discipline, psychiatry is in the process of finding the right set of concepts to organize research and guide treatment. Dissatisfaction with the status quo as expressed in standard manuals has animated a number of computational paradigms, each proposing to rectify the received concept of mental disorder. We explore how different computational paradigms: normative modeling, network theory and learning-theoretic approaches like reinforcement learning and active inference, reconceptualize mental disorders. Although each paradigm borrows heavily from machine learning, they differ significantly in their methodology, their preferred level of description, the role they assign to the environment and, especially, the degree to which they aim to assimilate psychiatric disorders to a standard medical disease model. By imagining how these paradigms might evolve, we bring into focus three rather different visions for the future of psychiatric research. Although machine learning plays a crucial role in the articulation of these paradigms, it is clear that we are far from automating the process of conceptual revision. The leading role continues to be played by the theoretical, metaphysical and methodological commitments of the competing paradigms.

van Oosterzee A.

AbstractMachine learning (ML) has emerged as a promising tool in psychiatry, revolutionising diagnostic processes and patient outcomes. In this paper, I argue that while ML studies show promising initial results, their application in mimicking clinician-based judgements presents inherent limitations (Shatte et al. in Psychol Med 49:1426–1448. https://doi.org/10.1017/S0033291719000151, 2019). Most models still rely on DSM (the Diagnostic and Statistical Manual of Mental Disorders) categories, known for their heterogeneity and low predictive value. DSM's descriptive nature limits the validity of psychiatric diagnoses, which leads to overdiagnosis, comorbidity, and low remission rates. The application in psychiatry highlights the limitations of supervised ML techniques. Supervised ML models inherit the validity issues of their training data set. When the model's outcome is a DSM classification, this can never be more valid or predictive than the clinician’s judgement. Therefore, I argue that these models have little added value to the patient. Moreover, the lack of known underlying causal pathways in psychiatric disorders prevents validating ML models based on such classifications. As such, I argue that high accuracy in these models is misleading when it is understood as validating the classification. In conclusion, these models will not will not offer any real benefit to patient outcomes. I propose a shift in focus, advocating for ML models to prioritise improving the predictability of prognosis, treatment selection, and prevention. Therefore, data selection and outcome variables should be geared towards this transdiagnostic goal. This way, ML can be leveraged to better support clinicians in personalised treatment strategies for mental health patients.