Answering the Call to Go Beyond Accuracy: An Online Tool for the Multidimensional Assessment of Decision Support Systems

The research about, and use of, AI-based Decision Support Systems (DSS) has been steadily increasing in the recent years: however, tools and techniques to validate and evaluate these systems in an holistic manner are still largely lacking, especially in regard to their potential impact on actual human decision-making. This paper challenges the accuracy-centric paradigm in DSS evaluation by introducing the nuanced, multi-dimensional approach of the DSS Quality Assessment Tool. Developed at MUDI Lab (University of Milano-Bicocca), this free, open-source tool supports the quality assessment of AI-based decision support systems (DSS) along six different and complementary dimensions: model robustness, data similarity, calibration, utility, data reliability and impact on human decision making. Each dimension is analyzed for its relevance in the Medical AI domain, the metrics employed, and their visualizations, designed according to the principle of vague visualizations to promote cognitive engagement. Such a tool can be instrumental to foster a culture of continuous oversight, outcome monitoring, and reflective technology assessment.