Production machine learning (ML) systems fail silently - not with crashes, but through wrong decisions. While observability is recognized as critical for ML operations, there is a lack empirical evidence of what practitioners actually capture. This study presents empirical results on ML observability in practice through seven focus group sessions in several domains. We catalog the information practitioners systematically capture across ML systems and their environment and map how they use it to validate models, detect and diagnose faults, and explain observed degradations. Finally, we identify gaps in current practice and outline implications for tooling design and research to establish ML observability practices.

Leest, J., Gerostathopoulos, I., Lago, P., Raibulet, C. (2026). Machine Learning Observability in Practice. In CAIN '26: Proceedings of the IEEE/ACM 5th International Conference on AI Engineering - Software Engineering for AI (pp.132-143). Association for Computing Machinery, Inc [10.1145/3793653.3793771].

Machine Learning Observability in Practice

Raibulet C.
2026

Abstract

Production machine learning (ML) systems fail silently - not with crashes, but through wrong decisions. While observability is recognized as critical for ML operations, there is a lack empirical evidence of what practitioners actually capture. This study presents empirical results on ML observability in practice through seven focus group sessions in several domains. We catalog the information practitioners systematically capture across ML systems and their environment and map how they use it to validate models, detect and diagnose faults, and explain observed degradations. Finally, we identify gaps in current practice and outline implications for tooling design and research to establish ML observability practices.
paper
machine learning; monitoring; observability;
English
5th IEEE International Conference on AI Engineering - Software Engineering for AI, CAIN 2026 - April 12 - 13, 2026
2026
CAIN '26: Proceedings of the IEEE/ACM 5th International Conference on AI Engineering - Software Engineering for AI
9798400724756
2026
132
143
open
Leest, J., Gerostathopoulos, I., Lago, P., Raibulet, C. (2026). Machine Learning Observability in Practice. In CAIN '26: Proceedings of the IEEE/ACM 5th International Conference on AI Engineering - Software Engineering for AI (pp.132-143). Association for Computing Machinery, Inc [10.1145/3793653.3793771].
File in questo prodotto:
File Dimensione Formato  
Leest et al-2026-IEEE/ACM-VoR.pdf

accesso aperto

Tipologia di allegato: Publisher’s Version (Version of Record, VoR)
Licenza: Creative Commons
Dimensione 652.25 kB
Formato Adobe PDF
652.25 kB Adobe PDF Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/10281/630683
Citazioni
  • Scopus 0
  • ???jsp.display-item.citation.isi??? ND
Social impact