Production machine learning (ML) systems fail silently - not with crashes, but through wrong decisions. While observability is recognized as critical for ML operations, there is a lack empirical evidence of what practitioners actually capture. This study presents empirical results on ML observability in practice through seven focus group sessions in several domains. We catalog the information practitioners systematically capture across ML systems and their environment and map how they use it to validate models, detect and diagnose faults, and explain observed degradations. Finally, we identify gaps in current practice and outline implications for tooling design and research to establish ML observability practices.
Leest, J., Gerostathopoulos, I., Lago, P., Raibulet, C. (2026). Machine Learning Observability in Practice. In CAIN '26: Proceedings of the IEEE/ACM 5th International Conference on AI Engineering - Software Engineering for AI (pp.132-143). Association for Computing Machinery, Inc [10.1145/3793653.3793771].
Machine Learning Observability in Practice
Raibulet C.
2026
Abstract
Production machine learning (ML) systems fail silently - not with crashes, but through wrong decisions. While observability is recognized as critical for ML operations, there is a lack empirical evidence of what practitioners actually capture. This study presents empirical results on ML observability in practice through seven focus group sessions in several domains. We catalog the information practitioners systematically capture across ML systems and their environment and map how they use it to validate models, detect and diagnose faults, and explain observed degradations. Finally, we identify gaps in current practice and outline implications for tooling design and research to establish ML observability practices.| File | Dimensione | Formato | |
|---|---|---|---|
|
Leest et al-2026-IEEE/ACM-VoR.pdf
accesso aperto
Tipologia di allegato:
Publisher’s Version (Version of Record, VoR)
Licenza:
Creative Commons
Dimensione
652.25 kB
Formato
Adobe PDF
|
652.25 kB | Adobe PDF | Visualizza/Apri |
I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.


