Interoperability in Deep Learning: A User Survey and Failure Analysis of ONNX Model Converters

Authors

Purvish Jajal, Purdue UniversityFollow
Wenxin Jiang, Purdue UniversityFollow
Arav Tewari, Purdue UniversityFollow
Erik Kocinare, Purdue UniversityFollow
Joseph Woo, Purdue UniversityFollow
Anusha Sarraf, Purdue UniversityFollow
Yung-Hsiang Lu, Purdue UniversityFollow
George Thiruvathukal, Loyola University ChicagoFollow
James C. Davis, Purdue UniversityFollow

Document Type

Conference Proceeding

Publication Date

9-11-2024

Publication Title

ISSTA 2024: Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis

Pages

1466 - 1478

Publisher Name

ACM

Abstract

Software engineers develop, fine-tune, and deploy deep learning (DL) models using a variety of development frameworks and runtime environments. DL model converters move models between frameworks and to runtime environments. Conversion errors compromise model quality and disrupt deployment. However, the failure characteristics of DL model converters are unknown, adding risk when using DL interoperability technologies. This paper analyzes failures in DL model converters. We survey software engineers about DL interoperability tools, use cases, and pain points (N=92). Then, we characterize failures in model converters associated with the main interoperability tool, ONNX (N=200 issues in PyTorch and TensorFlow). Finally, we formulate and test two hypotheses about structural causes for the failures we studied. We find that the node conversion stage of a model converter accounts for ∼75% of the defects and 33% of reported failure are related to semantically incorrect models. The cause of semantically incorrect models is elusive, but models with behaviour inconsistencies share operator sequences. Our results motivate future research on making DL interoperability software simpler to maintain, extend, and validate. Research into behavioural tolerances and architectural coverage metrics would be fruitful.

Comments

Author Posting © The Author(s), 2024. This conference proceeding is posted here by permission of ACM for personal use and redistribution. This proceeding was published open access in ISSTA 2024: Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, September 11, 2024 at https://doi.org/10.1145/3650212.3680374

Recommended Citation

Jajal, P., Jiang, W., Tewari, A., Woo, J., Lu, Y., Thiruvathukal, G.K., & Davis, J.C. (2023). Analysis of Failures and Risks in Deep Learning Model Converters: A Case Study in the ONNX Ecosystem. ArXiv, abs/2303.17708.

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

Copyright Statement

© The Author(s), 2024