An Experience Report on Machine Learning Reproducibility: Guidance for Practitioners and TensorFlow Model Garden Contributors

Authors

Vishnu Banna, Purdue University
Akhil Chinnakotla, Purdue University
Zhengxin Yan, Purdue University
Anirudh Vegesana, Purdue University
Naveen Vivek, Purdue University
Kruthi Krishnappa, Purdue University
Wenxin Jiang, Purdue University
Yung-Hisang Lu, Purdue University
George ThiruvathukalFollow
James C. Davis, Purdue UniversityFollow

Document Type

Technical Report

Publication Date

7-5-2021

Publication Title

ArXiv

Abstract

Machine learning techniques are becoming a fundamental tool for scientific and engineering progress. These techniques are applied in contexts as diverse as astronomy and spam filtering. However, correctly applying these techniques requires careful engineering. Much attention has been paid to the technical potential; relatively little attention has been paid to the software engineering process required to bring research-based machine learning techniques into practical utility. Technology companies have supported the engineering community through machine learning frameworks such as TensorFLow and PyTorch, but the details of how to engineer complex machine learning models in these frameworks have remained hidden.

To promote best practices within the engineering community, academic institutions and Google have partnered to launch a Special Interest Group on Machine Learning Models (SIGMODELS) whose goal is to develop exemplary implementations of prominent machine learning models in community locations such as the TensorFlow Model Garden (TFMG). The purpose of this report is to define a process for reproducing a state-of-the-art machine learning model at a level of quality suitable for inclusion in the TFMG. We define the engineering process and elaborate on each step, from paper analysis to model release. We report on our experiences implementing the YOLO model family with a team of 26 student researchers, share the tools we developed, and describe the lessons we learned along the way.

Identifier

arXiv:2107.00821

Comments

See https://github.com/tensorflow/models for additional information about the TensorFlow Models Garden Effort.

Recommended Citation

Vishnu Banna, Akhil Chinnakotla, Zhengxin Yan, Ani Vegesana and Naveen Vivek, Kruthi Krishnappa, Wenxin Jiang, Yung-Hsiang Lu, George K. Thiruvathukal, and James C. Davis, "An Experience Report on Machine Learning Reproducibility: Guidance for Practitioners and TensorFlow Model Garden Contributors", arXiv:2107.00821

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.