Hidden technical debt in Machine learning systems | Proceedings of the 29th International Conference on Neural Information Processing Systems - Volume 2

Source: original

• Home
• Conferences
• NIPS
• Proceedings
• NIPS'15
• Hidden technical debt in Machine learning systems

You are using the Basic Edition. Features requiring a subscription appear in grey.

Sign in to your subscription or learn more

Upgrade

Article

Share on

• *

Hidden technical debt in Machine learning systems

Authors: D. Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, + 6, Todd Phillips, Dietmar Ebner, + 4, Vinay Chaudhary, Michael Young, Jean-Francois Crespo, Dan Dennison (Less)Authors Info & Claims

NIPS'15: Proceedings of the 29th International Conference on Neural Information Processing Systems - Volume 2

Pages 2503 - 2511

Published: 07 December 2015 Publication History

__69 citation __10 Downloads

__

Add a Citation Alert

To add a citation alert, please log in to your account

__Contents

NIPS'15: Proceedings of the 29th International Conference on Neural Information Processing Systems - Volume 2

Hidden technical debt in Machine learning systems

Pages 2503 - 2511

PREVIOUS CHAPTERA Gaussian process model of quasar spectral energy distributions __Previous###### NEXT CHAPTERLocal causal discovery of direct causes and effectsNext __

• Abstract

• References

• __Information & Contributors

• __Bibliometrics & Citations
• __Reading Options
• __References
• __Figures
• __Tables
• __Media *

Abstract

Machine learning offers a fantastically powerful toolkit for building useful complex prediction systems quickly. This paper argues it is dangerous to think of these quick wins as coming for free. Using the software engineering framework of technical debt , we find it is common to incur massive ongoing maintenance costs in real-world ML systems. We explore several ML-specific risk factors to account for in system design. These include boundary erosion, entanglement, hidden feedback loops, undeclared consumers, data dependencies, configuration issues, changes in the external world, and a variety of system-level anti-patterns.

References

[1]

R. Ananthanarayanan, V. Basker, S. Das, A. Gupta, H. Jiang, T. Qiu, A. Reznichenko, D. Ryabkov, M. Singh, and S. Venkataraman. Photon: Fault-tolerant and scalable joining of continuous data streams. In SIGMOD '13: Proceedings of the 2013 international conference on Management of data , pages 577-588, New York, NY, USA, 2013.

Digital Library

Google Scholar

[2]

A. Anonymous. Machine learning: The high-interest credit card of technical debt. SE4ML: Software Engineering for Machine Learning (NIPS 2014 Workshop).

Google Scholar

[3]

L. Bottou, J. Peters, J. Quiñonero Candela, D. X. Charles, D. M. Chickering, E. Portugaly, D. Ray, P. Simard, and E. Snelson. Counterfactual reasoning and learning systems: The example of computational advertising. Journal of Machine Learning Research , 14(Nov), 2013.

Digital Library

Google Scholar

[4]

W. J. Brown, H. W. McCormick, T. J. Mowbray, and R. C. Malveau. Antipatterns: refactoring software, architectures, and projects in crisis. 1998.

Digital Library

Google Scholar

[5]

T. M. Chilimbi, Y. Suzue, J. Apacible, and K. Kalyanaraman. Project adam: Building an efficient and scalable deep learning training system. In 11th USENIX Symposium on Operating Systems Design and Implementation, OSDI '14, Broomfield, CO, USA, October 6-8, 2014. , pages 571-582, 2014.

Digital Library

Google Scholar

[6]

B. Dalessandro, D. Chen, T. Raeder, C. Perlich, M. Han Williams, and F. Provost. Scalable hands-free transfer learning for online advertising. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining , pages 1573-1582. ACM, 2014.

Digital Library

Google Scholar

[7]

M. Fowler. Code smells. http://http://martinfowler.com/bliki/CodeSmell.html.

Google Scholar

[8]

M. Fowler. Refactoring: improving the design of existing code. Pearson Education India, 1999.

Digital Library

Google Scholar

[9]

J. Langford and T. Zhang. The epoch-greedy algorithm for multi-armed bandits with side information. In Advances in neural information processing systems , pages 817-824, 2008.

Digital Library

Google Scholar

[10]

M. Li, D. G. Andersen, J. W. Park, A. J. Smola, A. Ahmed, V. Josifovski, J. Long, E. J. Shekita, and B. Su. Scaling distributed machine learning with the parameter server. In 11th USENIX Symposium on Operating Systems Design and Implementation, OSDI '14, Broomfield, CO, USA, October 6-8, 2014. , pages 583-598, 2014.

Digital Library

Google Scholar

[11]

J. Lin and D. Ryaboy. Scaling big data mining infrastructure: the twitter experience. ACM SIGKDD Explorations Newsletter , 14(2):6-19, 2013.

Digital Library

Google Scholar

[12]

H. B. McMahan, G. Holt, D. Sculley, M. Young, D. Ebner, J. Grady, L. Nie, T. Phillips, E. Davydov, D. Golovin, S. Chikkerur, D. Liu, M. Wattenberg, A. M. Hrafnkelsson, T. Boulos, and J. Kubica. Ad click prediction: a view from the trenches. In The 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2013, Chicago, IL, USA, August 11-14, 2013 , 2013.

Digital Library

Google Scholar

[13]

J. D. Morgenthaler, M. Gridnev, R. Sauciuc, and S. Bhansali. Searching for build debt: Experiences managing technical debt at google. In Proceedings of the Third International Workshop on Managing Technical Debt , 2012.

Digital Library

Google Scholar

[14]

D. Sculley, M. E. Otey, M. Pohl, B. Spitznagel, J. Hainsworth, and Y. Zhou. Detecting adversarial advertisements in the wild. In Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Diego, CA, USA, August 21-24, 2011 , 2011.

Digital Library

Google Scholar

[15]

Securities and E. Commission. SEC Charges Knight Capital With Violations of Market Access Rule , 2013.

Google Scholar

[16]

A. Spector, P. Norvig, and S. Petrov. Google's hybrid approach to research. Communications of the ACM , 55 Issue 7, 2012.

Digital Library

Google Scholar

[17]

A. Zheng. The challenges of building machine learning tools for the masses. SE4ML: Software Engineering for Machine Learning (NIPS 2014 Workshop).

Google Scholar

Cited By

View all __

• Arceo F Narsing V(2026)Securing the Agent: Vendor-Neutral, Multitenant Enterprise Retrieval and Tool UseProceedings of the ACM Conference on AI and Agentic Systems10.1145/3786335.3813145(862-872)Online publication date: 26-May-2026

https://dl.acm.org/doi/10.1145/3786335.3813145

• Sanchez TKalamara FStumpf SCaramiaux B(2026)Exploring people’s testing strategies in ML-based image classificationProceedings of the 1st International Conference on Human-Computer Interaction in the Alps10.1145/3780045.3780051(96-101)Online publication date: 1-Mar-2026

https://dl.acm.org/doi/10.1145/3780045.3780051

• Ranade DJaiswal R(2026)Right-to-be-Forgotten by Design in Adapter-Tuned TransformersProceedings of the 2026 Conference on Human Centred Artificial Intelligence - Education and Practice10.1145/3777490.3777507(93-99)Online publication date: 21-Jan-2026

https://dl.acm.org/doi/10.1145/3777490.3777507

• Show More Cited By

Index Terms

1. Hidden technical debt in Machine learning systems

2. Computing methodologies

3. Machine learning

4. Machine learning approaches

5. Social and professional topics

6. Professional topics

7. Management of computing and information systems

8. Implementation management

9. Pricing and resource allocation

10. Software management

11. Software and its engineering

12. Software creation and management

13. Software development process management

14. Software notations and tools

15. Software libraries and repositories

16. Software organization and properties

17. Software system structures

18. Software architectures

Index terms have been assigned to the content through auto-classification.

Recommendations

• Sell-Side Debt Analysts and Debt Market Efficiency

We explore sell-side debt analysts’ contributions to the efficiency of securities markets. We document that debt returns lag equity returns less when debt research coverage exists, which is consistent with debt analysts facilitating the process by which ...

Read More

• An Exploratory Study on Self-Admitted Technical Debt

ICSME '14: Proceedings of the 2014 IEEE International Conference on Software Maintenance and Evolution

Throughout a software development life cycle, developers knowingly commit code that is either incomplete, requires rework, produces errors, or is a temporary workaround. Such incomplete or temporary workarounds are commonly referred to as 'technical ...

Read More

• A portfolio approach to technical debt management

MTD '11: Proceedings of the 2nd Workshop on Managing Technical Debt

Technical debt describes the effect of immature software artifacts on software maintenance - the potential of extra effort required in future as if paying interest for the incurred debt. The uncertainty of interest payment further complicates the ...

Read More

Comments

__Information & Contributors

Information

Published In

NIPS'15: Proceedings of the 29th International Conference on Neural Information Processing Systems - Volume 2

December 2015

3626 pages

• Editors:
• C. Cortes,
• D. D. Lee,
• M. Sugiyama,
• R. Garnett

Publisher

MIT Press

Cambridge, MA, United States

Publication History

Published : 07 December 2015

Qualifiers

• Article

Contributors

Other Metrics

View Article Metrics

__Bibliometrics & Citations

Bibliometrics

Article Metrics

• 69

Total Citations

View Citations * 10

Total Downloads

• Downloads (Last 12 months)...
• Downloads (Last 6 weeks) ...

Reflects downloads up to 07 Jun 2026

Other Metrics

View Author Metrics

Citations

Cited By

View all __

• Arceo F Narsing V(2026)Securing the Agent: Vendor-Neutral, Multitenant Enterprise Retrieval and Tool UseProceedings of the ACM Conference on AI and Agentic Systems10.1145/3786335.3813145(862-872)Online publication date: 26-May-2026

https://dl.acm.org/doi/10.1145/3786335.3813145

• Sanchez TKalamara FStumpf SCaramiaux B(2026)Exploring people’s testing strategies in ML-based image classificationProceedings of the 1st International Conference on Human-Computer Interaction in the Alps10.1145/3780045.3780051(96-101)Online publication date: 1-Mar-2026

https://dl.acm.org/doi/10.1145/3780045.3780051

• Ranade DJaiswal R(2026)Right-to-be-Forgotten by Design in Adapter-Tuned TransformersProceedings of the 2026 Conference on Human Centred Artificial Intelligence - Education and Practice10.1145/3777490.3777507(93-99)Online publication date: 21-Jan-2026

https://dl.acm.org/doi/10.1145/3777490.3777507

• Olari VRomeike R(2025)Teaching Data Concepts and Practices in Secondary School Education on Artificial Intelligence: Approaches, Mechanisms, and Emerging Local TheoriesProceedings of the 25th Koli Calling International Conference on Computing Education Research10.1145/3769994.3770021(1-12)Online publication date: 11-Nov-2025

https://dl.acm.org/doi/10.1145/3769994.3770021

• Hassan ZTreude CNorrish MWilliams GPotanin Ad'Amorim M(2024)Reproducibility Debt: Challenges and Future PathwaysCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering10.1145/3663529.3663778(462-466)Online publication date: 10-Jul-2024

https://dl.acm.org/doi/10.1145/3663529.3663778

• Terzis PVeale MGaumann N(2024)Law and the Emerging Political Economy of Algorithmic AuditsProceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency10.1145/3630106.3658970(1255-1267)Online publication date: 3-Jun-2024

https://dl.acm.org/doi/10.1145/3630106.3658970

• Menshawy ANawaz ZFahmy M(2024)Navigating Challenges and Technical Debt in Large Language Models DeploymentProceedings of the 4th Workshop on Machine Learning and Systems10.1145/3642970.3655840(192-199)Online publication date: 22-Apr-2024

https://dl.acm.org/doi/10.1145/3642970.3655840

• Wang FAdebayo JTan SGarcia-Olano DKokhlikyan NOh ANaumann TGloberson ASaenko KHardt MLevine S(2023)Error discovery by clustering influence embeddingsProceedings of the 37th International Conference on Neural Information Processing Systems10.5555/3666122.3667931(41765-41777)Online publication date: 10-Dec-2023

https://dl.acm.org/doi/10.5555/3666122.3667931

• Min CMathur AAcer UMontanari AKawsar F(2023)SensiX++: Bringing MLOps and Multi-tenant Model Serving to Sensory Edge DevicesACM Transactions on Embedded Computing Systems10.1145/361750722 :6(1-27)Online publication date: 9-Nov-2023

https://dl.acm.org/doi/10.1145/3617507

• Sakuma KMatsuno RKameda YFrommholz IHopfgartner FLee MOakes MLalmas MZhang MSantos R(2023)Quantitative Decomposition of Prediction Errors Revealing Multi-Cause Impacts: An Insightful Framework for MLOpsProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3615238(4259-4263)Online publication date: 21-Oct-2023

https://dl.acm.org/doi/10.1145/3583780.3615238

• Show More Cited By

__Share

Share

Share this Publication link

__Copy Link

__Copied!

__Copying failed.

Share on social media

__X __LinkedIn __Reddit __Facebook __email