We propose a benchmarking strategy that is robust in the presence of timer error, OS jitter and other environmental fluctuations, and is insensitive to the highly nonideal statistics produced by timing measurements. We construct a model that explains how these strongly nonideal statistics can arise from environmental fluctuations, and also justifies our proposed strategy. We implement this strategy in the BenchmarkTools Julia package, where it is used in production continuous integration (CI) pipelines for developing the Julia language and its ecosystem.

Jiahao Chen works at Capital One in Emerging Technologies. He was a Research Scientist at the Massachusetts Institute of Technology in the Computer Science and Artificial Intelligence Laboratory (CSAIL), where he works toward solving data science problems using the Julia programming language. He is interested in end-to-end big data applications tying together parallel data ingest, analytics using high performance computing, numerical linear algebra, computational random matrix theory, and parallel computing, as well as interactive visualizations. Jiahao received his BS in Chemistry in 2002, MS in Applied Mathematics in 2008, and PhD in Chemical Physics with Computational Science and Engineering Option in 2009, all from the University of Illinois at Urbana-Champaign. He was previously a postdoctoral associate at the Massachusetts Institute of Technology, a Visiting Scholar at Ritsumeikan University (立命館大学) in Kusatsu City, Shiga Prefecture, Japan, and an acting Member of Technical Staff at DSO National Laboratories, Singapore.