Bayesian optimization

Bayesian optimization is a sequential model-based strategy for global optimization of black-box objective functions whose evaluations are costly. It is commonly used when a single observation requires an experiment, engineering computation, numerical simulation, or machine-learning run, and when derivatives are unavailable or unreliable. The objective need not have a closed-form expression.

The method constructs a probabilistic model of the unknown function, often a Gaussian process (GP), and uses the resulting predictive distribution to choose the next evaluation point. This choice is made by optimizing a sampling criterion, also called an acquisition function.

Common applications include hyperparameter optimization in machine learning, where each trial may require training and validating a model, and engineering design problems driven by expensive numerical simulations.

History

Early Bayesian approaches to global optimization include work by Harold J. Kushner on locating extrema of noisy functions and work by Jonas Mockus on Bayesian methods for seeking extrema.

Expected improvement is a prominent sampling criterion in this line of work. In 1998, Donald R. Jones, Matthias Schonlau, and William J. Welch introduced the efficient global optimization (EGO) algorithm, which used a kriging or Gaussian-process model with expected improvement for expensive black-box functions.

Later work extended Bayesian optimization to noisy observations, constraints, batch and parallel evaluations, multiple objectives, and mixed or high-dimensional search spaces.