Multi-Information Source Bayesian Optimization with Location-Varying Costs

This paper addresses black-box optimization over multiple information sources whose fidelity and query cost change across the search space. We propose an approach that uses: (i) an Augmented Gaussian Process as a single model of the objective function over the search space and sources, and (ii) a Gaussian Process to model the location-dependent cost of each source. The former is used in a Confidence Bound-based acquisition function to select the next source and location to query, while the latter is used to penalize the value of the acquisition depending on the expected query cost for any source-location pair. Multi-information source optimization (MISO) is characterized by the availability of inexpensive approximations (i.e., sources) of the more expensive objective function. In contrast, cost-aware Bayesian optimization addresses settings where the query cost varies across the search space. When information sources come with explicit information about their approximation quality (aka fidelity), MISO specializes in multi-fidelity optimization. However, most multi-fidelity approaches require hierarchically organized sources and cannot handle location-dependent fidelities. A key contribution of this manuscript is a multi-information source setting in which both fidelity and query cost of the sources are black-box and location-dependent, endowed with a cost and location-aware acquisition function. We provide an empirical evaluation of the proposed method on an AutoML task: hyperparameter optimization of two classifiers on three datasets of different sizes. The acquisition function proposed can be naturally generalized to more complex cost models, including movement costs, which represents an important perspective for multi-source Bayesian optimization.