Bayesian optimization, machine learning, and probabilistic numerics

Prompt optimization is a crucial task for improving the performance of Large Language Models for down-stream tasks. In this paper a prompt is a sequence of n-grams, selected from a vocabulary. Consequently, the aim is to select the optimal prompt with concerning a certain performance metric. Prompt optimization can be considered as a combinatorial optimization problem, with the number of possible prompts (i.e., the combinatorial search space) given by the size of vocabulary (i.e., all the possible n-grams) powered to the length of the prompt. Exhaustive search is impractical; thus, an efficient search strategy is needed. We propose a Bayesian Optimization method per-formed over a continuous relaxation of the combinatorial search space. Bayesian Optimization is the dominant approach in black-box optimization for its sample efficiency along with its modular structure and versatility. We use BoTorch, a library for Bayesian Optimization research built on top of PyTorch. Specifically, we focus on Hard Prompt Tuning which directly searches for an optimal prompt to be added to the text input without requiring access to the Large Language Model as well as using it as a black-box (such as for GPT-4 which is available as a Model as a Service). Albeit preliminary and based on "vanilla"Bayesian Optimization algorithms, our experiments with RoBERTa as Large Language Model, on six benchmark datasets, show good performances when compared against other state-of-the-art black-box prompt optimization methods and enable an analysis of the trade-off between size of the search space, accuracy, and wall clock time.