Bayesian Hybrid Model Search and Averaging for Sparse Gaussian Process Regression

Gaussian process (GP) regression has been a popular nonparametric Bayesian approach for nonlinear modeling and prediction in the fields of statistics and machine learning. However, when many predictors are considered for the construction of the kernel function, the GP approach provides unacceptable performance in both estimation and prediction. To overcome this limitation, some attempts have been made to exploit a fully Bayesian model selection approach or a penalized likelihood approach. However, the fully Bayesian framework turns out to be extremely expensive in computational terms, and the penalized likelihood method oversimplifies model uncertainties. In this paper, we propose a new sparse GP method that reduces the computational burden of fully Bayesian inference by incorporating a hybrid deterministic-stochastic search approach into Bayesian model averaging. In addition, we develop a scalable extension of the proposed method to high-dimensional massive data settings. The merits of the proposed methods are demonstrated via simulation experiments and real data applications.