Gaussian Process
- Definition
- A Gaussian Process (GP) is a way to model unknown functions probabilistically. Instead of learning a single curve, a GP gives a distribution over possible functions that fit the data, and it provides both predictions and uncertainty estimates for any input. Think of a GP as a flexible, Bayesian regression method that tells you not only the predicted value but also how confident it is. GPs differ from parametric models (like simple linear regression) by being nonparametric: complexity grows with data instead of being fixed by a small set of parameters.
Links
- Surrogate modelling — Model expensive-to-evaluate functions (e.g., engineering simulations) to guide Bayesian optimization.
- Spatial interpolation (kriging) — Predict environmental measurements at unsampled locations with uncertainty estimates.
- Time-series smoothing — Nonparametric regression for small-data forecasts with calibrated uncertainty bands.
- Cubic time complexity in the number of training points, limiting scalability
- Choosing an appropriate kernel and its hyperparameters
- Numerical instability when inverting near-singular covariance matrices
- Poor performance on high-dimensional inputs
- Difficulty handling non-Gaussian or heteroscedastic noise
How does it work?
Gaussian Process algorithms typically represent uncertainty explicitly and use probability calculations to combine data and model assumptions. Implementations focus on numerical stability, how to represent distributions (parametric vs sampled), and efficient updates when new data arrive.