Bayesian Learning Algorithm

machine-learning statistics

Definition

Bayesian Learning Algorithm

A Bayesian learning algorithm is a procedure that treats model parameters $\theta$ as random variables and utilises Bayes’ theorem to update the probability distribution over the hypothesis class $\mathcal{H}$ based on observed data $D$. Formally, the learner computes the posterior distribution $P(\theta |D)$:

$$P(\theta |D)=\frac{P(D|\theta )P(\theta )}{P(D)}$$

where $P(\theta )$ is the prior distribution, $P(D|\theta )$ is the likelihood function, and $P(D)$ is the evidence.

Inference and Prediction

Posterior Predictive Distribution: Predictions for a new instance $x^{\ast }$ are made by integrating over the parameter space $\Theta$, weighting each possible model by its posterior probability:

$$P(y^{\ast }|x^{\ast },D)={\int }_{\Theta }P(y^{\ast }|x^{\ast },\theta )P(\theta |D)d\theta$$

Uncertainty Quantification: Unlike frequentist methods that produce point estimates, Bayesian learning provides a full distribution over the parameters, allowing for the explicit modelling of epistemic uncertainty.