Maximum Likelihood Estimation

statistics machine-learning optimisation

Definition

Maximum Likelihood Estimation

Maximum Likelihood Estimation (MLE) is a frequentist method for estimating the parameters $\theta$ of a statistical model. Formally, given observed data $D$, MLE identifies the parameter values that maximise the likelihood function $P(D|\theta )$, making the observed data as probable as possible under the model:

${\theta }_{\text{MLE}}^{\ast }=\arg {\max }_{\theta \in \Theta }P(D|\theta )$

Example: Bernoulli Parameter

Consider a sequence of $m$ independent coin tosses where $k$ heads are observed. The maximum likelihood estimate for the probability of heads $\theta$ is the relative frequency:

${\hat{\theta }}_{\text{MLE}}=\frac{k}{m}$

Implementation Details

In practice, the log-likelihood $\ln P(D|\theta )$ is typically maximised to simplify computations for independent samples.

Limitations

Overfitting: MLE is highly sensitive to the specific dataset provided and does not incorporate any prior knowledge. In settings with sparse data, this can lead to extreme parameter estimates (e.g., assigning zero probability to an event not seen in the training set).

Comparison with MAP: Unlike MAP, MLE assumes a uniform or non-informative prior distribution over the parameters, ignoring any initial beliefs about the hypothesis space.