Maximum Likelihood Estimation

Created December 16, 2020 ยท Updated March 4, 2026

Maximum likelihood principle states that the most likely explanation of data D is given by $W_{ML}$ which maximizes the likelihood function.

$$ W_{ML} = \underset{\mathbf{w}}{\arg \max } p(D|w) $$

Let's assume the data is i.i.d. This means the joint probability reduces to the product of individual PDFs. (For correlated data ex. time series, we can't assume this)

$$ p(D \mid \mathbf{w})=p\left(x_{1}, x_{2}, \ldots, x_{N} \mid \mathbf{w}\right)=\prod_{i=1}^{N} p\left(x_{i} \mid \mathbf{w}\right) $$

So, maximum likelihood estimation is given as:

$$ \mathbf{w}_{\mathrm{ML}}=\underset{\mathbf{w}}{\arg \max } p(D \mid \mathbf{w})=\underset{\mathbf{w}}{\arg \max } \prod_{i=1}^{N} p\left(x_{i} \mid \mathbf{w}\right) $$

Since logarithm is monotonically increasing function of its argument, maximizing log of a function is equivalent to maximization of function itself. Using logarithm also helps with preventing numerical underflow as product of small numbers grows smaller very fast. So,

$$ \mathbf{w}_{\mathrm{ML}}=\underset{\mathbf{w}}{\arg \max } \sum_{i=1}^{N} \log p\left(x_{i} \mid \mathbf{w}\right) $$

We find analytical solutions of ML estimates of parameters by taking the derivative of likelihood function with respect to the parameters and setting them to zero.

MLE of Gaussian Distribution

Let's assume the dataset consists of i.i.d Gaussian distributed real variables. Then we the likelihood function is given as the product of individual PDFs,

$$ p(x \mid \mathbf{w})=\mathcal{N}\left(x \mid \mu, \sigma^{2}\right) = p\left(D \mid \mu, \sigma^{2}\right)=\frac{1}{\left(2 \pi \sigma^{2}\right)^{N / 2}} \prod_{i=1}^{N} \exp \left[-\frac{1}{2 \sigma^{2}}\left(x_{i}-\mu\right)^{2}\right] $$

Now, using log of the the distribution,

$$ \begin{align} \log p\left(D \mid \mu, \sigma^{2}\right)&=\log \left(2 \pi \sigma^{2}\right)^{-N / 2}+\sum_{i=1}^{N} \log \exp \left[-\frac{1}{2 \sigma^{2}}\left(x_{i}-\mu\right)^{2}\right] \\ &= -\frac{N}{2} \log \left(2 \pi \sigma^{2}\right)+\sum_{i=1}^{N}-\frac{1}{2 \sigma^{2}}\left(x_{i}-\mu\right)^{2} \end{align} $$

Now for maximizing likelihood for $\mu$, we take the derivative wtih respect to $\mu$,

$$ \begin{align} \frac{\partial}{\partial \mu} \log p\left(D \mid \mu, \sigma^{2}\right) &= 0 \\ \frac{1}{2\sigma^2} \sum_{i=1^N}2(x_i - \mu) &= 0 \\ \sum_{i=1}^N(x_i - \mu) &= 0 \\ \sum_{i=1}^N\mu &= \sum_{i=1^N}x_i \\ N.\mu &= \sum_{i=1}^N x_i \\ \mu &= \frac{1}{N}\sum_{i=1}^N x_i \end{align} $$

which is the sample mean. Therefore, $\mu_{ML}$ is equal to the sample mean.

Now, maximizing the likelihood for $\sigma^2$, we take the derivative with respect to $\sigma^2$,

$$ \frac{\partial}{\partial \sigma^{2}} \log p\left(D \mid \mu, \sigma^{2}\right) = -\frac{N}{2}\frac{1}{2\pi\sigma^2}.2\pi + \frac{1}{2\sigma^4}\sum_{i=1}{N}(x_i - \mu^2) = 0 $$

Multiplying by $2\sigma^2$,

$$ \begin{align} -N\sigma^2 + \sigma_{i=1}^{N}(x_i - \mu^2) &= 0 \\ \sigma^2 &= \frac{1}{N}\sum_{i=1}^N (x - \mu^2) \end{align} $$

which is the sample variance. Therefore, $\sigma^2_{ML}$ is equal to the sample variance.

MLE of Binomial Distribution

The likelihood function

$$ f(m \mid n, p)=\left(\begin{array}{c} n \\ m \end{array}\right) p^{m}(1-p)^{n-m} $$

Find extremum using the log-likelihood.

$$ \frac{d}{d p} \ln f(m \mid n, p)=\frac{m}{p}-\frac{n-m}{1-p}=\Longleftrightarrow m(1-p)=p(n-m) \Longleftrightarrow m=p n \Longleftrightarrow p=\frac{m}{n} $$

Bias in Maximum Likelihood Estimators

Bias is the difference in the statistic of the estimator and the true statistic. To check MLE for bias, let's do a sanity check.

Let's suppose we draw two datasets $D_1$ and $D2$ from a Gaussian distribution. We know that $\mu_{ML}$ and $\mu^2_{ML}$ are the functions of dataset values.

Now, for case of $\mu_{ML}$,

$$ \begin{align} \mathbb{E}_{D \sim p\left(D \mid \mu, \sigma^{2}\right)}\left[\mu_{M L}\right] &=\mathbb{E}\left[\frac{1}{N} \sum_{i=1}^{N} x_{i}\right] \\ &= \frac{1}{N} \sum_{i=1}^{N} \mathbb{E}_{D \sim p\left(D \mid \mu, \sigma^{2}\right)}[x_{i}] \\ &= \frac{1}{N} \sum_{i=1}^{N} \mathbb{E}_{x_i \sim p\left(x_i \mid \mu, \sigma^{2}\right)}[x_{i}] \\ &= \frac{1}{N} \sum_{i=1}^{N} \mu \\ &= \mu \end{align} $$

Therefore, bias of the estimator $\mathbb{E}[\mu_{ML}] - \mu = 0$

In case of variance,

$$ \begin{align} \mathbb{E}_{D \sim p(D \mid \mu, \sigma)}\left[\sigma_{M L}^{2}\right]&=E\left[\frac{1}{N} \sum_{i=1}^{N}\left(x_{i}-\frac{1}{N} \sum_{n=1}^{N} x_{n}\right)^{2}\right] \\ &=\frac{1}{N} \sum_{i=1}^{N} E\left[\left(x i-\frac{1}{N} \sum_{n=1}^{N} x_{n}\right)^{2}\right]\\ &= \frac{1}{N} \sum_{i=1}^{N} E\left[x_{i}^{2}-\frac{2 x_{i}}{N} \sum_{n=1}^{N} x_{n}+\frac{1}{N^{2}} \sum_{m=1}^{N} \sum_{n=1}^{N} x_{m} x_{n}\right]\\ &=\frac{1}{N} \sum_{i=1}^{N}\left\{E\left[x_{i}^{2}\right]-\frac{2}{N} \sum_{n=1}^{N} \mathbb{E}\left[x_{i} x_{n}\right]+\frac{1}{N^{2}} \sum_{m=1}^{N} \sum_{n=1}^{N} E\left[x_{m} x_{n}\right]\right\} \end{align} $$

Now,

$$ \begin{align} E[x_i,x_i] = cov[x_i] \Rightarrow &E[x_i^2] - E[x_i]^2 = \sigma^2 \\ & E[x_i^2] = \mu^2 + \sigma^2\\ \\ E[x_i,x_j] = cov[x_i,x_j] \Rightarrow &E[x_i.x_j] - E[x_i]E[x_j] = 0 \\ & E[x_i.x_j] = \mu^2 & \end{align} $$

Using these values in the expression above,

$$ \begin{align} \mathbb{E}_{D \sim p(D \mid \mu, \sigma)}\left[\sigma_{M L}^{2}\right]&= \frac{1}{N} \sum_{i=1}^{N}\left\{\mu^{2}+\sigma^{2}-\frac{2}{N}\left(N \mu^{2}+\sigma^{2}\right)+\frac{1}{N^{2}}\left(N^{2} \mu^{2}+N \sigma^{2}\right)\right\} \\ &= \frac{N-1}{N}\sigma^2 \end{align} $$

Therefore, bias of the estimator is $\frac{N-1}{N}$.

This means that MLE behaves in a biased way when N is small, but as $N\Rightarrow\infty$ the bias is reduced. This can also be adjusted by apply the transformation of $\frac{N}{N-1}$.