Maximum Likelihood Estimation (MLE) Formula for Machine Learning

In the Logistic Regression for Machine Learning using Python blog, I have introduced the basic idea of the logistic function. The likelihood, finding the best fit for the sigmoid curve.

We have discussed the cost function. And we also saw two ways of optimizing the cost function

1. Closed-form solution
2. Iterative form solution

And in the iterative method, we focus on the Gradient descent optimization method. (An Intuition Behind Gradient Descent using Python).

Now so in this section, we are going to introduce the Maximum Likelihood cost function. And we would like to maximize this cost function.

The MLE formula is the foundation of maximum likelihood estimation in machine learning.

MLE formula (step by step)

Maximum Likelihood Estimation (MLE) picks the model parameters that make your observed data most probable. The core MLE formula is:

L(θ) = Π P(xᵢ | θ) → maximize log-likelihood: ℓ(θ) = Σ log P(xᵢ | θ)

In practice, we maximize ℓ(θ) (log-likelihood) instead of L(θ) because products become sums and optimization is stable.

For a normal distribution with mean μ and variance σ², a common MLE formula example is:

μ̂_MLE = (1/n) Σ xᵢ
σ²_MLE = (1/n) Σ (xᵢ − μ̂)²

This is why MLE shows up everywhere in machine learning: linear models, logistic regression, and many probabilistic classifiers use the same idea—find θ that best explains the data you actually saw.

Maximum Likelihood Cost Function

There are two types of random variables.

• Discrete
• Continuous

The discrete variable that can take a finite number. A discrete variable can be separated. For example, in a coin toss experiment, only heads or tails will appear. If the dice tosses only 1 to 6 values can appear.
A continuous variable example is the height of a man or a woman, such as 5ft, 5.5ft, 6ft, etc.

The random variable whose value is determined by a probability distribution.

Let say you have N observations x1, x2, x3,…xN.

For example, each data point represents the height of the person. For these data points, we’ll assume that the data generation process is described by a Gaussian (normal) distribution.

As we know, any Gaussian (Normal) distribution has two parameters. The mean μ, and the standard deviation σ. So if we minimize or maximize as per need, the cost function. We will get the optimized μ and σ.

In the above example Red curve is the best distribution for the cost function to maximize.

Since we chose Theta Red, we want the probability to be high for this. We would like to maximize the probability of observation x1, x2, x3, xN, based on the higher probability of theta.

Now once we have this cost function defined in terms of θ. In order to simplify, we need to add some assumptions.

X1, X2, X3… XN are independent. Let say X1,X2,X3,…XN is a joint distribution, which means the observation sample is a random selection. With this random sampling, we can pick this as a product of the cost function.

General Steps

We choose log to simplify the exponential terms into a linear form. So in general, these three steps are used.

• Define the Cost function
• Making the independent assumption
• Taking the log to simplify

So let’s follow all three steps for the Gaussian distribution, where θ is nothing but μ and σ.

Maximum Likelihood Estimation for Continuous DistributionsThe

MLE technique finds the parameter that maximizes the likelihood of the observation. For example, in a normal (or Gaussian) distribution, the parameters are the mean μ and the standard deviation σ.

For example, we have the ages of 1000 random people, which is normally distributed. There is a general thumb rule that nature follows the Gaussian distribution. The central limit theorem plays a gin role but only applies to large datasets.

The equation of the normal distribution or the Gaussian distribution is as below

Upon differentiating the log-likelihood function with respect to μ and σ respectively, we’ll get the following estimates:

Additional reading

• Difference between likelihood and probability
• Maximum Likelihood Estimation (MLE) in layman’s terms

Maximum Likelihood Estimation for Discrete Distributions

The Bernoulli distribution models events with two possible outcomes: either success or failure.

If the probability of a successful event is P then the probability of Failure would be (1-P).

The Binary Logistic Regression problem is also a Bernoulli distribution. And thus, a Bernoulli distribution will help you understand MLE for logistic regression.

Now lets say we have N desecrate observation {H,T} heads and Tails. So will define the cost function first for Likelihood as bellow:

In order to do a closed-form solution, we can differentiate and equate to 0.

So we got a very intuitive observation here. The likelihood for p based on X is defined as the joint probability distribution of X₁, X₂,…, X_n.

Now we can say that Maximum Likelihood Estimation (MLE) is a very general procedure not only for the Gaussian. But the observation where the distribution is Desecrate.

Maximum Likelihood Estimation for Logistic Regression

So let’s say we have datasets X with m data points. Now the logistic regression says that the probability of the outcome can be modeled as follows.

Based on the probability rule. If the success event probability is P than fail event would be (1-P). That’s how the Yi indicates above.

This can be combined into a single form as below.

Which means, what is the probability of Xi occurring for a given Yi value, P(x|y)?

The likelihood of the entire dataset X is the product of an individual data point. Which means forgiven event (coin toss) H or T. If H probability is P, then T probability is (1-P).

So, the likelihood of these two events is.

Now the principle of maximum likelihood says. we need to find the probability that maximizes the likelihood P(X|Y). Recall the odds and log-odds.

So as we can see now. After taking a log, we can end up with a linear equation.

So in order to get the parameter θ of the hypothesis. We can either maximize the likelihood or minimize the cost function.

Now we can take a log from the above logistic regression likelihood equation. This will normalize the equation into log-odds.

Now Maximum likelihood estimation (MLE) is as follows.