Logistic_Regression_MLE

Logistic regression parameters are estimated using Maximum Likelihood Estimation (MLE).

The reason for using MLE in logistic regression is that it is a statistical method that estimates the parameters of a probability distribution or statistical model. In the context of logistic regression, MLE is used to find the best-fitting parameters for the model, which are the coefficients that describe the relationship between the independent variables and the probability of the dependent variable.

The process involves finding the parameters that maximize the likelihood function, which is a measure of how well the model explains the observed data. The likelihood function is the probability of observing the given data under the model with the parameters in question. In logistic regression, the log-likelihood function is typically used because it is mathematically more convenient to work with, as it turns the product of probabilities into a sum of log probabilities, which is easier to optimize.

MLE is preferred over other methods, such as least squares estimation, because it provides estimates that are asymptotically unbiased, consistent, and efficient, assuming the model is correctly specified. This means that as the sample size increases, the MLE will converge to the true parameter values. Additionally, MLE takes into account the binary nature of the logistic regression model, where the outcome is a probability that is bounded between 0 and 1.

The Newton-Raphson method is often used to solve the MLE problem in logistic regression because it is an iterative method that can handle the nonlinear nature of the logistic function. This method adjusts the parameters iteratively to find the values that maximize the log-likelihood function. The process continues until convergence, which means that the estimates no longer change significantly from one iteration to the next.

Given 大数定理 和 中心极限定理，MLE is “good”.

Asymptotic Normality 渐近正态 of Maximum Likelihood Estimators:
https://gregorygundersen.com/blog/2019/11/28/asymptotic-normality-mle/

MIT: Properties of MLE: consistency, asymptotic normality. Fisher information.
https://ocw.mit.edu/courses/18-443-statistics-for-applications-fall-2006/03b407da8a94b3fe22d987453807ca46_lecture3.pdf