Logistic Regression

Logistic Regression Review

Posted by CHENEY WANG on November 17, 2018

逻辑回归回顾

Logistic Regression Summary

Logistic regression is the appropriate regression analysis to conduct when the dependent variable is dichotomous (binary). Like all regression analyses, the logistic regression is a predictive analysis. Logistic regression is used to describe data and to explain the relationship between one dependent binary variable and one or more nominal, ordinal, interval or ratio-level independent variables.

Mathmatical function of LR:

Base function( hypothesis fucntion of linear regression )

\begin{align} \text{Hypothesis:} \quad H_w(X) = W_1+W_1X_2+ ... + W_nX_n \end{align}

Sigmoid Function of LR:

Sigmoid function is implemented to compress predict value into {0,1}, which can help us to do classification. %

Hypothesis function of LR:

\begin{align} \text{Hypothesis of LR} = \frac{1}{1+e^{-w^{T}x}} \end{align} And different from linear regression, the graph of this function is not a bowl-shaped funciton . Therefore, it would be hard to get a global minimum. So loss function of logistic regression is different from linear regression.

Cost function of LR:

If we use least square as our loss function, the cost function would be a non-convex function that we can not get global minimum.
\begin{align} J(\theta) = (y^i - H_\theta(X^{i})) \end{align}

In order to overcome this issue, we use logistic loss as our loss function and cross-entry as our cost function. % % % Therefore, it will penalize learning algorithm by a large cost if y is not equal to $h_\theta(x)$

Regularization

Same as regularization in linear regression, here we always use L2-norm as our penality item.

Overfitting

There are several options to deal with overfitting:

1. Reduce number of features
2. Regularization
3. Enlarger the data set