Logistic Regression

Logistic Regression Review

Posted by CHENEY WANG on November 17, 2018

Recently by the same author:



You may find interesting:

Support Vector Machine

Support Vector Machine Review

Support Vector Machine

Support Vector Machine Review


这篇分享,我仅从频率学学派派去推导逻辑回归。后期会加上从贝叶斯学派进行的逻辑回归推导 This post, I’m gonna introduce logistic regression from frequentist perspective, and I will add conduction of logistic regression from bayes perspective.

Logistic Regression Summary

Logistic regression is the appropriate regression analysis to conduct when the dependent variable is dichotomous (binary). Like all regression analyses, the logistic regression is a predictive analysis. Logistic regression is used to describe data and to explain the relationship between one dependent binary variable and one or more nominal, ordinal, interval or ratio-level independent variables. image

Mathmatical function of LR:

Base function( hypothesis fucntion of linear regression )

Sigmoid Function of LR:

Sigmoid function is implemented to compress predict value into {0,1}, which can help us to do classification.

Hypothesis function of LR:

And different from linear regression, the graph of this function is not a bowl-shaped funciton . Therefore, it would be hard to get a global minimum. So loss function of logistic regression is different from linear regression.

Cost function of LR:

If we use least square as our loss function, the cost function would be a non-convex function that we can not get global minimum.

In order to overcome this issue, we use logistic loss as our loss function and cross-entry as our cost function. image aas Therefore, it will penalize learning algorithm by a large cost if y is not equal to $h_\theta(x)$


Same as regularization in linear regression, here we always use L2-norm as our penality item.


There are several options to deal with overfitting:

  1. Reduce number of features
  2. Regularization
  3. Enlarger the data set