Logistic Regression

Logistic Regression Review

Posted by CHENEY WANG on November 17, 2018

Recently by the same author:


实体关系抽取

文本中实体关系的抽取

You may find interesting:


Support Vector Machine

Support Vector Machine Review


Support Vector Machine

Support Vector Machine Review

逻辑回归回顾


这篇分享,我仅从频率学学派派去推导逻辑回归。后期会加上从贝叶斯学派进行的逻辑回归推导 This post, I’m gonna introduce logistic regression from frequentist perspective, and I will add conduction of logistic regression from bayes perspective.

Logistic Regression Summary

Logistic regression is the appropriate regression analysis to conduct when the dependent variable is dichotomous (binary). Like all regression analyses, the logistic regression is a predictive analysis. Logistic regression is used to describe data and to explain the relationship between one dependent binary variable and one or more nominal, ordinal, interval or ratio-level independent variables. image

Mathmatical function of LR:

Base function( hypothesis fucntion of linear regression )

Sigmoid Function of LR:

Sigmoid function is implemented to compress predict value into {0,1}, which can help us to do classification.

Hypothesis function of LR:

And different from linear regression, the graph of this function is not a bowl-shaped funciton . Therefore, it would be hard to get a global minimum. So loss function of logistic regression is different from linear regression.

Cost function of LR:

If we use least square as our loss function, the cost function would be a non-convex function that we can not get global minimum.
imgae

In order to overcome this issue, we use logistic loss as our loss function and cross-entry as our cost function. image aas Therefore, it will penalize learning algorithm by a large cost if y is not equal to $h_\theta(x)$

Regularization

Same as regularization in linear regression, here we always use L2-norm as our penality item.

Overfitting

There are several options to deal with overfitting:

  1. Reduce number of features
  2. Regularization
  3. Enlarger the data set