Support Vector Machine

Support Vector Machine Review

Posted by CHENEY WANG on November 19, 2018

Recently by the same author:



You may find interesting:

Logistic Regression

Logistic Regression Review

Logistic Regression

Logistic Regression Review


Support Vector Machine Conduction By Andrew Wu

这部分是根据吴恩达对于SVM的推导。 In this cost function, C is used to do regularization which is similar to $\lambda$ in logistic regression.

Vector Inner Product (内积)

Inner product of two vectors.

SVM Decision Boundary.



所以异类向量(即点到平面的距离): $r = \frac{W^Tx+b}{||W||}$。
Then, $r = \frac{2|W^Tx_i+b|} = \frac{2}{||W||}$.
Then, we have to minimize $\frac{2}{||W||}$. In the other words, $min\frac{2}{||W||} = max\frac{||W||}{2}$ .
所以我们的objective function就是:


Lagrange multiplier(拉格朗日乘子)

拉格朗日乘子在这用于解决约束性问题,从而解出SVM的损失函数。 以下是维基百科对拉格朗日乘子的解释:
For the case of only one constraint and only two choice variables (as exemplified in Figure 1), consider the optimization problem maximize f(x, y) subject to g(x, y) = 0. (Sometimes an additive constant is shown separately rather than being included in g, in which case the constraint is written g(x, y) = c, as in Figure 1.) We assume that both f and g have continuous first partial derivatives. We introduce a new variable (λ) called a Lagrange multiplier and study the Lagrange function (or Lagrangian or Lagrangian expression) defined by where the λ term may be either added or subtracted.

Obective Function after add Lagrange multiplier

其中,对于非支持向量,$\alpha_i = 0 $, 因为SVM计算时,只考虑超平面附近的点,所以其他的点是不影响超平面的。从另外一个方向上来说,这也是为什么对于多纬空间来说,支持向量机的计算量要远小于LR,并且可以使用kerne来进行空间映射。