Support Vector Machine

Support Vector Machine Review

Posted by CHENEY WANG on November 19, 2018

Recently by the same author:


实体关系抽取

文本中实体关系的抽取

You may find interesting:


Logistic Regression

Logistic Regression Review


Logistic Regression

Logistic Regression Review

支持向量机


Support Vector Machine Conduction By Andrew Wu

这部分是根据吴恩达对于SVM的推导。 In this cost function, C is used to do regularization which is similar to $\lambda$ in logistic regression.
其实,我还是没有理解,为什么我们可以直接获得这个代价函数,可能是要从原理推导太难了吧,作为ML的入门课程,他并没有从最初的状态去推导这个方程。

Vector Inner Product (内积)

Inner product of two vectors.

SVM Decision Boundary.

SVM(周志华西瓜书版本)

西瓜书中从几何角度对支持向量机进行了讨论。

所以异类向量(即点到平面的距离): $r = \frac{W^Tx+b}{||W||}$。
Then, $r = \frac{2|W^Tx_i+b|} = \frac{2}{||W||}$.
Then, we have to minimize $\frac{2}{||W||}$. In the other words, $min\frac{2}{||W||} = max\frac{||W||}{2}$ .
所以我们的objective function就是:

然后对于求解这个目标函数,我们需要加入拉格朗日乘子来将约束条件加入到目标函数中形成一个对偶问题。

Lagrange multiplier(拉格朗日乘子)

拉格朗日乘子在这用于解决约束性问题,从而解出SVM的损失函数。 以下是维基百科对拉格朗日乘子的解释:
For the case of only one constraint and only two choice variables (as exemplified in Figure 1), consider the optimization problem maximize f(x, y) subject to g(x, y) = 0. (Sometimes an additive constant is shown separately rather than being included in g, in which case the constraint is written g(x, y) = c, as in Figure 1.) We assume that both f and g have continuous first partial derivatives. We introduce a new variable (λ) called a Lagrange multiplier and study the Lagrange function (or Lagrangian or Lagrangian expression) defined by where the λ term may be either added or subtracted.

Obective Function after add Lagrange multiplier

其中,对于非支持向量,$\alpha_i = 0 $, 因为SVM计算时,只考虑超平面附近的点,所以其他的点是不影响超平面的。从另外一个方向上来说,这也是为什么对于多纬空间来说,支持向量机的计算量要远小于LR,并且可以使用kerne来进行空间映射。
接下去还有支持向量机的对偶问题求解。目前还需要回顾一部分高数知识才能继续进行。