Support Vector Machines

1. Cost Function

$$J(\theta)= C \left[\sum_{i=1}^m y^{(i)} \text{cost}_1(\theta^T x^{(i)})+(1-y^{(i)}) \text{cost}_0(\theta^T x^{(i)})\right] + \frac{1}{2}\sum_{j=1}^n\theta^2_j$$

2. Hypothesis

$$f(x)= \begin{cases} 1& \text{if $\theta^T$x>=0}\\ 0& \text{otherwise} \end{cases}$$

3. Margin of SVM

4. Kernels

Define landmarks $l$



Using $f$ and $\theta$ when making predictions..

5. How to Get Landmarks

One way is to use the first m training examples.

6. The Effects of Parameters in SVM

1) For $C$
Large C : λ small, low bias, high variance
Small C : λ big, high bias, low variance
2) For $\sigma^2$
Large $\sigma^2$ : high bias, low variance
Small $\sigma^2$ : low bias, high variance

7. Choice of Kernal

Need to satisfy Mercer’s Theorem.
1) No kernal (Linear Kernal)
when n is large/ n is small && m is large
2) Gaussian Kernal
when n is small, m is intermediate
Need to use feature scaling before using!
3) Other Alternative Choices:
Polynomial Kernal, String Kernal, Chi-Square Kernal, Intersection Kernal…