The decision boundary is fully determined(or supported) by the most extreme instances of the class, or, in other words,

While Linear SVM’s work well in most cases, it is extremely rare to have a dataset that is linearly separable. One approach to combat this is to add more features, such as polynomial features(theses essentially transform your features by raising the values to an N degree polynomial(think X²,X³, etc..)).,Note how the red point is an extreme outlier, and hence the SVM algorithm uses it as a support vector. Because the Hard Margin classifier finds the maximum distance between the support vectors, it uses the red outlier and the blue support vectors to set a decision boundary.,In the subsequent manhunts, Knowles roamed through woods, swampland, and avoided the police, the hunting dogs, and the helicopters searching him. Eventually, he was captured by a civilian: A Vietnam veteran armed with a shotgun. He tried escaping and denied confinement.,Again, the important idea to take from this is that the kernel function only calculates the high dimensional relationship between the points as if they were in high dimensions, but does not create or transform new features.,Although this approach can work, we have to figure out the optimal C parameter using cross-validation techniques. This can take a considerable amount of time. Additionally, one may want to create an optimal model and not have any “slack” variables that cross margin violations. So what is our solution now?,However, kernel functions only calculate the high dimensional relationships between the data points as if they were in a higher dimension; they do not actually do the transformation, meaning that the kernel function does not add any features, but we get the same results as if we id.,This trade-off is usually controlled by a hyperparameter that can be denoted by λ, or, more commonly(in scikit-learn) the C parameter. This essentially controls the misclassification cost. Concretely,,One approach is to find a good balance between keeping the streets as wide as possible(maximising the margin) and limiting the margin violations(these are instances that end up in the middle of the street or even on the wrong side of the street). This is called a soft margin SVM.,Essentially, this uses a Polynomial Kernel to calculate the high dimensional relationships between the data points and map the data into a higher dimension without adding any features.,Another SVM kernel that is extremely popular is the Gaussian Radial Based Function(Gaussian RBF). Essentially, this is a similarity function that computes the distance between instance and a landmark. The formula for the kernel function is given below:,C: the classification error that basically controls the trade-off between having the largest possible margin and maximise the number of points correctly classified by the decision boundary.,However, this is not feasible for large datasets; the computational complexity and the time it will take for the polynomial transformation to happen would be simply too long and computationally expensive.,Essentially, kernels are different functions that calculates the relationships between non-linearly separable data points and maps them into higher dimensions. It then fits a standard Support Vector Classifier. It effectively maps features from being in a relatively low dimension to a relatively high dimension.,In this example, we can clearly observe that there is no possible linear classifier that will separate the classes. Additionally, there is a major outlier. So, the question is, how can an SVM separate non-linearly separable data?,So essentially, a hard margin SVM basically tries to fit a decision boundary that maximises the distance between the support vectors of the two classes. However, there are a few issues with this model: