Nearest Neighbors in Python From Scratch
The k-Nearest Neighbors algorithm (or kNN for short) is an easy algorithm to understand and to implement, and a powerful tool to have at your disposal.
In this tutorial you will implement the k-Nearest Neighbors algorithm from scratch in Python (2.7). The implementation will be specific for classification problems and will be demonstrated using the Iris flowers classification problem.
This tutorial is for you if you are a Python programmer, or a programmer who can pick-up python quickly, and you are interested in how to implement the k-Nearest Neighbors algorithm from scratch.
k-Nearest Neighbors algorithm
Image from Wikipedia, all rights reserved
What is k-Nearest Neighbors
The model for kNN is the entire training dataset. When a prediction is required for a unseen data instance, the kNN algorithm will search through the training dataset for the k-most similar instances. The prediction attribute of the most similar instances is summarized and returned as the prediction for the unseen instance.
The similarity measure is dependent on the type of data. For real-valued data, the Euclidean distance can be used. Other other types of data such as categorical or binary data, Hamming distance can be used.
In the case of regression problems, the average of the predicted attribute may be returned. In the case of classification, the most prevalent class may be returned.
How does k-Nearest Neighbors Work
The kNN algorithm is belongs to the family of instance-based, competitive learning and lazy learning algorithms.
Instance-based algorithms are those algorithms that model the problem using data instances (or rows) in order to make predictive decisions. The kNN algorithm is an extreme form of instance-based methods because all training observations are retained as part of the model.
It is a competitive learning algorithm, because it internally uses competition between model elements (data instances) in order to make a predictive decision. The objective similarity measure between data instances causes each data instance to compete to “win” or be most similar to a given unseen data instance and contribute to a prediction.
Lazy learning refers to the fact that the algorithm does not build a model until the time that a prediction is required. It is lazy because it only does work at the last second. This has the benefit of only including data relevant to the unseen data, called a localized model. A disadvantage is that it can be computationally expensive to repeat the same or similar searches over larger training datasets.
Finally, kNN is powerful because it does not assume anything about the data, other than a distance measure can be calculated consistently between any two instances. As such, it is called non-parametric or non-linear as it does not assume a functional form.
Get your FREE Algorithms Mind Map
Sample of the handy machine learning algorithms mind map.
I've created a handy mind map of 60+ algorithms organized by type.
Download it, print it and use it.
Also get exclusive access to the machine learning algorithms email mini-course.
Classify Flowers Using Measurements
The test problem we will be using in this tutorial is iris classification.
The problem is comprised of 150 observations of iris flowers from three different species. There are 4 measurements of given flowers: sepal length, sepal width, petal length and petal width, all in the same unit of centimeters. The predicted attribute is the species, which is one of setosa, versicolor or virginica.
It is a standard dataset where the species is known for all instances. As such we can split the data into training and test datasets and use the results to evaluate our algorithm implementation. Good classification accuracy on this problem is above 90% correct, typically 96% or better.
Save the file in your current working directory with the file name “iris.data“.
How to implement k-Nearest Neighbors in Python
This tutorial is broken down into the following steps:
- Handle Data: Open the dataset from CSV and split into test/train datasets.
- Similarity: Calculate the distance between two data instances.
- Neighbors: Locate k most similar data instances.
- Response: Generate a response from a set of data instances.
- Accuracy: Summarize the accuracy of predictions.
- Main: Tie it all together.
1. Handle Data
The first thing we need to do is load our data file. The data is in CSV format without a header line or any quotes. We can open the file with the open function and read the data lines using the reader function in the csv module.
Read a CSV File in Python Python| 12345 | importcsvwithopen('iris.data','rb')ascsvfile:lines=csv.reader(csvfile)forrow inlines:print', '.join(row) |
Next we need to split the data into a training dataset that kNN can use to make predictions and a test dataset that we can use to evaluate the accuracy of the model.
We first need to convert the flower measures that were loaded as strings into numbers that we can work with. Next we need to split the data set randomly into train and datasets. A ratio of 67/33 for train/test is a standard ratio used.
Pulling it all together, we can define a function called loadDataset that loads a CSV with the provided filename and splits it randomly into train and test datasets using the provided split ratio.
Load a dataset and spit into train and test sets in Python Python| 12345678910111213 | importcsvimportrandomdefloadDataset(filename,split,trainingSet=[],testSet=[]):withopen(filename,'rb')ascsvfile:lines=csv.reader(csvfile)dataset=list(lines)forxinrange(len(dataset)-1):foryinrange(4):dataset[x][y]=float(dataset[x][y])ifrandom.random()<split:trainingSet.append(dataset[x])else:testSet.append(dataset[x]) |
Download the iris flowers dataset CSV file to the local directory. We can test this function out with our iris dataset, as follows:
Test the loadDataset function in Python Python| 12345 | trainingSet=[]testSet=[]loadDataset('iris.data',0.66,trainingSet,testSet)print'Train: '+repr(len(trainingSet))print'Test: '+repr(len(testSet)) |
2. Similarity
In order to make predictions we need to calculate the similarity between any two given data instances. This is needed so that we can locate the k most similar data instances in the training dataset for a given member of the test dataset and in turn make a prediction.
Given that all four flower measurements are numeric and have the same units, we can directly use the Euclidean distance measure. This is defined as the square root of the sum of the squared differences between the two arrays of numbers (read that again a few times and let it sink in).
Additionally, we want to control which fields to include in the distance calculation. Specifically, we only want to include the first 4 attributes. One approach is to limit the euclidean distance to a fixed length, ignoring the final dimension.
Putting all of this together we can define the euclideanDistance function as follows:
Calculate Euclidean distance in Python Python| 123456 | importmathdefeuclideanDistance(instance1,instance2,length):distance=0forxinrange(length):distance+=pow((instance1[x]-instance2[x]),2)returnmath.sqrt(distance) |
We can test this function with some sample data, as follows:
Test the euclideanDistance function in Python Python| 1234 | data1=[2,2,2,'a']data2=[4,4,4,'b']distance=euclideanDistance(data1,data2,3)print'Distance: '+repr(distance) |
3. Neighbors
Now that we have a similarity measure, we can use it collect the k most similar instances for a given unseen instance.
This is a straight forward process of calculating the distance for all instances and selecting a subset with the smallest distance values.
Below is the getNeighbors function that returns k most similar neighbors from the training set for a given test instance (using the already defined euclideanDistance function)
Locate most similar neighbours in Python Python| 123456789101112 | importoperatordefgetNeighbors(trainingSet,testInstance,k):distances=[]length=len(testInstance)-1forxinrange(len(trainingSet)):dist=euclideanDistance(testInstance,trainingSet[x],length)distances.append((trainingSet[x],dist))distances.sort(key=operator.itemgetter(1))neighbors=[]forxinrange(k):neighbors.append(distances[x][0])returnneighbors |
We can test out this function as follows:
Test the getNeighbors function in Python Python| 12345 | trainSet=[[2,2,2,'a'],[4,4,4,'b']]testInstance=[5,5,5]k=1neighbors=getNeighbors(trainSet,testInstance,1)print(neighbors) |
4. Response
Once we have located the most similar neighbors for a test instance, the next task is to devise a predicted response based on those neighbors.
We can do this by allowing each neighbor to vote for their class attribute, and take the majority vote as the prediction.
Below provides a function for getting the majority voted response from a number of neighbors. It assumes the class is the last attribute for each neighbor.
Summarize a prediction from neighbours in Python Python| 1234567891011 | importoperatordefgetResponse(neighbors):classVotes={}forxinrange(len(neighbors)):response=neighbors[x][-1]ifresponse inclassVotes:classVotes[response]+=1else:classVotes[response]=1sortedVotes=sorted(classVotes.iteritems(),key=operator.itemgetter(1),reverse=True)returnsortedVotes[0][0] |
We can test out this function with some test neighbors, as follows:
Test the getResponse function in Python Python| 123 | neighbors=[[1,1,1,'a'],[2,2,2,'a'],[3,3,3,'b']]response=getResponse(neighbors)print(response) |
This approach returns one response in the case of a draw, but you could handle such cases in a specific way, such as returning no response or selecting an unbiased random response.
5. Accuracy
We have all of the pieces of the kNN algorithm in place. An important remaining concern is how to evaluate the accuracy of predictions.
An easy way to evaluate the accuracy of the model is to calculate a ratio of the total correct predictions out of all predictions made, called the classification accuracy.
Below is the getAccuracy function that sums the total correct predictions and returns the accuracy as a percentage of correct classifications.
Calculate accuracy of predictions in Python Python| 123456 | defgetAccuracy(testSet,predictions):correct=0forxinrange(len(testSet)):iftestSet[x][-1]ispredictions[x]:correct+=1return(correct/float(len(testSet)))*100.0 |
We can test this function with a test dataset and predictions, as follows:
Test the getAccuracy function in Python Python| 1234 | testSet=[[1,1,1,'a'],[2,2,2,'a'],[3,3,3,'b']]predictions=['a','a','a']accuracy=getAccuracy(testSet,predictions)print(accuracy) |
6. Main
We now have all the elements of the algorithm and we can tie them together with a main function.
Below is the complete example of implementing the kNN algorithm from