Spark/Scala在推薦系統中實現相似度演算法(歐氏距離、皮爾遜相關係數、餘弦相似度:帶實現程式碼)
阿新 • • 發佈:2020-12-12
在推薦系統中,協同過濾演算法被廣泛使用,主要分為基於使用者和基於專案的協同過濾演算法。核心點基於“一個人”或者“一個物品”。根據這個人或者物品的屬性,比如性別、年齡、工作、收入、喜好等。,找到與此人或物品相似的人或物。當然,實際處理中的參考因素會複雜得多。
本文不介紹相關的數學概念,主要給出常用相似度演算法的程式碼實現,同樣的演算法有多種實現方式。
歐幾里得距離
def euclidean2(v1: Vector, v2: Vector): Double = {
require(v1.size == v2.size, s"SimilarityAlgorithms:Vector dimensions do not match: Dim(v1)=${v1.size} and Dim(v2)" +
s"=${v2.size}.")
val x = v1.toArray
val y = v2.toArray
euclidean(x, y)
}
def euclidean(x: Array[Double], y: Array[Double]): Double = {
require(x.length == y.length, s"SimilarityAlgorithms:Array length do not match: Len(x)=${x.length} and Len(y)" +
s"=${y.length}." )
math.sqrt(x.zip(y).map(p => p._1 - p._2).map(d => d * d).sum)
}
def euclidean(v1: Vector, v2: Vector): Double = {
val sqdist = Vectors.sqdist(v1, v2)
math.sqrt(sqdist)
}
皮爾遜相關係數
def pearsonCorrelationSimilarity(arr1: Array[Double], arr2: Array[Double]): Double = {
require( arr1.length == arr2.length, s"SimilarityAlgorithms:Array length do not match: Len(x)=${arr1.length} and Len(y)" +
s"=${arr2.length}.")
val sum_vec1 = arr1.sum
val sum_vec2 = arr2.sum
val square_sum_vec1 = arr1.map(x => x * x).sum
val square_sum_vec2 = arr2.map(x => x * x).sum
val zipVec = arr1.zip(arr2)
val product = zipVec.map(x => x._1 * x._2).sum
val numerator = product - (sum_vec1 * sum_vec2 / arr1.length)
val dominator = math.pow((square_sum_vec1 - math.pow(sum_vec1, 2) / arr1.length) * (square_sum_vec2 - math.pow(sum_vec2, 2) / arr2.length), 0.5)
if (dominator == 0) Double.NaN else numerator / (dominator * 1.0)
}
餘弦相似性
def cosineSimilarity(v1: DoubleMatrix, v2: DoubleMatrix): Double = {
require(x.length == y.length, s"SimilarityAlgorithms:Array length do not match: Len(v1)=${x.length} and Len(v2)" +
s"=${y.length}.")
v1.dot(v2) / (v1.norm2() * v2.norm2())
}
def cosineSimilarity(v1: Vector, v2: Vector): Double = {
require(v1.size == v2.size, s"SimilarityAlgorithms:Vector dimensions do not match: Dim(v1)=${v1.size} and Dim(v2)" +
s"=${v2.size}.")
val x = v1.toArray
val y = v2.toArray
cosineSimilarity(x, y)
}
def cosineSimilarity(x: Array[Double], y: Array[Double]): Double = {
require(x.length == y.length, s"SimilarityAlgorithms:Array length do not match: Len(x)=${x.length} and Len(y)" +
s"=${y.length}.")
val member = x.zip(y).map(d => d._1 * d._2).sum
val temp1 = math.sqrt(x.map(math.pow(_, 2)).sum)
val temp2 = math.sqrt(y.map(math.pow(_, 2)).sum)
val denominator = temp1 * temp2
if (denominator == 0) Double.NaN else member / (denominator * 1.0)
}
修正餘弦相似性
def adjustedCosineSimJblas(x: DoubleMatrix, y: DoubleMatrix): Double = {
require(x.length == y.length, s"SimilarityAlgorithms:DoubleMatrix length do not match: Len(x)=${x.length} and Len(y)" +
s"=${y.length}.")
val avg = (x.sum() + y.sum()) / (x.length + y.length)
val v1 = x.sub(avg)
val v2 = y.sub(avg)
v1.dot(v2) / (v1.norm2() * v2.norm2())
}
def adjustedCosineSimJblas(x: Array[Double], y: Array[Double]): Double = {
require(x.length == y.length, s"SimilarityAlgorithms:Array length do not match: Len(x)=${x.length} and Len(y)" +
s"=${y.length}.")
val v1 = new DoubleMatrix(x)
val v2 = new DoubleMatrix(y)
adjustedCosineSimJblas(v1, v2)
}
def adjustedCosineSimilarity(v1: Vector, v2: Vector): Double = {
require(v1.size == v2.size, s"SimilarityAlgorithms:Vector dimensions do not match: Dim(v1)=${v1.size} and Dim(v2)" +
s"=${v2.size}.")
val x = v1.toArray
val y = v2.toArray
adjustedCosineSimilarity(x, y)
}
def adjustedCosineSimilarity(x: Array[Double], y: Array[Double]): Double = {
require(x.length == y.length, s"SimilarityAlgorithms:Array length do not match: Len(x)=${x.length} and Len(y)" +
s"=${y.length}.")
val avg = (x.sum + y.sum) / (x.length + y.length)
val member = x.map(_ - avg).zip(y.map(_ - avg)).map(d => d._1 * d._2).sum
val temp1 = math.sqrt(x.map(num => math.pow(num - avg, 2)).sum)
val temp2 = math.sqrt(y.map(num => math.pow(num - avg, 2)).sum)
val denominator = temp1 * temp2
if (denominator == 0) Double.NaN else member / (denominator * 1.0)
}
如果在實際業務處理中有相關需求,可以根據實際場景對上述程式碼進行優化或轉換。當然,很多演算法框架提供的一些演算法封裝了這些相似度演算法,底層還是依賴於這個集合,可以幫助你有更好的理解。比如Spark MLlib在KMeans演算法中實現,歐氏距離在底部計算。