樸素貝葉斯

貝葉斯定理用來計算條件概率，即：

然後進行一種樸素(naive)的假設-每對特徵之間都相互獨立:

在給定的輸入中 P(x_1, \dots, x_n) 是一個常量，我們使用下面的分類規則:

可以使用最大後驗概率(Maximum A Posteriori, MAP) 來估計 P(y) 和 P(x_i | y) ; 前者是訓練集中類別 y 的相對頻率。

各種各樣的的樸素貝葉斯分類器的差異大部分來自於處理 P(x_i | y) 分佈時的所做的假設不同。

P(x_i | y)在有足夠多資料時，也可以從資料集中統計出來，或者使用各種分佈模型估計。

概率分佈

描述連續隨機變數概率分佈的函式稱為概率密度函式

描述離散隨機變數概率分佈的函式叫概率質量函式。典型的概率質量函式是多項式分佈函式。可用於文字分類或者垃圾郵件檢測。

這兩種分佈函式統稱為概率分佈函式。

伯努利高斯和多項式分佈類似，但是它明確地懲罰類 y 中沒有出現作為預測因子的特徵 i ，而多項分佈分佈樸素貝葉斯只是簡單地忽略沒出現的特徵。在文字分類的例子中，詞頻向量(word occurrence vectors)的 BernoulliNB 可能在一些資料集上可能表現得更好，特別是那些更短的文件。 如果時間允許，建議對兩個模型都進行評估。

程式碼：使用多項式分佈的樸素貝葉斯進行文字分類

from time
 
 import time
from sklearn.datasets import load_files
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix

# 步驟1：載入新聞語料文件
print("loading train dataset ..."
 
)
t = time()
# load file 專門用於載入分類的文件，每個分類一個單獨的目錄，目錄名就是類名
news_train = load_files('D:/專案/機器學習和機器視覺/scikit-learn機器學習-原始碼/code/datasets/mlcomp/379/train')
print("summary: {0} documents in {1} categories.".format(
    len(news_train.data), len(news_train.target_names)))
# news_train.target是長度為13180的一維向量，每個值代表相應文章的分類id
print('news_categories_names:\n{}, \nlen(target):{}, target:{}'.format(news_train.target_names,
                                                                       len(news_train.target), news_train.target))
print("done in {0} seconds\n".format(round(time() - t, 2)))

# 步驟2：將文件資料轉化為TF-IDF向量
print("vectorizing train dataset ...")
t = time()
vectorizer = TfidfVectorizer(encoding='latin-1')
X_train = vectorizer.fit_transform((d for d in news_train.data))
print("n_samples: %d, n_features: %d" % X_train.shape)
# X_train每一行代表一篇文件，每個成員表示一個詞的TF-IDF值，表示這個詞對這個文章的重要性。
# X_train的形狀是13180X130274
print("number of non-zero features in sample [{0}]: {1}".format(
    news_train.filenames[0], X_train[0].getnnz()))
print("done in {0} seconds\n".format(round(time() - t, 2)))

# 步驟3：使用多項式分佈的樸素貝葉斯演算法訓練
print("traning models ...".format(time() - t))
t = time()
y_train = news_train.target
clf = MultinomialNB(alpha=0.0001)
clf.fit(X_train, y_train)
train_score = clf.score(X_train, y_train)
print("train score: {0}".format(train_score))
print("done in {0} seconds\n".format(round(time() - t, 2)))

# 步驟4：載入測試資料集
print("loading test dataset ...")
t = time()
news_test = load_files('D:/專案/機器學習和機器視覺/scikit-learn機器學習-原始碼/code/datasets/mlcomp/379/test')
print("summary: {0} documents in {1} categories.".format(
    len(news_test.data), len(news_test.target_names)))
print("done in {0} seconds\n".format(round(time() - t, 2)))

# 步驟5:把測試資料集向量化
print("vectorizing test dataset ...")
t = time()
# 注意這裡呼叫的是transform而非上面的fit_transform。因為上面已經把資料統計好了
X_test = vectorizer.transform((d for d in news_test.data))
y_test = news_test.target
print("n_samples: %d, n_features: %d" % X_test.shape)
print("number of non-zero features in sample [{0}]: {1}".format(
    news_test.filenames[0], X_test[0].getnnz()))
print("done in %fs\n" % (time() - t))

# 步驟6：使用測試資料集測試。測試第一篇文章
print("predict for {} ...".format(news_test.filenames[0]))
pred = clf.predict(X_test[0])
print("predict: {0} is in category {1}".format(
    news_test.filenames[0], news_test.target_names[pred[0]]))
print("actually: {0} is in category {1}\n".format(
    news_test.filenames[0], news_test.target_names[news_test.target[0]]))

# 步驟7：評估演算法的預測效果
print("predicting test dataset ...")
t = time()
pred = clf.predict(X_test)
print("done in %fs" % (time() - t))
print("classification report on test set for classifier:")
print(clf)
print(classification_report(y_test, pred,
                            target_names=news_test.target_names))

# 步驟8：生成混淆矩陣
cm = confusion_matrix(y_test, pred)
print("confusion matrix:")
print(cm)

輸出為：

loading train dataset ...
summary: 13180 documents in 20 categories.
news_categories_names:
['alt.atheism', 'comp.graphics', 'comp.os.ms-windows.misc', 'comp.sys.ibm.pc.hardware', 'comp.sys.mac.hardware', 'comp.windows.x', 'misc.forsale', 'rec.autos', 'rec.motorcycles', 'rec.sport.baseball', 'rec.sport.hockey', 'sci.crypt', 'sci.electronics', 'sci.med', 'sci.space', 'soc.religion.christian', 'talk.politics.guns', 'talk.politics.mideast', 'talk.politics.misc', 'talk.religion.misc'], 
len(target):13180, target:[18 13  1 ... 14 15  4]
done in 1.97 seconds

vectorizing train dataset ...
n_samples: 13180, n_features: 130274
number of non-zero features in sample [D:/專案/機器學習和機器視覺/scikit-learn機器學習-原始碼/code/datasets/mlcomp/379/train\talk.politics.misc\17860-178992]: 108
done in 5.35 seconds

traning models ...
train score: 0.9978755690440061
done in 0.37 seconds

loading test dataset ...
summary: 5648 documents in 20 categories.
done in 0.86 seconds

vectorizing test dataset ...
n_samples: 5648, n_features: 130274
number of non-zero features in sample [D:/專案/機器學習和機器視覺/scikit-learn機器學習-原始碼/code/datasets/mlcomp/379/test\rec.autos\7429-103268]: 61
done in 2.129905s

predict for D:/專案/機器學習和機器視覺/scikit-learn機器學習-原始碼/code/datasets/mlcomp/379/test\rec.autos\7429-103268 ...
predict: D:/專案/機器學習和機器視覺/scikit-learn機器學習-原始碼/code/datasets/mlcomp/379/test\rec.autos\7429-103268 is in category rec.autos
actually: D:/專案/機器學習和機器視覺/scikit-learn機器學習-原始碼/code/datasets/mlcomp/379/test\rec.autos\7429-103268 is in category rec.autos

predicting test dataset ...
done in 0.041888s
classification report on test set for classifier:
MultinomialNB(alpha=0.0001, class_prior=None, fit_prior=True)
                          precision    recall  f1-score   support

             alt.atheism       0.90      0.91      0.91       245
           comp.graphics       0.80      0.90      0.85       298
 comp.os.ms-windows.misc       0.82      0.79      0.80       292
comp.sys.ibm.pc.hardware       0.81      0.80      0.81       301
   comp.sys.mac.hardware       0.90      0.91      0.91       256
          comp.windows.x       0.88      0.88      0.88       297
            misc.forsale       0.87      0.81      0.84       290
               rec.autos       0.92      0.93      0.92       324
         rec.motorcycles       0.96      0.96      0.96       294
      rec.sport.baseball       0.97      0.94      0.96       315
        rec.sport.hockey       0.96      0.99      0.98       302
               sci.crypt       0.95      0.96      0.95       297
         sci.electronics       0.91      0.85      0.88       313
                 sci.med       0.96      0.96      0.96       277
               sci.space       0.94      0.97      0.96       305
  soc.religion.christian       0.93      0.96      0.94       293
      talk.politics.guns       0.91      0.96      0.93       246
   talk.politics.mideast       0.96      0.98      0.97       296
      talk.politics.misc       0.90      0.90      0.90       236
      talk.religion.misc       0.89      0.78      0.83       171

             avg / total       0.91      0.91      0.91      5648

confusion matrix:
[[224   0   0   0   0   0   0   0   0   0   0   0   0   0   2   5   0   0   1  13]
 [  1 267   5   5   2   8   1   1   0   0   0   2   3   2   1   0   0   0   0   0]
 [  1  13 230  24   4  10   5   0   0   0   0   1   2   1   0   0   0   0   1   0]
 [  0   9  21 242   7   2  10   1   0   0   1   1   7   0   0   0   0   0   0   0]
 [  0   1   5   5 233   2   2   2   1   0   0   3   1   0   1   0   0   0   0   0]
 [  0  20   6   3   1 260   0   0   0   2   0   1   0   0   2   0   2   0   0   0]
 [  0   2   5  12   3   1 235  10   2   3   1   0   7   0   2   0   2   1   4   0]
 [  0   1   0   0   1   0   8 300   4   1   0   0   1   2   3   0   2   0   1   0]
 [  0   1   0   0   0   2   2   3 283   0   0   0   1   0   0   0   0   0   1   1]
 [  0   1   1   0   1   2   1   2   0 297   8   1   0   1   0   0   0   0   0   0]
 [  0   0   0   0   0   0   0   0   2   2 298   0   0   0   0   0   0   0   0   0]
 [  0   1   2   0   0   1   1   0   0   0   0 284   2   1   0   0   2   1   2   0]
 [  0  11   3   5   4   2   4   5   1   1   0   4 266   1   4   0   1   0   1   0]
 [  1   1   0   1   0   2   1   0   0   0   0   0   1 266   2   1   0   0   1   0]
 [  0   3   0   0   1   1   0   0   0   0   0   1   0   1 296   0   1   0   1   0]
 [  3   1   0   1   0   0   0   0   0   0   1   0   0   2   1 280   0   1   1   2]
 [  1   0   2   0   0   0   0   0   1   0   0   0   0   0   0   0 236   1   4   1]
 [  1   0   0   0   0   1   0   0   0   0   0   0   0   0   0   3   0 290   1   0]
 [  2   1   0   0   1   1   0   1   0   0   0   0   0   0   0   1  10   7  212  0]
 [ 16   0   0   0   0   0   0   0   0   0   0   0   0   0   0  12   4   1   4 134]]