機器學習專案實戰----泰坦尼克號獲救預測(二)
阿新 • • 發佈：2019-08-12
四、特徵重要性衡量
通過上面可以發現準確率有小幅提升，但是似乎得到的結果還是不太理想。我們可以發現模型似乎優化的差不多了，使用的特徵似乎也已經使用完了。準確率已經達到了瓶頸，但是如果我們還想提高精度的話，還是要回到最原始的資料集裡面。對分類器的結果最大的影響還是輸入的資料本身。接下來採用的方法一般是從原始的資料集裡面構造出新的特徵。新增特徵，家庭成員數和名字長度。
# Generating a familysize column titanic["FamilySize"] = titanic["SibSp"] + titanic["Parch"] # The .apply method generates a new series titanic["NameLength"] = titanic["Name"].apply(lambda x: len(x))
提取名字(名字裡麵包含稱呼，如小姐，女士，先生等等)，這些稱呼也是有可能對結果產生影響的。
import re
# A function to get the title from a name.
def get_title(name):
# Use a regular expression to search for a title.
# Titles always consist of capital and lowercase letters, and end with a period.
title_search = re.search(' ([A-Za-z]+)\.', name)
# If the title exists, extract and return it.
if title_search:
return title_search.group(1)
return ""
# Get all the titles and print how often each one occurs.
titles = titanic["Name"].apply(get_title)
print(pandas.value_counts(titles))
# Map each title to an integer. Some titles are very rare, and are compressed into the same codes as other titles.
title_mapping = {
"Mr": 1,
"Miss": 2,
"Mrs": 3,
"Master": 4,
"Dr": 5,
"Rev": 6,
"Major": 7,
"Col": 7,
"Mlle": 8,
"Mme": 8,
"Don": 9,
"Lady": 10,
"Countess": 10,
"Jonkheer": 10,
"Sir": 9,
"Capt": 7,
"Ms": 2
}
for k, v in title_mapping.items():
titles[titles == k] = v
# Verify that we converted everything.
# 驗證我們是否轉換了所有內容
print(pandas.value_counts(titles))
# Add in the title column.
titanic["Title"] = titles
得到的結果，發現前三個稱呼佔據資料集的一大半，毫無疑問，這個特徵對結果也是有較大影響的。
Mr 517 Miss 182 Mrs 125 Master 40 Dr 7 Rev 6 Major 2 Mlle 2 Col 2 Sir 1 Mme 1 Lady 1 Countess 1 Capt 1 Ms 1 Don 1 Jonkheer 1 Name: Name, dtype: int64 1 517 2 183 3 125 4 40 5 7 6 6 7 5 10 3 8 3 9 2 Name: Name, dtype: int64
通過前面的步驟發現特徵有點太多了，我們可以通過特徵的重要性來篩選出哪些特徵比較重要，而隨機森林的好處就是特徵重要性衡量。
特徵重要性解釋：在機器學習的訓練過程中，對於多個特徵來說，假如要對其中某一個特徵來衡量它的重要性，我們就不用這個特徵的資料來進行訓練，而是把這個特徵裡面的資料全部替換為噪音資料，假如得到的準確率沒有太大的變化，那就說明這個特徵其實不那麼重要，如果得到的準確率相差太大的話，說明這個特徵很重要。其他特徵的重要衡量以此類推。
import numpy as np
from sklearn.feature_selection import SelectKBest, f_classif # 選擇最好特徵
import matplotlib.pyplot as plt
predictors = [
"Pclass", "Sex", "Age", "SibSp", "Parch", "Fare", "Embarked", "FamilySize",
"Title", "NameLength"
]
# Perform feature selection
# 執行特徵選擇
selector = SelectKBest(f_classif, k=5)
selector.fit(titanic[predictors], titanic["Survived"])
# Get the raw p-values for each feature, and transform from p-values into scores
scores = -np.log10(selector.pvalues_)
# Plot the scores. See how "Pclass", "Sex", "Title", and "Fare" are the best?
plt.bar(range(len(predictors)), scores)
plt.xticks(range(len(predictors)), predictors, rotation='vertical')
plt.show()
# Pick only the four best features.
# 只選擇4個最好的特徵
predictors = ["Pclass", "Sex", "Fare", "Title"]
alg = RandomForestClassifier(random_state=1,
n_estimators=50,
min_samples_split=8,
min_samples_leaf=4)
得到的結果為：
上圖就是特徵重要性的一個柱狀圖，發現Age等一些特徵好像影響不大，和剛開始的假設有較大出入，那麼這些沒用的特徵就可以刪除掉，只保留有用的特徵即可。
五、整合演算法
使用整合演算法來提升準確率
from sklearn.ensemble import GradientBoostingClassifier
import numpy as np
# The algorithms we want to ensemble.
# We're using the more linear predictors for the logistic regression, and everything with the gradient boosting classifier.
algorithms = [
[GradientBoostingClassifier(random_state=1, n_estimators=25, max_depth=3), ["Pclass", "Sex", "Age", "Fare", "Embarked", "FamilySize", "Title",]],
[LogisticRegression(random_state=1,solver='liblinear'), ["Pclass", "Sex", "Fare", "FamilySize", "Title", "Age", "Embarked"]]
]
# Initialize the cross validation folds
kf = KFold(n_splits=3,shuffle=False, random_state=1)
predictions = []
for train, test in kf.split(titanic):
train_target = titanic["Survived"].iloc[train]
full_test_predictions = []
# Make predictions for each algorithm on each fold
for alg, predictors in algorithms:
# Fit the algorithm on the training data.
alg.fit(titanic[predictors].iloc[train,:], train_target)
# Select and predict on the test fold.
# The .astype(float) is necessary to convert the dataframe to all floats and avoid an sklearn error.
test_predictions = alg.predict_proba(titanic[predictors].iloc[test,:].astype(float))[:,1]
full_test_predictions.append(test_predictions)
# Use a simple ensembling scheme -- just average the predictions to get the final classification.
test_predictions = (full_test_predictions[0] + full_test_predictions[1]) / 2 # 兩個分類器的平均結果
# Any value over .5 is assumed to be a 1 prediction, and below .5 is a 0 prediction.
test_predictions[test_predictions <= .5] = 0
test_predictions[test_predictions > .5] = 1
predictions.append(test_predictions)
# Put all the predictions together into one array.
# 將所有的預測放在一個數組中
predictions = np.concatenate(predictions, axis=0)
# Compute accuracy by comparing to the training data.
accuracy = sum(predictions == titanic["Survived"]) / len(predictions)
print(accuracy)
得到的準確率為：
0.8215488215488216
接下來用測試資料集來進行預測(注意：在測試資料集裡面沒有"Survived"這一列，所以我們得不到測試結果的準確率，只能進行預測)
titles = titanic_test["Name"].apply(get_title)
# We're adding the Dona title to the mapping, because it's in the test set, but not the training set
title_mapping = {
"Mr": 1,
"Miss": 2,
"Mrs": 3,
"Master": 4,
"Dr": 5,
"Rev": 6,
"Major": 7,
"Col": 7,
"Mlle": 8,
"Mme": 8,
"Don": 9,
"Lady": 10,
"Countess": 10,
"Jonkheer": 10,
"Sir": 9,
"Capt": 7,
"Ms": 2,
"Dona": 10
}
for k, v in title_mapping.items():
titles[titles == k] = v
titanic_test["Title"] = titles
# Check the counts of each unique title.
print(pandas.value_counts(titanic_test["Title"]))
# Now, we add the family size column.
titanic_test["FamilySize"] = titanic_test["SibSp"] + titanic_test["Parch"]
得到測試資料集裡面Name裡面稱呼的次數：
1 240 2 79 3 72 4 21 7 2 6 2 10 1 5 1 Name: Title, dtype: int64
最終對測試資料集裡面的乘客能否獲救進行預測
predictors = [
"Pclass", "Sex", "Age", "Fare", "Embarked", "FamilySize", "Title"
]
algorithms = [
[
GradientBoostingClassifier(random_state=1,
n_estimators=25,
max_depth=3), predictors
],
[
LogisticRegression(random_state=1, solver='liblinear'),
["Pclass", "Sex", "Fare", "FamilySize", "Title", "Age", "Embarked"]
]
]
full_predictions = []
for alg, predictors in algorithms:
# Fit the algorithm using the full training data.
alg.fit(titanic[predictors], titanic["Survived"])
# Predict using the test dataset. We have to convert all the columns to floats to avoid an error.
predictions = alg.predict_proba(
titanic_test[predictors].astype(float))[:, 1]
predictions[predictions <= .5] = 0
predictions[predictions > .5] = 1
full_predictions.append(predictions)
# The gradient boosting classifier generates better predictions, so we weight it higher.
# predictions = (full_predictions[0] * 3 + full_predictions[1]) / 4
predictions
得到的結果(1表示能夠獲救，0表示不能被獲救)：
array([0., 0., 0., 0., 1., 0., 1., 0., 1., 0., 0., 0., 1., 0., 1., 1., 0.,
0., 1., 1., 0., 0., 1., 0., 1., 0., 1., 0., 0., 0., 0., 0., 0., 1.,
0., 0., 1., 1., 0., 0., 0., 0., 0., 1., 1., 0., 0., 0., 1., 0., 0.,
0., 1., 1., 0., 0., 0., 0., 0., 1., 0., 0., 0., 1., 1., 1., 1., 0.,
0., 1., 1., 0., 1., 0., 1., 1., 0., 1., 0., 1., 0., 0., 0., 0., 0.,
0., 1., 1., 1., 1., 1., 0., 1., 0., 0., 0., 1., 0., 1., 0., 1., 0.,
0., 0., 1., 0., 0., 0., 0., 0., 0., 1., 1., 1., 1., 0., 0., 1., 0.,
1., 1., 0., 1., 0., 0., 1., 0., 1., 0., 0., 0., 1., 0., 0., 0., 0.,
0., 0., 1., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0.,
0., 0., 0., 1., 1., 0., 1., 1., 0., 1., 0., 0., 1., 0., 0., 1., 1.,
0., 0., 0., 0., 0., 1., 1., 0., 1., 1., 0., 0., 1., 0., 1., 0., 1.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 1., 0., 1., 1., 0., 1., 1.,
0., 0., 1., 0., 1., 0., 0., 0., 0., 1., 0., 0., 1., 0., 1., 0., 1.,
0., 1., 0., 1., 1., 0., 1., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0.,
1., 1., 1., 1., 0., 0., 0., 0., 1., 0., 1., 1., 1., 0., 1., 0., 0.,
0., 0., 0., 1., 0., 0., 0., 1., 1., 0., 0., 0., 0., 1., 0., 0., 0.,
1., 1., 0., 1., 0., 0., 0., 0., 1., 0., 1., 1., 1., 0., 0., 0., 0.,
0., 0., 1., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 1., 1.,
0., 0., 0., 0., 0., 0., 0., 1., 1., 1., 0., 0., 0., 0., 0., 0., 0.,
0., 1., 0., 1., 0., 0., 0., 1., 0., 0., 1., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 1., 0., 1., 0., 1., 0., 1., 1., 0., 0., 0., 1., 0., 1.,
0., 0., 1., 0., 1., 1., 0., 1., 0., 0., 1., 1., 0., 0., 1., 0., 0.,
1., 1., 1., 0., 0., 0., 0., 0., 1., 1., 0., 1., 0., 0., 0., 0., 1.,
1., 0., 0., 0., 1., 0., 1., 0., 0., 1., 0., 1., 1., 0., 0., 0., 0.,
1., 1., 1., 1., 1., 0., 1., 0., 0., 0.])
六、總結
首先考慮資料集裡面的所有特徵，儘可能提取出來對結果有影響的一些資訊。然後缺失值的處理，字元資料的對映，機器學習演算法的改變，模型引數的優化，最後使用整合演算法提升準確率。還包括對資料集的特徵重要性的衡量和篩選。
&n