Pandas快速教程(5)-缺失值處理
缺失值的產生有很多原因,在Pandas中,使用 NaN 來代表缺失值.
本文將從缺失值的檢測,填充,刪除,插入.替換幾個方面來介紹pandas中對於缺失值的處理.
一.缺失值的檢測
df2 Out[7]: first secend third fourth a 6.0 8.0 3.0 6.0 b 2.0 5.0 9.0 8.0 c 4.0 4.0 1.0 0.0 d 6.0 2.0 2.0 6.0 e 2.0 0.0 8.0 9.0 f NaN NaN NaN NaN
不管是Series還是DataFrame,Pandas 中提供了 isna()和notna()兩個方法來檢測缺失值.
pd.isna(df2) Out[8]: first secend third fourth a False False False False b False False False False c False False False False d False False False False e False False False False f True True True True pd.notna(df2) Out[9]: first secend third fourth a True True True True b True True True True c True True True True d True True True True e True True True True f False False False False df2['first'].isna() Out[12]: a False b False c False d False e False f True Name: first, dtype: bool
需要特別提出的一點.在python中,None==None是成立的,但在pandas中,np.nan和np.nan是不相等的.
None==None
Out[13]: True
np.nan==np.nan
Out[14]: False基於此特性,對於缺失值的比較判斷資訊就會產生誤導.因為實際上nan和nan相比較用於返回的是False.比如:
df2.loc['f'] == np.nan
Out[15]:
first False
secend False
third False
fourth False
Name: f, dtype: bool在時間序列中,缺失值的表示不再是NaN,而是NaT(not a time).
df2['time']=pd.Timestamp('20180930')
df2
Out[17]:
first secend third fourth time
a 6.0 8.0 3.0 6.0 2018-09-30
b 2.0 5.0 9.0 8.0 2018-09-30
c 4.0 4.0 1.0 0.0 2018-09-30
d 6.0 2.0 2.0 6.0 2018-09-30
e 2.0 0.0 8.0 9.0 2018-09-30
f NaN NaN NaN NaN 2018-09-30
df2.loc[['d','e','f'],'time']=np.nan
df2
Out[19]:
first secend third fourth time
a 6.0 8.0 3.0 6.0 2018-09-30
b 2.0 5.0 9.0 8.0 2018-09-30
c 4.0 4.0 1.0 0.0 2018-09-30
d 6.0 2.0 2.0 6.0 NaT
e 2.0 0.0 8.0 9.0 NaT
f NaN NaN NaN NaN NaT從上面時間序列的例子,我們可以發現,缺失值的顯示方式是和資料型別相關的.在Pandas中,數字型別的資料的缺失值是NaN,時間序列是NaT,object型別可以時None或者NaN,關鍵在該缺失值被賦於哪個符號顯示.
s=pd.Series(['a','b','c'])
s
Out[21]:
0 a
1 b
2 c
dtype: object
s[0]=np.nan
s[1]=None
s
Out[24]:
0 NaN
1 None
2 c
dtype: object由於資料自動對齊功能的存在,在Pandas中,缺失值的傳播很多情況下都因為涉及到數學計算.
在0.22版本的以後的Pandas中,規範了缺失值的計算準則.
在求和,求均值等描述性統計方法計算中,NaN預設視為為0
df
Out[39]:
first secend third fourth
a 6 8 NaN NaN
b 2 5 NaN NaN
c 4 4 1.0 0.0
d 6 2 2.0 6.0
e 2 0 8.0 9.0
df.sum()
Out[40]:
first 20.0
secend 19.0
third 11.0
fourth 15.0
dtype: float64
df.mean(1)
Out[43]:
a 7.00
b 3.50
c 2.25
d 4.00
e 4.75
dtype: float64在求累積和或者累計積時,採用跳過缺失值的方式處理(skipna=True/False)
df.cumsum()
Out[44]:
first secend third fourth
a 6.0 8.0 NaN NaN
b 8.0 13.0 NaN NaN
c 12.0 17.0 1.0 0.0
d 18.0 19.0 3.0 6.0
e 20.0 19.0 11.0 15.0
df.cumprod()
Out[45]:
first secend third fourth
a 6.0 8.0 NaN NaN
b 12.0 40.0 NaN NaN
c 48.0 160.0 1.0 0.0
d 288.0 320.0 2.0 0.0
e 576.0 0.0 16.0 0.0全為NaN值的物件之和(sum) 為0,但prod為1.
s1=pd.Series([np.nan,np.nan])
s1.sum()
Out[47]: 0.0
s2=pd.Series([np.nan,np.nan])
s2.prod()
Out[49]: 1.0二.缺失值的刪除
資料清洗中,經常選擇刪除缺失值.pandas中提供了dropna()函式來刪除缺失值.
dropna(axis=0, how='any'/'all', thresh=None, subset=None, inplace=False)
axis:表示以x或y 軸為方向
how:表示採用any或者all的哪種方式來刪除
thresh:表示最少包含多少個非NaN值,同樣與axis結合使用
df
Out[54]:
first secend third fourth
a 6.0 8.0 NaN NaN
b 2.0 5.0 NaN NaN
c 4.0 4.0 1.0 0.0
d NaN NaN NaN NaN
e 2.0 0.0 8.0 9.0
df.dropna(axis=0,how='all')
Out[55]:
first secend third fourth
a 6.0 8.0 NaN NaN
b 2.0 5.0 NaN NaN
c 4.0 4.0 1.0 0.0
e 2.0 0.0 8.0 9.0
df.dropna(axis=0,how='any')
Out[56]:
first secend third fourth
c 4.0 4.0 1.0 0.0
e 2.0 0.0 8.0 9.0
df.dropna(axis=1,how='any')
Out[57]:
Empty DataFrame
Columns: []
Index: [a, b, c, d, e]
df.dropna(axis=1,how='all')
Out[58]:
first secend third fourth
a 6.0 8.0 NaN NaN
b 2.0 5.0 NaN NaN
c 4.0 4.0 1.0 0.0
d NaN NaN NaN NaN
e 2.0 0.0 8.0 9.0
df.dropna(thresh=3,axis=1)
Out[81]:
first secend
a 6.0 8.0
b 2.0 5.0
c 4.0 4.0
d NaN NaN
e 2.0 0.0
df.dropna(thresh=3,axis=0)
Out[82]:
first secend third fourth
c 4.0 4.0 1.0 0.0
e 2.0 0.0 8.0 9.0三.缺失值的填充
fillna(value=None, method=None, axis=None, inplace=False, limit=None, **kwargs)
value:表示要填充的值,可以用字典來表示精確的填充(比如哪列/行填充什麼值)
method:pad/ffill表示向前填充, bfill/backfill表示向後填充
asix:表示填充方向
limit: 表示每個單元(列或者行)的最大的填充數量
df
Out[84]:
first secend third fourth
a 6.0 8.0 NaN NaN
b 2.0 5.0 NaN NaN
c 4.0 4.0 1.0 0.0
d NaN NaN NaN NaN
e 2.0 0.0 8.0 9.0
df.fillna(100)
Out[85]:
first secend third fourth
a 6.0 8.0 100.0 100.0
b 2.0 5.0 100.0 100.0
c 4.0 4.0 1.0 0.0
d 100.0 100.0 100.0 100.0
e 2.0 0.0 8.0 9.0
df.fillna({'third':100,'fourth':99,'first':10,'secend':1})
Out[86]:
first secend third fourth
a 6.0 8.0 100.0 99.0
b 2.0 5.0 100.0 99.0
c 4.0 4.0 1.0 0.0
d 10.0 1.0 100.0 99.0
e 2.0 0.0 8.0 9.0
df.fillna(method='ffill')
Out[87]:
first secend third fourth
a 6.0 8.0 NaN NaN
b 2.0 5.0 NaN NaN
c 4.0 4.0 1.0 0.0
d 4.0 4.0 1.0 0.0
e 2.0 0.0 8.0 9.0
df.fillna(method='bfill')
Out[88]:
first secend third fourth
a 6.0 8.0 1.0 0.0
b 2.0 5.0 1.0 0.0
c 4.0 4.0 1.0 0.0
d 2.0 0.0 8.0 9.0
e 2.0 0.0 8.0 9.0
df.fillna(method='ffill',axis=1)
Out[91]:
first secend third fourth
a 6.0 8.0 8.0 8.0
b 2.0 5.0 5.0 5.0
c 4.0 4.0 1.0 0.0
d NaN NaN NaN NaN
e 2.0 0.0 8.0 9.0
df.fillna(100,limit=1)
Out[94]:
first secend third fourth
a 6.0 8.0 100.0 100.0
b 2.0 5.0 NaN NaN
c 4.0 4.0 1.0 0.0
d 100.0 100.0 NaN NaN
e 2.0 0.0 8.0 9.0類似的還有一種便捷的填充方式,就是用where方法:
df.where(df.notna(),100)
Out[97]:
first secend third fourth
a 6.0 8.0 100.0 100.0
b 2.0 5.0 100.0 100.0
c 4.0 4.0 1.0 0.0
d 100.0 100.0 100.0 100.0
e 2.0 0.0 8.0 9.0四.缺失值的替換
replace(to_replace=None, value=None, inplace=False, limit=None, regex=False, method='pad', axis=None)
to_replace:表示要被替換的元素或者表示式,可以使用字典表達精確替換
value:被替換的目標值
method:可以指定替換的方法(與fillna類似),但只能在Series上執行
regex:設定為True時,可以接受正則表示式的替換
df
Out[100]:
First Secend Third
0 a 1 1
1 b 2 2
2 c 3 a
3 d 4 b
df.replace('a',100)
Out[102]:
First Secend Third
0 100 1 1
1 b 2 2
2 c 3 100
3 d 4 b
df.replace({'First':['c','d'],'Secend':[3,4]},1000)
Out[104]:
First Secend Third
0 a 1 1
1 b 2 2
2 1000 1000 a
3 1000 1000 b
df['First'].replace(['b'],method='ffill')
Out[108]:
0 a
1 a
2 c
3 d
Name: First, dtype: object
df.replace(r'a|b|c|d','abc',regex=True)
Out[113]:
First Secend Third
0 abc 1 1
1 abc 2 2
2 abc 3 abc
3 abc 4 abc五.缺失值的插值法(interpolate)
簡單的刪除.填充.替換缺失資料會導致整體資料方差的變化,從而導致資料資訊量的變換.
插值法就是為了解決這一問題.
常用的有線性插值法,多項式插值法,樣條插值法等.
DataFrame.interpolate(method='linear', axis=0, limit=None, inplace=False, limit_direction='forward', limit_area=None, downcast=None, **kwargs)
預設採用的是線性插入法
該方法中最主要的引數就是method
指定不同的method方法,就可以用不同的演算法進行缺失值插入
常用的插入方法有:
{‘linear’, ‘time’, ‘index’, ‘values’, ‘nearest’, ‘zero’,‘slinear’, ‘quadratic’, ‘cubic’, ‘barycentric’, ‘krogh’, ‘polynomial’, ‘spline’, ‘piecewise_polynomial’, ‘from_derivatives’, ‘pchip’, ‘akima’}每種方法有不同的演算法和要求,詳情可以查閱官網API:
對該方法的簡單舉例:
df
Out[131]:
First Secend Third
0 2.0 1.0 1
1 4.0 2.0 2
2 NaN 3.0 a
3 8.0 NaN b
df.interpolate()
Out[130]:
First Secend Third
0 2.0 1.0 1
1 4.0 2.0 2
2 6.0 3.0 a
3 8.0 3.0 b小結
對於缺失值的處理,填充,刪除或者對指定的元素進行替換都比較簡單.
但插值法涉及到一些統計學的專業知識,稍微比較複雜,為了處理資料時保持數量資訊量的準確,需要多加研究才是.