python時間序列-----前半篇---python進行資料分析
阿新 • • 發佈：2018-12-16
目錄
簡介
時間序列是一種重要的格式化資料形式，在多個時間點觀察或測量到的任何事務都可以形成一段時間序列。時間序列主要有以下幾種：
時間戳-timestamp，特定時刻
固定時期-period 如2007年1月，或2010年全年
時間間隔-interval 有起始和結束時間戳表示。時期可以看成間隔的特例。
日期和時間資料型別及工具
datetime以毫秒德形式儲存日期和時間。
>>> from datetime import datetime >>> now = datetime.now() >>> now datetime.datetime(2018, 10, 21, 12, 12, 46, 472000) >>> now.year,now.month,now.day (2018, 10, 21)
datetime.timedelta表示兩個datetime物件之間的時間差。
>>> delta = datetime(2001,1,5) - datetime(2002,6,7)
>>> delta
datetime.timedelta(-518)
>>> delta.days
-518給datetime物件加上一個或減去多個timedelta物件，就會產生一個新物件：
>>> from datetime import timedelta >>> start = datetime(2011,1,7) >>> start + timedelta(12) datetime.datetime(2011, 1, 19, 0, 0)
datetime模組的資料型別表
字串和datetime的互相轉換
datetime.datetime(2011, 1, 19, 0, 0)
>>> stamp = datetime(2011,1,3)
>>> str(stamp)
'2011-01-03 00:00:00'
>>> stamp.strftime('%Y-%m-%d')
'2011-01-03'
datetime.strptime是通過已知格式進行日期解析的最佳方式
>>> datetime.strptime(value,'%Y-%m-%d') datetime.datetime(2011, 1, 3, 0, 0) >>> [datetime.strptime(x,'%m%d%Y') for x in datestrs] >>> [datetime.strptime(x,'%m/%d/%Y') for x in datestrs] [datetime.datetime(2011, 7, 6, 0, 0), datetime.datetime(2011, 8, 6, 0, 0)]
>>> from dateutil.parser import parse
>>> parse('2011-01-03')
datetime.datetime(2011, 1, 3, 0, 0)
>>> parse('Jan 31,1997 10:45 PM')
datetime.datetime(1997, 1, 31, 22, 45)
>>> parse('6/12/2011',dayfirst=True)
datetime.datetime(2011, 12, 6, 0, 0)
>>> datestrs
['7/6/2011', '8/6/2011']
>>> import pandas as pd
Backend TkAgg is interactive backend. Turning interactive mode on.
>>> from pandas import DataFrame,Series
>>> pd.to_datetime(datestrs)
DatetimeIndex(['2011-07-06', '2011-08-06'], dtype='datetime64[ns]', freq=None)
>>> idx = pd.to_datetime(datestrs + [None])
>>> idx
DatetimeIndex(['2011-07-06', '2011-08-06', 'NaT'], dtype='datetime64[ns]', freq=None)
>>> idx[2]
NaT
>>> pd.isnull(idx)
array([False, False, True])
datetime物件還有一些特定於當前環境的格式化選項。
時間序列基礎
pandas最基本的時間序列型別就是以時間戳為索引的Series:
>>> from datetime import datetime
>>> dates = [datetime(1900,1,1),datetime(1932,1,5),datetime(1964,1,1),datetime(1980,1,1),datetime(1999,1,1),datetime(2006,1,1),]
>>> import numpy as np
>>> ts = Series(np.random.randn(6),index=dates)
>>> ts
1900-01-01 -1.578267
1932-01-05 0.072027
1964-01-01 0.984831
1980-01-01 0.910140
1999-01-01 -0.917276
2006-01-01 -0.824397
dtype: float64
>>> ts + ts[::2]
1900-01-01 -3.156535
1932-01-05 NaN
1964-01-01 1.969661
1980-01-01 NaN
1999-01-01 -1.834552
2006-01-01 NaN
dtype: float64
>>> ts.index.dtype
dtype('<M8[ns]')
>>> stamp = ts.index.dtype
>>> stamp
dtype('<M8[ns]')
索引、選取、子集構造
>>> stamp = ts.index[2]
>>> ts[stamp]
0.984830525622642
>>>
>>> ts
1900-01-01 -1.578267
1932-01-05 0.072027
1964-01-01 0.984831
1980-01-01 0.910140
1999-01-01 -0.917276
2006-01-01 -0.824397
dtype: float64
>>> stamp
Timestamp('1964-01-01 00:00:00')
>>> ts['1/1/1900']
-1.578267392616083>>> long_ts = Series(np.random.randn(1000),index=pd.date_range('1/1/2000',periods=1000))
>>> long_ts
2000-01-01 1.189341
2000-01-02 0.158855
2000-01-03 0.735436
2000-01-04 -0.043352
2000-01-05 1.371487
...
>>> long_ts['2001'][:5]
2001-01-01 0.792097
2001-01-02 -0.631906
2001-01-03 -0.770665
2001-01-04 -0.702957
2001-01-05 -1.270367
Freq: D, dtype: float64
>>> ts[datetime(1932, 1, 5):]
1932-01-05 0.072027
1964-01-01 0.984831
1980-01-01 0.910140
1999-01-01 -0.917276
2006-01-01 -0.824397
dtype: float64
>>> ts['1950':'2200']
1964-01-01 0.984831
1980-01-01 0.910140
1999-01-01 -0.917276
2006-01-01 -0.824397
dtype: float64
>>> ts.truncate(after='1980')
1900-01-01 -1.578267
1932-01-05 0.072027
1964-01-01 0.984831
1980-01-01 0.910140
dtype: float64帶有重複索引的時間序列
>>> dates = pd.DatetimeIndex(['1/1/2000','1/2/2000','1/2/2000','1/2/2000','1/3/2000'])
>>> dup_ts = Series(np.arange(5),index=dates)
>>> dup_ts
2000-01-01 0
2000-01-02 1
2000-01-02 2
2000-01-02 3
2000-01-03 4
dtype: int32
>>> dup_ts.index.is_unique
False
>>> dup_ts['1/3/2000']
4
>>> dup_ts['1/2/2000']
2000-01-02 1
2000-01-02 2
2000-01-02 3
dtype: int32
>>> grouped = dup_ts.groupby(level=0)
>>> grouped.mean()
2000-01-01 0
2000-01-02 2
2000-01-03 4
dtype: int32
>>> grouped.count()
2000-01-01 1
2000-01-02 3
2000-01-03 1
dtype: int64
日期的範圍、頻率及移動
pandas中的時間序列一般被認為是不規則的，也就是說，他們沒有固定的頻率。
但是，有時候需要一某種固定的頻率進行分析。下面開始練習：
>>> ts.resample('D').count()
1900-01-01 1
1900-01-02 0
1900-01-03 0
...
2005-12-27 0
2005-12-28 0
2005-12-29 0
2005-12-30 0
2005-12-31 0
2006-01-01 1
Freq: D, Length: 38717, dtype: int64
生成日期範圍
>>> index = pd.date_range('4/1/2012','6/1/2012')
>>> index
DatetimeIndex(['2012-04-01', '2012-04-02', '2012-04-03', '2012-04-04',
'2012-04-05', '2012-04-06', '2012-04-07', '2012-04-08',
'2012-04-09', '2012-04-10', '2012-04-11', '2012-04-12',
'2012-04-13', '2012-04-14', '2012-04-15', '2012-04-16',
'2012-04-17', '2012-04-18', '2012-04-19', '2012-04-20',
'2012-04-21', '2012-04-22', '2012-04-23', '2012-04-24',
'2012-04-25', '2012-04-26', '2012-04-27', '2012-04-28',
'2012-04-29', '2012-04-30', '2012-05-01', '2012-05-02',
'2012-05-03', '2012-05-04', '2012-05-05', '2012-05-06',
'2012-05-07', '2012-05-08', '2012-05-09', '2012-05-10',
'2012-05-11', '2012-05-12', '2012-05-13', '2012-05-14',
'2012-05-15', '2012-05-16', '2012-05-17', '2012-05-18',
'2012-05-19', '2012-05-20', '2012-05-21', '2012-05-22',
'2012-05-23', '2012-05-24', '2012-05-25', '2012-05-26',
'2012-05-27', '2012-05-28', '2012-05-29', '2012-05-30',
'2012-05-31', '2012-06-01'],
dtype='datetime64[ns]', freq='D')20日頻率
>>> index = pd.date_range('4/1/2012','6/1/2012',periods=20)
>>> index
DatetimeIndex([ '2012-04-01 00:00:00',
'2012-04-04 05:03:09.473684224',
'2012-04-07 10:06:18.947368448',
'2012-04-10 15:09:28.421052672',
'2012-04-13 20:12:37.894736896',
'2012-04-17 01:15:47.368421120',
'2012-04-20 06:18:56.842105344',
'2012-04-23 11:22:06.315789568',
'2012-04-26 16:25:15.789473792',
'2012-04-29 21:28:25.263158016',
'2012-05-03 02:31:34.736841984',
'2012-05-06 07:34:44.210526208',
'2012-05-09 12:37:53.684210432',
'2012-05-12 17:41:03.157894656',
'2012-05-15 22:44:12.631578880',
'2012-05-19 03:47:22.105263104',
'2012-05-22 08:50:31.578947328',
'2012-05-25 13:53:41.052631552',
'2012-05-28 18:56:50.526315776',
'2012-06-01 00:00:00'],
dtype='datetime64[ns]', freq=None)每月最後工作日 頻率
>>> pd.date_range('1/1/2000','12/1/2000',freq='BM')
DatetimeIndex(['2000-01-31', '2000-02-29', '2000-03-31', '2000-04-28',
'2000-05-31', '2000-06-30', '2000-07-31', '2000-08-31',
'2000-09-29', '2000-10-31', '2000-11-30'],
dtype='datetime64[ns]', freq='BM')
規範化時間資訊
>>> pd.date_range('5/2/2012 12:56:31',periods=5)
DatetimeIndex(['2012-05-02 12:56:31', '2012-05-03 12:56:31',
'2012-05-04 12:56:31', '2012-05-05 12:56:31',
'2012-05-06 12:56:31'],
dtype='datetime64[ns]', freq='D')
>>> pd.date_range('5/2/2012 12:56:31',periods=5,normalize=True)
DatetimeIndex(['2012-05-02', '2012-05-03', '2012-05-04', '2012-05-05',
'2012-05-06'],
dtype='datetime64[ns]', freq='D')
頻率和日期偏移量
>>> from pandas.tseries.offsets import Hour,Minute
>>> hour = Hour()
>>> hour
<Hour>
>>> four_hours = Hour(4)
>>> four_hours
<4 * Hours>
>>> pd.date_range('1/1/2000','1/3/2000 23:59',freq='4H')
DatetimeIndex(['2000-01-01 00:00:00', '2000-01-01 04:00:00',
'2000-01-01 08:00:00', '2000-01-01 12:00:00',
'2000-01-01 16:00:00', '2000-01-01 20:00:00',
'2000-01-02 00:00:00', '2000-01-02 04:00:00',
'2000-01-02 08:00:00', '2000-01-02 12:00:00',
'2000-01-02 16:00:00', '2000-01-02 20:00:00',
'2000-01-03 00:00:00', '2000-01-03 04:00:00',
'2000-01-03 08:00:00', '2000-01-03 12:00:00',
'2000-01-03 16:00:00', '2000-01-03 20:00:00'],
dtype='datetime64[ns]', freq='4H')
Hour(2) + Minute(30)
<150 * Minutes>
pd.date_range('1/1/2000',periods=10,freq='1h30min')
DatetimeIndex(['2000-01-01 00:00:00', '2000-01-01 01:30:00',
'2000-01-01 03:00:00', '2000-01-01 04:30:00',
'2000-01-01 06:00:00', '2000-01-01 07:30:00',
'2000-01-01 09:00:00', '2000-01-01 10:30:00',
'2000-01-01 12:00:00', '2000-01-01 13:30:00'],
dtype='datetime64[ns]', freq='90T')
WOM日期
Week of Month 是一種非常實用的頻率類，它能是你獲得諸如‘每月第三個星期五之類‘的日期
>>> rng = pd.date_range('1/1/2012','9/1/2012',freq='WOM-3FRI')
>>> list(rng)
[Timestamp('2012-01-20 00:00:00', freq='WOM-3FRI'),
Timestamp('2012-02-17 00:00:00', freq='WOM-3FRI'),
Timestamp('2012-03-16 00:00:00', freq='WOM-3FRI'),
Timestamp('2012-04-20 00:00:00', freq='WOM-3FRI'),
Timestamp('2012-05-18 00:00:00', freq='WOM-3FRI'),
Timestamp('2012-06-15 00:00:00', freq='WOM-3FRI'),
Timestamp('2012-07-20 00:00:00', freq='WOM-3FRI'),
Timestamp('2012-08-17 00:00:00', freq='WOM-3FRI')]
移動（超前和滯後）資料
>>> ts = Series(np.random.randn(4),index=pd.date_range('1/1/2000',periods=4,freq='M'))
>>> ts
2000-01-31 -0.349239
2000-02-29 -0.409400
2000-03-31 -0.468117
2000-04-30 -0.628233
Freq: M, dtype: float64
>>> ts.shift(2)
2000-01-31 NaN
2000-02-29 NaN
2000-03-31 -0.349239
2000-04-30 -0.409400
Freq: M, dtype: float64
>>> ts.shift(-2)
2000-01-31 -0.468117
2000-02-29 -0.628233
2000-03-31 NaN
2000-04-30 NaN
Freq: M, dtype: float64
>>> ts.shift(2,freq='M')
2000-03-31 -0.349239
2000-04-30 -0.409400
2000-05-31 -0.468117
2000-06-30 -0.628233
Freq: M, dtype: float64其他頻率
>>> ts.shift(3,freq='D')
2000-02-03 -0.349239
2000-03-03 -0.409400
2000-04-03 -0.468117
2000-05-03 -0.628233
dtype: float64
>>> ts.shift(1,freq='3D')
2000-02-03 -0.349239
2000-03-03 -0.409400
2000-04-03 -0.468117
2000-05-03 -0.628233
dtype: float64
>>> ts.shift(1,freq='90T')
2000-01-31 01:30:00 -0.349239
2000-02-29 01:30:00 -0.409400
2000-03-31 01:30:00 -0.468117
2000-04-30 01:30:00 -0.628233
Freq: M, dtype: float64
通過偏移量對日期進行位移
>>> from pandas.tseries.offsets import Day,MonthEnd
>>> now = datetime(2011,11,17)
>>> now + 3 * Day()
Timestamp('2011-11-20 00:00:00')
>>> now + MonthEnd()
Timestamp('2011-11-30 00:00:00')
>>> now + MonthEnd(2)
Timestamp('2011-12-31 00:00:00')
>>> offset = MonthEnd()
>>> offset.rollforward(now)
Timestamp('2011-11-30 00:00:00')
>>> offset.rollback(now)
Timestamp('2011-10-31 00:00:00')
>>> ts = Series(np.random.randn(20),index=pd.date_range('1/15/2000',periods=20,freq='4d'))
>>> ts.groupby(offset.rollforward).mean()
2000-01-31 -0.637592
2000-02-29 -0.302083
2000-03-31 -0.531805
dtype: float64
>>> ts.resample('M',how='mean')
<string>:1: FutureWarning: how in .resample() is deprecated
the new syntax is .resample(...).mean()
2000-01-31 -0.637592
2000-02-29 -0.302083
2000-03-31 -0.531805
Freq: M, dtype: float64
時區處理
>>> import pytz
>>> pytz.common_timezones[-5:]
['US/Eastern', 'US/Hawaii', 'US/Mountain', 'US/Pacific', 'UTC']
>>> tz = pytz.timezone('US/Eastern')
>>> tz
<DstTzInfo 'US/Eastern' LMT-1 day, 19:04:00 STD>
本地化和轉換
>>> import pytz
>>> pytz.common_timezones[-5:]
['US/Eastern', 'US/Hawaii', 'US/Mountain', 'US/Pacific', 'UTC']
>>> tz = pytz.timezone('US/Eastern')
>>> tz
<DstTzInfo 'US/Eastern' LMT-1 day, 19:04:00 STD>
>>> rng = pd.date_range('3/9/2012 9:30',periods=6,freq='D')
>>> ts = Series(np.random.randn(len(rng)),index=rng)
>>> print(ts.index.tz)
None
>>> pd.date_range('3/9/2012 9:30',periods=10,freq='D',tz='UTC')
DatetimeIndex(['2012-03-09 09:30:00+00:00', '2012-03-10 09:30:00+00:00',
'2012-03-11 09:30:00+00:00', '2012-03-12 09:30:00+00:00',
'2012-03-13 09:30:00+00:00', '2012-03-14 09:30:00+00:00',
'2012-03-15 09:30:00+00:00', '2012-03-16 09:30:00+00:00',
'2012-03-17 09:30:00+00:00', '2012-03-18 09:30:00+00:00'],
dtype='datetime64[ns, UTC]', freq='D')
本地化轉換
>>> ts_utc = ts.tz_localize('UTC')
>>> ts_utc
2012-03-09 09:30:00+00:00 -0.900121
2012-03-10 09:30:00+00:00 0.629368
2012-03-11 09:30:00+00:00 -0.141687
2012-03-12 09:30:00+00:00 0.299048
2012-03-13 09:30:00+00:00 0.033719
2012-03-14 09:30:00+00:00 -0.185427
Freq: D, dtype: float64
>>> ts_utc.index
DatetimeIndex(['2012-03-09 09:30:00+00:00', '2012-03-10 09:30:00+00:00',
'2012-03-11 09:30:00+00:00', '2012-03-12 09:30:00+00:00',
'2012-03-13 09:30:00+00:00', '2012-03-14 09:30:00+00:00'],
dtype='datetime64[ns, UTC]', freq='D')
轉換時區
>>> ts_utc.tz_convert('US/Eastern')
2012-03-09 04:30:00-05:00 -0.900121
2012-03-10 04:30:00-05:00 0.629368
2012-03-11 05:30:00-04:00 -0.141687
2012-03-12 05:30:00-04:00 0.299048
2012-03-13 05:30:00-04:00 0.033719
2012-03-14 05:30:00-04:00 -0.185427
Freq: D, dtype: float64
操作時區意識型Timestamp物件
Timestamp物件也能被從單純型 naive 本地化為時區意識型 time zone-aware
>>> stamp = pd.Timestamp('2011-03-12 04:00')
>>> stamp_utc = stamp.tz_localize('utc')
>>> stamp_utc.tz_convert('US/Eastern')
Timestamp('2011-03-11 23:00:00-0500', tz='US/Eastern')
建立timpstamp時，還可以傳入一個時區資訊：
>>> stamp_moscow = pd.Timestamp('2011-03-12 04:00',tz='Europe/Moscow')
>>> stamp_moscow
Timestamp('2011-03-12 04:00:00+0300', tz='Europe/Moscow')時區意識型Timestamp物件在內部儲存了一個UTC時間戳值（自UNIX紀元1970.1.1算起的納秒數）
>>> stamp_utc.value
1299902400000000000L
>>> stamp_utc.tz_convert('US/Eastern').value
1299902400000000000L
當使用pandas的DateOffset物件執行時間算數運算時，運算過程自動關注是否存在夏令時轉變期
>>> stamp = pd.Timestamp('2011-03-12 04:00')
>>> stamp_utc = stamp.tz_localize('utc')
>>> stamp_utc.tz_convert('US/Eastern')
Timestamp('2011-03-11 23:00:00-0500', tz='US/Eastern')
>>> stamp_moscow = pd.Timestamp('2011-03-12 04:00',tz='Europe/Moscow')
>>> stamp_moscow
Timestamp('2011-03-12 04:00:00+0300', tz='Europe/Moscow')
>>> stamp_utc.value
1299902400000000000L
>>> stamp_utc.tz_convert('US/Eastern').value
1299902400000000000L
>>> from pandas.tseries.offsets import Hour
>>> stamp = pd.Timestamp('2012-03-12 01:30',tz='US/Eastern')
>>> stamp
Timestamp('2012-03-12 01:30:00-0400', tz='US/Eastern')
>>> stamp+Hour()
Timestamp('2012-03-12 02:30:00-0400', tz='US/Eastern')
>>> stamp = pd.Timestamp('2012-11-04 00:30',tz='US/Eastern')
>>> stamp
Timestamp('2012-11-04 00:30:00-0400', tz='US/Eastern')
>>> stamp + 2*Hour()
Timestamp('2012-11-04 01:30:00-0500', tz='US/Eastern')
不同時區之間的運算
如果兩個時間序列的時區不同，再將它們合併時，最終結果就是UTC.由於時間戳是以UTC儲存的，所以這個運算很簡單，不需要任何轉換。
>>> rng = pd.date_range('3/7/2012 9:30',periods=10,freq='B')
>>> ts = Series(np.random.randn(len(rng)),index=rng)
>>> ts
2012-03-07 09:30:00 0.629019
2012-03-08 09:30:00 -0.708171
2012-03-09 09:30:00 -0.715323
2012-03-12 09:30:00 -0.844492
2012-03-13 09:30:00 -1.743470
2012-03-14 09:30:00 0.074501
2012-03-15 09:30:00 -0.551764
2012-03-16 09:30:00 0.141351
2012-03-19 09:30:00 0.125488
2012-03-20 09:30:00 0.888226
Freq: B, dtype: float64
>>> ts1 = ts[:7].tz_localize('Europe/London')
>>> ts2 = ts1[2:].tz_convert('Europe/Moscow')
>>> ts1
2012-03-07 09:30:00+00:00 0.629019
2012-03-08 09:30:00+00:00 -0.708171
2012-03-09 09:30:00+00:00 -0.715323
2012-03-12 09:30:00+00:00 -0.844492
2012-03-13 09:30:00+00:00 -1.743470
2012-03-14 09:30:00+00:00 0.074501
2012-03-15 09:30:00+00:00 -0.551764
Freq: B, dtype: float64
>>> ts2
2012-03-09 13:30:00+04:00 -0.715323
2012-03-12 13:30:00+04:00 -0.844492
2012-03-13 13:30:00+04:00 -1.743470
2012-03-14 13:30:00+04:00 0.074501
2012-03-15 13:30:00+04:00 -0.551764
Freq: B, dtype: float64
>>> result = ts1 + ts2
>>> result.index
DatetimeIndex(['2012-03-07 09:30:00+00:00', '2012-03-08 09:30:00+00:00',
'2012-03-09 09:30:00+00:00', '2012-03-12 09:30:00+00:00',
'2012-03-13 09:30:00+00:00', '2012-03-14 09:30:00+00:00',
'2012-03-15 09:30:00+00:00'],
dtype='datetime64[ns, UTC]', freq='B')