如何使用Python快速高效地統計出大檔案的總行數, 下面是一些實現方法和效能的比較。

• 1.readline讀所有行
  使用readlines方法讀取所有行:

def readline_count(file_name):
    return len(open(file_name).readlines())

• 2.依次讀取每行
  依次讀取檔案每行內容進行計數:

def simple_count(file_name):
    lines = 0
    for _ in open(file_name):
        lines += 1
    return lines

• 3.sum計數
  使用sum
  函式計數:

def sum_count(file_name):
    return sum(1 for _ in open(file_name))

• 4.enumerate列舉計數:

def enumerate_count(file_name):
    with open(file_name) as f:
        for count, _ in enumerate(f, 1):
            pass
    return count

• 5.buff count
  每次讀取固定大小,然後統計行數:

def buff_count(file_name):
    with open(file_name, 'rb') as f:
        count = 0
        buf_size = 1024 * 1024
        buf = f.read(buf_size)
        while buf:
            count += buf.count(b'\n')
            buf = f.read(buf_size)
        return count
 

• 6.wc count
  呼叫使用wc命令計算行:

def wc_count(file_name):
    import subprocess
    out = subprocess.getoutput("wc -l %s" % file_name)
    return int(out.split()[0])

• 7.partial count
  在buff_count基礎上引入partial:

def partial_count(file_name):
    from functools import partial
    buffer = 1024 * 1024
    with open(file_name) as f:
        return sum(x.count('\n') for x in iter(partial(f.read, buffer), ''))
 

• 8.iter count
  在buff_count基礎上引入itertools模組 :

def iter_count(file_name):
    from itertools import (takewhile, repeat)
    buffer = 1024 * 1024
    with open(file_name) as f:
        buf_gen = takewhile(lambda x: x, (f.read(buffer) for _ in repeat(None)))
        return sum(buf.count('\n') for buf in buf_gen)

下面是在我本機 4c8g python3.6的環境下,分別測試100m、500m、1g、10g大小檔案執行的時間，單位秒：