前言

windows對於utf-8編碼的檔案自帶BOM，但是其他系統utf-8編碼預設不帶BOM。

這就造成在某些情況下字元解碼會出現問題，比如python自帶的json在讀取在window下編碼得來的utf-8檔案時，會報如下錯誤:

ValueError: No JSON object could be decoded

BOM

From Wikipedia, the free encyclopedia

BOM：The byte order mark (BOM) is a Unicode character, U+FEFF BYTE ORDER MARK (BOM), whose appearance as a magic number at the start of a text stream can signal several things to a program consuming the text:

• What byte order, or endianness, the text stream is stored in;
• The fact that the text stream is Unicode, to a high level of confidence;
• Which of several Unicode encodings that text stream is encoded as.

BOM use is optional, and, if used, should appear at the start of the text stream.

Unicode can be encoded as 8-bit, 16-bit, or 32-bit integers. For the 16- and 32-bit representations, a computer receiving text from arbitrary sources needs to know which byte order the integers are encoded in. Because the BOM itself is encoded in the same scheme as the rest of the document, but has a known value, the consumer of the text can examine these first few bytes to determine the encoding. The BOM thus gives the producer of the text a way to describe the text stream’s endianness to the consumer of the text without requiring some contract or metadata outside of the text stream itself.

Once the receiving computer has consumed the text stream, it is free to process the characters in its own native byte order and no longer needs the BOM. Hence the need for a BOM arises in the context of text interchange, rather than in text processing within a closed environment.

一句話總結：BOM對於utf-16和utf-32有用，對於utf-8沒啥大用。。所以能去掉就去掉好了。。

去掉BOM

方法1：

f = open("data","r")
s = f.read()
u = s.decode("utf-8-sig") # 得到一個不含BOM的unicode string
s = u.encode("utf-8")   # 將unicode轉換為utf-8
f.close()

方法2：

import codecs

f = open("data","r")
s = f.read()        
if s.startswith(codecs.BOM_UTF8):
    s = s[len(codecs.BOM_UTF8):]
f.close()