前言：

不小心看到一條留言：

然後就去該地址看了一下，這一看，順帶折騰了一天。

今天，就和大夥分享下折騰的感覺。

在該開源地址中，程式碼有C++和C#兩個版本，編碼的整體風格傾向與於C++。

主要的時間，花了在對於檢測無BOM的部分，順帶重溫了各種編碼的基礎。

建議在看此文之前，先了解下編碼、和BOM的概念。

有BOM的編碼檢測

對於一個檔案，或者位元組流，就是一堆二進位制：

如果傳輸的過程，有指定BOM，就是前面兩三個位元組是固定的255,254之類的，那麼解碼起來就很簡單了。

像之前IOHelper內部讀檔案的程式碼是這麼寫的：

 /// <summary>
 

        /// 先自動識別UTF8，否則歸到Default編碼讀取
        /// </summary>
        /// <returns></returns>
        public static string ReadAllText(string fileName)
        {
            return ReadAllText(fileName, DefaultEncoding);
        }
        public static string ReadAllText(string fileName, Encoding encoding)
        {
            
 
try
            {
                if (!File.Exists(fileName))
                {
                    return string.Empty;
                }
                Byte[] buff = null;
                lock (GetLockObj(fileName.Length))
                {
                    if (!File.Exists(fileName))//
 
多執行緒情況處理
                    {
                        return string.Empty;
                    }
                    buff = File.ReadAllBytes(fileName);
                }
                if (buff.Length == 0) { return ""; }
                if (buff[0] == 239 && buff[1] == 187 && buff[2] == 191)
                {
                    return Encoding.UTF8.GetString(buff, 3, buff.Length - 3);
                }
                else if (buff[0] == 255 && buff[1] == 254)
                {
                    return Encoding.Unicode.GetString(buff, 2, buff.Length - 2);
                }
                else if (buff[0] == 254 && buff[1] == 255)
                {
                    if (buff.Length > 3 && buff[2] == 0 && buff[3] == 0)
                    {
                        return Encoding.UTF32.GetString(buff, 4, buff.Length - 4);
                    }
                    return Encoding.BigEndianUnicode.GetString(buff, 2, buff.Length - 2);
                }
                return encoding.GetString(buff);
            }
            catch (Exception err)
            {
                Log.WriteLogToTxt(err);
            }
            return string.Empty;
        }

程式碼說白了，就是檢測BOM頭，然後識別編碼，用對應的編碼解碼。

測試的結果：

中文都能正確顯示。

windows下文字的另存為只有：ANSI、UTF8､Unicode(UTF16LE)、BigEndianUnicode（UTF16BE）。

這四種有BOM的都是輕鬆檢測了。

 那如果檔案或位元組沒有BOM頭呢？如果用預設的編碼，由有一定概率會亂碼。

無BOM的編碼檢測

如果一堆位元組流，沒有指定BOM，就要分析出編碼型別，還是挺有難度的。

這需要對各種編碼的規則有一定的熟悉度。

先看看網友給出的Github上的 原始原始碼：

public Encoding DetectEncoding(byte[] buffer, int size)
        {
            // First check if we have a BOM and return that if so
            Encoding encoding = CheckBom(buffer, size);
            if (encoding != Encoding.None)
            {
                return encoding;
            }

            // Now check for valid UTF8
            encoding = CheckUtf8(buffer, size);
            if (encoding != Encoding.None)
            {
                return encoding;
            }

            // Now try UTF16 
            encoding = CheckUtf16NewlineChars(buffer, size);
            if (encoding != Encoding.None)
            {
                return encoding;
            }

            encoding = CheckUtf16Ascii(buffer, size);
            if (encoding != Encoding.None)
            {
                return encoding;
            }

            // ANSI or None (binary) then
            if (!DoesContainNulls(buffer, size))
            {
                return Encoding.Ansi;
            }

            // Found a null, return based on the preference in null_suggests_binary_
            return _nullSuggestsBinary ? Encoding.None : Encoding.Ansi;
        }

程式碼流程（和內涵）翻譯下來是這樣的：

1､檢測BOM頭，這個很Easy。

2､檢測UTF8編碼（這個還是很有創意的），如果編碼的規則完全符合UTF8,則認為是UTF8。

3､檢測位元組中是否有換行符（根據換行符中的0的位置，區分是Utf16的BE大尾還是LE小尾）。

這個概率要看位元組抽樣的長度，帶不帶換行符。

4､檢測位元組中，單偶數出現的0的概率，設定了一個期望值來預判（對於中文而言，基本沒用），大概是老外寫的，只根據英文情況分析的概率。

5､檢測位元組中，有沒有出現0，如果沒有，返回系統預設編碼（不同系統環境編碼是不同的）。

首先，不得不說，原作者還是有一定想法的。

雖然程式碼中除了UTF8按規則寫的分析外，其它的都無法代入中文環境裡通過。

但至少思路上，就能得到不少啟發。

於是，坑了我大半天，進行重寫，改造，代入中文環境測試。

無BOM程式碼檢測的改造過程：

改造後的程式碼流程是這樣的：

public Encoding DetectWithoutBom(byte[] buffer, int size)
        {
            // Now check for valid UTF8
            Encoding encoding = CheckUtf8(buffer, size);
            if (encoding != Encoding.None)
            {
                return encoding;
            }

            // ANSI or None (binary) then 一個零都沒有情況。
            if (!ContainsZero(buffer, size))
            {
                CheckChinese(buffer, size);
                return Encoding.Ansi;
            }

            // Now try UTF16  按尋找換行字元先進行判斷
            encoding = CheckByNewLineChar(buffer, size);
            if (encoding != Encoding.None)
            {
                return encoding;
            }

            // 沒辦法了，只能按0出現的次數比率，做大體的預判
            encoding = CheckByZeroNumPercent(buffer, size);
            if (encoding != Encoding.None)
            {
                return encoding;
            }

            // Found a null, return based on the preference in null_suggests_binary_
            return Encoding.None;
        }

用中文解釋流程是這樣的：

1､UTF8編碼的檢測規則，這個是通用的有效，可以保留。

2､調整順序：先檢測位元組有沒有0位元組，若無，補一個是否中文的編碼的檢測（GB2312､GBK、Big5)。

這個後續有點用。

3､檢測換行符：增加UTF-32編碼的檢測（原來的思路只有UTF16)。

4､預判概率：改造成同時適應中文環境。

測試的結果是這樣的：

A、純中文的：

該測試下，對於BigEndianUnicode的會產生亂碼。

B、非純中文的

一切編碼正常通用。

改進後的完整原始碼：

using System;
using System.Collections.Generic;
using System.IO;
using System.Text;

namespace CYQ.Data.Tool
{
    internal static class IOHelper
    {
        internal static Encoding DefaultEncoding = Encoding.Default;

        private static List<object> tenObj = new List<object>(10);
        private static List<object> TenObj
        {
            get
            {
                if (tenObj.Count == 0)
                {
                    for (int i = 0; i < 10; i++)
                    {
                        tenObj.Add(new object());
                    }
                }
                return tenObj;
            }
        }
        private static object GetLockObj(int length)
        {
            int i = length % 9;
            return TenObj[i];
        }
        /// <summary>
        /// 先自動識別UTF8，否則歸到Default編碼讀取
        /// </summary>
        /// <returns></returns>
        public static string ReadAllText(string fileName)
        {
            return ReadAllText(fileName, DefaultEncoding);
        }
        public static string ReadAllText(string fileName, Encoding encoding)
        {
            try
            {
                if (!File.Exists(fileName))
                {
                    return string.Empty;
                }
                Byte[] buff = null;
                lock (GetLockObj(fileName.Length))
                {
                    if (!File.Exists(fileName))//多執行緒情況處理
                    {
                        return string.Empty;
                    }
                    buff = File.ReadAllBytes(fileName);
                    return BytesToText(buff, encoding);
                }

            }
            catch (Exception err)
            {
                Log.WriteLogToTxt(err);
            }
            return string.Empty;
        }
        public static bool Write(string fileName, string text)
        {
            return Save(fileName, text, false, DefaultEncoding, true);
        }
        public static bool Write(string fileName, string text, Encoding encode)
        {
            return Save(fileName, text, false, encode, true);
        }
        public static bool Append(string fileName, string text)
        {
            return Save(fileName, text, true, true);
        }

        internal static bool Save(string fileName, string text, bool isAppend, bool writeLogOnError)
        {
            return Save(fileName, text, true, DefaultEncoding, writeLogOnError);
        }
        internal static bool Save(string fileName, string text, bool isAppend, Encoding encode, bool writeLogOnError)
        {
            try
            {
                string folder = Path.GetDirectoryName(fileName);
                if (!Directory.Exists(folder))
                {
                    Directory.CreateDirectory(folder);
                }

                lock (GetLockObj(fileName.Length))
                {
                    using (StreamWriter writer = new StreamWriter(fileName, isAppend, encode))
                    {
                        writer.Write(text);
                    }
                }
                return true;
            }
            catch (Exception err)
            {
                if (writeLogOnError)
                {
                    Log.WriteLogToTxt(err);
                }
                else
                {
                    Error.Throw("IOHelper.Save() : " + err.Message);
                }
            }
            return false;
        }

        internal static bool Delete(string fileName)
        {
            try
            {
                if (File.Exists(fileName))
                {
                    lock (GetLockObj(fileName.Length))
                    {
                        if (File.Exists(fileName))
                        {
                            File.Delete(fileName);
                            return true;
                        }
                    }
                }
            }
            catch
            {

            }
            return false;
        }

        public static bool IsLastFileWriteTimeChanged(string fileName, ref DateTime compareTimeUtc)
        {
            bool isChanged = false;
            IOInfo info = new IOInfo(fileName);
            if (info.Exists && info.LastWriteTimeUtc != compareTimeUtc)
            {
                isChanged = true;
                compareTimeUtc = info.LastWriteTimeUtc;
            }
            return isChanged;
        }
        public static string BytesToText(byte[] buff, Encoding encoding)
        {
            if (buff.Length == 0) { return ""; }
            //if (buff[0] == 239 && buff[1] == 187 && buff[2] == 191)
            //{
            //    return Encoding.UTF8.GetString(buff, 3, buff.Length - 3);
            //}
            //else if (buff[0] == 255 && buff[1] == 254)
            //{
            //    return Encoding.Unicode.GetString(buff, 2, buff.Length - 2);
            //}
            //else if (buff[0] == 254 && buff[1] == 255)
            //{
            //    if (buff.Length > 3 && buff[2] == 0 && buff[3] == 0)
            //    {
            //        return Encoding.UTF32.GetString(buff, 4, buff.Length - 4);
            //    }
            //    return Encoding.BigEndianUnicode.GetString(buff, 2, buff.Length - 2);
            //}
            //else
            //{
            TextEncodingDetect detect = new TextEncodingDetect();

            //檢測Bom
            switch (detect.DetectWithBom(buff))
            {
                case TextEncodingDetect.Encoding.Utf8Bom:
                    return Encoding.UTF8.GetString(buff, 3, buff.Length - 3);
                case TextEncodingDetect.Encoding.UnicodeBom:
                    return Encoding.Unicode.GetString(buff, 2, buff.Length - 2);
                case TextEncodingDetect.Encoding.BigEndianUnicodeBom:
                    return Encoding.BigEndianUnicode.GetString(buff, 2, buff.Length - 2);
                case TextEncodingDetect.Encoding.Utf32Bom:
                    return Encoding.UTF32.GetString(buff, 4, buff.Length - 4);
            }
            if (encoding != DefaultEncoding && encoding != Encoding.ASCII)//自定義設定編碼，優先處理。
            {
                return encoding.GetString(buff);
            }
            switch (detect.DetectWithoutBom(buff, buff.Length > 1000 ? 1000 : buff.Length))//自動檢測。
            {

                case TextEncodingDetect.Encoding.Utf8Nobom:
                    return Encoding.UTF8.GetString(buff);

                case TextEncodingDetect.Encoding.UnicodeNoBom:
                    return Encoding.Unicode.GetString(buff);

                case TextEncodingDetect.Encoding.BigEndianUnicodeNoBom:
                    return Encoding.BigEndianUnicode.GetString(buff);

                case TextEncodingDetect.Encoding.Utf32NoBom:
                    return Encoding.UTF32.GetString(buff);

                case TextEncodingDetect.Encoding.Ansi:
                    if (IsChineseEncoding(DefaultEncoding) && !IsChineseEncoding(encoding))
                    {
                        if (detect.IsChinese)
                        {
                            return Encoding.GetEncoding("gbk").GetString(buff);
                        }
                        else//非中文時，預設選一個。
                        {
                            return Encoding.Unicode.GetString(buff);
                        }
                    }
                    else
                    {
                        return encoding.GetString(buff);
                    }

                case TextEncodingDetect.Encoding.Ascii:
                    return Encoding.ASCII.GetString(buff);

                default:
                    return encoding.GetString(buff);
            }
            // }
        }
        private static bool IsChineseEncoding(Encoding encoding)
        {
            return encoding == Encoding.GetEncoding("gb2312") || encoding == Encoding.GetEncoding("gbk") || encoding == Encoding.GetEncoding("big5");
        }
    }
    internal class IOInfo : FileSystemInfo
    {
        public IOInfo(string fileName)
        {
            base.FullPath = fileName;
        }
        public override void Delete()
        {
        }

        public override bool Exists
        {
            get
            {
                return File.Exists(base.FullPath);
            }
        }

        public override string Name
        {
            get
            {
                return null;
            }
        }
    }
    /// <summary>
    /// 位元組文字編碼檢測
    /// </summary>
    internal class TextEncodingDetect
    {
        private readonly byte[] _UTF8Bom =
        {
            0xEF,
            0xBB,
            0xBF
        };
        //utf16le _UnicodeBom
        private readonly byte[] _UTF16LeBom =
        {
            0xFF,
            0xFE
        };

        //utf16be _BigUnicodeBom
        private readonly byte[] _UTF16BeBom =
        {
            0xFE,
            0xFF
        };

        //utf-32le
        private readonly byte[] _UTF32LeBom =
        {
            0xFF,
            0xFE,
            0x00,
            0x00
        };
        //utf-32Be
        //private readonly byte[] _UTF32BeBom =
        //{
        //    0x00,
        //    0x00,
        //    0xFE,
        //    0xFF
        //};
        /// <summary>
        /// 是否中文
        /// </summary>
        public bool IsChinese = false;

        public enum Encoding
        {
            None, // Unknown or binary
            Ansi, // 0-255
            Ascii, // 0-127
            Utf8Bom, // UTF8 with BOM
            Utf8Nobom, // UTF8 without BOM
            UnicodeBom, // UTF16 LE with BOM
            UnicodeNoBom, // UTF16 LE without BOM
            BigEndianUnicodeBom, // UTF16-BE with BOM
            BigEndianUnicodeNoBom, // UTF16-BE without BOM

            Utf32Bom,//UTF-32LE with BOM
            Utf32NoBom //UTF-32 without BOM

        }

        public Encoding DetectWithBom(byte[] buffer)
        {
            if (buffer != null)
            {
                int size = buffer.Length;
                // Check for BOM
                if (size >= 2 && buffer[0] == _UTF16LeBom[0] && buffer[1] == _UTF16LeBom[1])
                {
                    return Encoding.UnicodeBom;
                }

                if (size >= 2 && buffer[0] == _UTF16BeBom[0] && buffer[1] == _UTF16BeBom[1])
                {
                    if (size >= 4 && buffer[2] == _UTF32LeBom[2] && buffer[3] == _UTF32LeBom[3])
                    {
                        return Encoding.Utf32Bom;
                    }
                    return Encoding.BigEndianUnicodeBom;
                }

                if (size >= 3 && buffer[0] == _UTF8Bom[0] && buffer[1] == _UTF8Bom[1] && buffer[2] == _UTF8Bom[2])
                {
                    return Encoding.Utf8Bom;
                }
            }
            return Encoding.None;
        }

        /// <summary>
        ///     Automatically detects the Encoding type of a given byte buffer.
        /// </summary>
        /// <param name="buffer">The byte buffer.</param>
        /// <param name="size">The size of the byte buffer.</param>
        /// <returns>The Encoding type or Encoding.None if unknown.</returns>
        public Encoding DetectWithoutBom(byte[] buffer, int size)
        {
            // Now check for valid UTF8
            Encoding encoding = CheckUtf8(buffer, size);
            if (encoding != Encoding.None)
            {
                return encoding;
            }

            // ANSI or None (binary) then 一個零都沒有情況。
            if (!ContainsZero(buffer, size))
            {
                CheckChinese(buffer, size);
                return Encoding.Ansi;
            }

            // Now try UTF16  按尋找換行字元先進行判斷
            encoding = CheckByNewLineChar(buffer, size);
            if (encoding != Encoding.None)
            {
                return encoding;
            }

            // 沒辦法了，只能按0出現的次數比率，做大體的預判
            encoding = CheckByZeroNumPercent(buffer, size);
            if (encoding != Encoding.None)
            {
                return encoding;
            }

            // Found a null, return based on the preference in null_suggests_binary_
            return Encoding.None;
        }

        /// <summary>
        ///     Checks if a buffer contains text that looks like utf16 by scanning for
        ///     newline chars that would be present even in non-english text.
        ///     以檢測換行符標識來判斷。
        /// </summary>
        /// <param name="buffer">The byte buffer.</param>
        /// <param name="size">The size of the byte buffer.</param>
        /// <returns>Encoding.none, Encoding.Utf16LeNoBom or Encoding.Utf16BeNoBom.</returns>
        private static Encoding CheckByNewLineChar(byte[] buffer, int size)
        {
            if (size < 2)
            {
                return Encoding.None;
            }

            // Reduce size by 1 so we don't need to worry about bounds checking for pairs of bytes
            size--;

            int le16 = 0;
            int be16 = 0;
            int le32 = 0;//檢測是否utf32le。
            int zeroCount = 0;//utf32le 每4位後面多數是0
            uint pos = 0;
            while (pos < size)
            {
                byte ch1 = buffer[pos++];
                byte ch2 = buffer[pos++];

                if (ch1 == 0)
                {
                    if (ch2 == 0x0a || ch2 == 0x0d)//\r \t 換行檢測。
                    {
                        ++be16;
                    }
                }
                if (ch2 == 0)
                {
                    zeroCount++;
                    if (ch1 == 0x0a || ch1 == 0x0d)
                    {
                        ++le16;
                        if (pos + 1 <= size && buffer[pos] == 0 && buffer[pos + 1] == 0)
                        {
                            ++le32;
                        }

                    }
                }

                // If we are getting both LE and BE control chars then this file is not utf16
                if (le16 > 0 && be16 > 0)
                {
                    return Encoding.None;
                }
            }

            if (le16 > 0)
            {
                if (le16 == le32 && buffer.Length % 4 == 0)
                {
                    return Encoding.Utf32NoBom;
                }
                return Encoding.UnicodeNoBom;
            }
            else if (be16 > 0)
            {
                return Encoding.BigEndianUnicodeNoBom;
            }
            else if (buffer.Length % 4 == 0 && zeroCount >= buffer.Length / 4)
            {
                return Encoding.Utf32NoBom;
            }
            return Encoding.None;
        }

        /// <summary>
        /// Checks if a buffer contains any nulls. Used to check for binary vs text data.
        /// </summary>
        /// <param name="buffer">The byte buffer.</param>
        /// <param name="size">The size of the byte buffer.</param>
        private static bool ContainsZero(byte[] buffer, int size)
        {
            uint pos = 0;
            while (pos < size)
            {
                if (buffer[pos++] == 0)
                {
                    return true;
                }
            }

            return false;
        }

        /// <summary>
        ///     Checks if a buffer contains text that looks like utf16. This is done based
        ///     on the use of nulls which in ASCII/script like text can be useful to identify.
        ///     按照一定的空0數的概率來預測。
        /// </summary>
        /// <param name="buffer">The byte buffer.</param>
        /// <param name="size">The size of the byte buffer.</param>
        /// <returns>Encoding.none, Encoding.Utf16LeNoBom or Encoding.Utf16BeNoBom.</returns>
        private Encoding CheckByZeroNumPercent(byte[] buffer, int size)
        {
            //單數
            int oddZeroCount = 0;
            //雙數
            int evenZeroCount = 0;

            // Get even nulls
            uint pos = 0;
            while (pos < size)
            {
                if (buffer[pos] == 0)
                {
                    evenZeroCount++;
                }

                pos += 2;
            }

            // Get odd nulls
            pos = 1;
            while (pos < size)
            {
                if (buffer[pos] == 0)
                {
                    oddZeroCount++;
                }

                pos += 2;
            }

            double evenZeroPercent = evenZeroCount * 2.0 / size;
            double oddZeroPercent = oddZeroCount * 2.0 / size;

            // Lots of odd nulls, low number of even nulls 這裡的條件做了修改
            if (evenZeroPercent < 0.1 && oddZeroPercent > 0)
            {
                return Encoding.UnicodeNoBom;
            }

            // Lots of even nulls, low number of odd nulls 這裡的條件也做了修改
            if (oddZeroPercent < 0.1 && evenZeroPercent > 0)
            {
                return Encoding.BigEndianUnicodeNoBom;
            }

            // Don't know
            return Encoding.None;
        }

        /// <summary>
        ///     Checks if a buffer contains valid utf8.
        ///     以UTF8 的位元組範圍來檢測。
        /// </summary>
        /// <param name="buffer">The byte buffer.</param>
        /// <param name="size">The size of the byte buffer.</param>
        /// <returns>
        ///     Encoding type of Encoding.None (invalid UTF8), Encoding.Utf8NoBom (valid utf8 multibyte strings) or
        ///     Encoding.ASCII (data in 0.127 range).
        /// </returns>
        /// <returns>2</returns&g