問題與背景

訂單導出需要將交易數據通過報表的形式導出並提供下載給商家，供商家發貨、對賬等。由於交易的場景非常多，承接多個業務（微商城、零售單店、零售連鎖版、餐飲），訂單類型很多，新老報表的字段覆蓋交易、支付、會員、優惠、發貨、退款、特定業務等，合計多達120個。每次代碼變更（尤其是比較大的改動），如果想要手工驗證指定時間段內的絕大多數場景下絕大多數訂單類型的所有字段都沒有問題，在前端頁面點擊下載報表，然後手工對比，將是非常大的工作量。因此，迫切需要一個自動化的對比工具，對比變更分支與線上分支的導出報表，找出和分析差異，修復問題。

為什麽選擇要在預發而不在QA進行呢？ 因為訂單導出的準確性不僅包含導出和下載功能（20%），更重要的是數據的準確性(80%)。而QA的數據不一定準確，且涵蓋面不廣，不準確的數據會導致錯誤的對比結果，對變更的影響造成很大的幹擾，延誤時間。 因此，這裏直接選擇用線上的數據來做對比，有時也會意外發現線上數據的一點問題。

整體思路

先做出一個假定：如果master分支的線上邏輯是沒有問題的，那麽預發的branch分支導出的結果，應該跟線上保持一致； 如果線上的邏輯有問題，那麽預發 branch 分支導出的結果，應該有部分跟線上不一致，且不一致的地方根據推斷應該僅跟改動部分有關。 分兩種情況：

• 系統代碼優化與重構：邏輯沒有改動，那麽預發和線上的導出結果應該完全一致。如果有不一致的情況發生，那麽需要分析不一致的原因，決定是否可以接受和取舍。
• 業務邏輯優化：比如在某個場景下，“訂單類型”字段原來輸出“分銷買家訂單”，現在需要輸出“分銷買家訂單/拼團訂單”，那麽導出結果的不一致應該限於“訂單類型”。當然，如果有其他報表字段的輸出也依賴於“訂單類型”字段，那麽可能其他字段也會不一致，這時候需要進一步分析。

整體思路如下：

• 使用 Python 來完成該任務，因為 Python 非常簡潔實用 ，適合做質量要求不是非常高的接口測試工具；
• 分別往預發和線上發送相同的請求，然後通過導出ID拿到預發請求的文件和線上請求的文件，然後讀取並逐字段對比，打印出差異；
• 將對比結果保存在 /tmp/cmp_export.txt ， 發送郵件保存。
• 不同店鋪的不同業務配置的導出測試用例通過一個單獨的配置文件來給出，測試用例配置與請求測試功能分離。

這裏使用了閉包的技術來配置化地構造大量測試用例。參閱：Python使用閉包結合配置自動生成函數。

源代碼

test.py : 主測試程序。 只要運行 python test.py 即可。然後看看是否有 diff 。如果沒有 diff ，那就說明預發和線上導出結果一致； 如果有 diff ，就需要仔細分析 diff ，找出原因並解決。

#!/usr/bin/env python
# -*- coding: utf-8 -*-
#-------------------------------------------------------------------------------
# Name:        test.py
# Purpose:     test if exports from pre is consistent with exports from production
# USAGE:       python test.py
# When:        before deploy to production
#                  STEP1: login in bc-pre-order-export0 and vim test.py, cases.py in your directory ,
#                             enter :set paste ,  copy this script and save ;
#                  STEP2: run python test.py
#
# Author:      qin.shuq
#
# Created:     12/22/2017
# Copyright:   (c) qin.shuq 2017
# Licence:     <your licence>
#-------------------------------------------------------------------------------
import requests
import os
import json
import time
import math
import urllib2
import traceback

import smtplib
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart
from email.header import Header


from cases import *

import sys
import codecs
import locale
sys.stdout = codecs.getwriter(locale.getpreferredencoding())(sys.stdout)

preUrl = 'http://pre-host:7001/api/general/export'
prodUrl = 'http://prod-host:7001/api/general/export'
queryUrl = 'http://prod-host:7001/api/general/queryRecords'

filedir = './files/'
resultfile = '/tmp/export_cmp.txt'

def extractFields(fieldsStr):
    return map(lambda x: x.strip(), fieldsStr.split(','))

templateIdFieldsMap = {     "1": extractFields("OrderNo,ExpressCompany,ExpressNo,OrderState,BuyerName,BuyerSex,BuyerProvince,BuyerCity,IsFans,ShouldPay,Postage,TotalPay,RealPay,BuyWay,SKU,SKUCode,GoodsCode,FeedBackInfo,ReceiverName,ReceiverProvince,ReceiverCity,ReceiverCounty,ReceiverDetailAddress,PostCode,ExpressWay,SelfFetchAddress,SelfFetcher,SelfFetchPhone,SelfFetchTime,Phone,BookTime,PayTime,Verificator,VerificateTime,Title,GoodsPrice,OrderRemark,GoodsNum,ShopId,TeamName,GoodsMsg,OrderMsg,OuterTransactionNo,InnerTransactionNo,Star,Remark,FenxiaoOrderNo,FenxiaoPay,GroupNo,StoreId,StoreName,ItemRealPay,ItemRefundPay,ItemExpressTime, DeliveryTime,SuccessTime,PeriodInfo,OrderType") ,     "2": extractFields("TeamName,OrderNo,OrderType,OrderState,OrderMsg,OrderRemark,AllGoodsTotalNum,AllGoodsOriginTotalPrice, AllGoodsPromotionTotalPrice,FoodBoxFee,DeliveryFee,Coupon,ManLiMinus,TotalPay,RealPay,OrderRefundFee,BuyWay,BookTime,PayTime,SuccessTime"),     "3": extractFields("TeamName,OrderNo,OrderType,OrderState,OrderMsg,OrderRemark,ReceiverName,Phone,ReceiverDetailAddress,AllGoodsTotalNum,AllGoodsOriginTotalPrice, AllGoodsPromotionTotalPrice,FoodBoxFee,DeliveryFee,Coupon,ManLiMinus,TotalPay,RealPay,OrderRefundFee,BuyWay,BookTime,PayTime,SuccessTime"),     "4": extractFields("OrderNo, Saleway, OrderType, OrderState, OrderSource, BookTime, PayTime, SuccessTime, BuyWay, InnerTransactionNo, OrderTotalPrice, OrderPostage, TotalPromotion, ShouldPay, RealPay, CashReturn, PromotionDetail, AllGoodsTitle, AllGoodsKinds, DeliveryType, DeliveryTime, ReceiverProvince, ReceiverCity, ReceiverCounty, ReceiverDetail, Receiver, ReceiverTel, OrderMsg, OrderStar, FansName, IsMember, FansTel, StoreName, Cashier, OrderRefundState, OrderRefundFee, OrderRemark, PeriodInfo, IDCard"),     "5": extractFields("OrderId, SimpleOrderState, GoodsTitle, GoodsType, GoodsKind, GoodsSaleway, GoodsSku, GoodsSkuCode, GoodsBizCode, GoodsOriginPrice, GoodsPromotionDetail, GoodsActivityPrice, GoodsTotalNum, GoodsUnit, GoodsTotalPrice, GoodsSharedShopPromotion, GoodsActualDealPay, GoodsPointsPrice, GoodsRemark, GoodsExpressState, GoodsExpressWay, GoodsExpressCorp, GoodsExpressNo, GoodsExpressPerson, GoodsExpressTime, GoodsRefundState, GoodsRefundFee"),     "7": extractFields("OrderID, OrderType, OrderState, OrderSource, BookTime, PayTime, SuccessTime, BuyWay, OuterTransactionNo, InnerTransactionNo, OrderTotalPrice, OrderPostage, TotalPromotion, ShouldPay, RealPay, CashReturn, PromotionDetail, AllGoodsTitle, AllGoodsKinds, DeliveryType, AppointmentTime, Receiver, ReceiverTel, ReceiverProvince, ReceiverCity, ReceiverCounty, ReceiverDetail, OrderMsg, OrderStar, FansName, IsMember, FansTel, BookStoreName, OrderRefundState, OrderRefundFee, OrderRemark, PeriodInfo, IDCard"),     "8": extractFields("OrderId, OuterTransactionNo, SimpleOrderState, SuccessTime, GoodsTitle, GoodsType, GoodsCategory, GoodsSku, GoodsSkuCode, GoodsBizCode, GoodsOriginPrice, GoodsPromotionDetail, GoodsActivityPrice, GoodsTotalNum, GoodsTotalPrice, GoodsSharedShopPromotion, GoodsActualDealPay, GoodsPointsPrice, GoodsRemark, Receiver, ReceiverTel, ReceiverProvince, ReceiverCity, ReceiverCounty, ReceiverDetail, OrderMsg, GoodsExpressState, GoodsExpressWay, GoodsExpressCorp, GoodsExpressNo, WscGoodsExpressPerson, WscGoodsExpressTime, GoodsRefundState, GoodsRefundFee")     }

def mkdir(filedir):
    isExists=os.path.exists(filedir)
    if not isExists:
        os.makedirs(filedir)
        return True
    else:
        return False

def sendRequest(url, query):
    r = requests.post(url, data=query, headers={"Content-type":"application/json"})
    return r.json()

def getData(url, query):
    try:
        resp = sendRequest(url, query)
        if resp['result'] and resp['data']['success']:
            return resp['data']['data']
        return None
    except:
        return None

def download(url, tag, exportId):
    f = urllib2.urlopen(url)
    data = f.read()
    filename = filedir + tag + "_" + str(exportId) + ".csv"
    with open(filename, "w") as csvFile:
        csvFile.write(data)
    return filename

def getFileLines(filename):
    with open(filename, 'r') as f:
        lines = f.readlines()
    return (filename,lines)

def getExportId(url, query):
    exportId = getData(url,query)
    if not exportId:
        return 0
    return int(exportId)

def getExportRecord(exportId):
    rec = {}
    while len(rec) == 0:
        time.sleep(5)
        queryReq = '{"export_id": %d, "page": 1, "size": 1, "export_state": 10}' % (exportId)
        resp = getData(queryUrl, queryReq)
        if resp['total'] > 0:
            rec = resp['list'][0]
    return rec

def getExportedFile(url, query, tag):
    exportId = getExportId(url, query)
    return getByExportId(exportId, tag)

def getExportedFileWithExportId(url, query, tag, exportId):
    getExportId(url, query)
    return getByExportId(exportId, tag)

def getByExportId(exportId, tag):
    exportRecord = getExportRecord(exportId)
    print 'tag=%s, exportId=%s, url=%s' % (tag, exportId, exportRecord['url'])
    csvFile = download(exportRecord['url'], tag, exportId)
    return csvFile

def cmpByOldExportReq(oldExportReq):
    preOldExportUrl = 'http://pre-host:7001/api/order/export'
    prodOldExportUrl = 'http://prod-host:7001/api/order/export'
    query = oldExportReq
    exportId = int(json.loads(oldExportReq)['export_id'])
    updateRecord(exportId)
    preFile = getExportedFileWithExportId(preOldExportUrl, query, 'pre', exportId)
    updateRecord(exportId)
    prodFile = getExportedFileWithExportId(prodOldExportUrl, query, 'prod', exportId)
    cmpExportFile(preFile, prodFile, query, "1")

def updateRecord(exportId):
    updateParam = '{"source":"AVeryComplexSecretTestHacker", "export_id": %s, "url":"", "export_state": 1, "token": "0"}' % (exportId)
    updateUrl = 'http://pre-host:7001/api/general/update'
    updateResult = getData(updateUrl, updateParam)
    if updateResult:
        print 'update record to init success !'

def cmplines(prodLines, preLines, fields, keyIndex=0):
    print 'length: online=%d, pre=%d' % (len(prodLines), len(preLines))
    try:
        for i in range(len(prodLines)):
            online = prodLines[i].strip().split(',')
            preline = preLines[i].strip().split(',')
            for t in range(len(online)):
                try:
                    if online[t] != preline[t]:
                        print 'diff: field=%s, online=%s, pre=%s, orderNo=%s' % (fields[t], online[t].decode('gb18030'), preline[t].decode('gb18030'), online[keyIndex])
                except:
                    print 'compare failed. field=%s orderNo=%s %s' % (fields[t], online[keyIndex], traceback.format_exc())
        print 'passed.'
    except:
        print 'compare failed. %s' % traceback.format_exc()

def cmpExport(exportReq):
    preFile = getExportedFile(preUrl, exportReq, 'pre')
    prodFile = getExportedFile(prodUrl, exportReq, 'prod')
    templateId = json.loads(exportReq)['template_id']
    cmpExportFile(preFile, prodFile, exportReq, templateId)

def cmpExportFile(preFile, prodFile, exportReq, templateId="1"):
    fields = templateIdFieldsMap[templateId]
    keyIndex = 0
    if templateId == "2" or templateId == "3":  #餐飲的報表,訂單號在第二列
        keyIndex = 1

    # 按照訂單號排序，因為下單時間可能相同，對比時有不必要的不一致
    preSortedFile = getSortedFile(preFile, keyIndex)
    prodSortedFile = getSortedFile(prodFile, keyIndex)
    preSorted = getFileLines(preSortedFile)[1]
    prodSorted = getFileLines(prodSortedFile)[1]
    print 'exportReq=[ %s ], prodFile=%s, preFile=%s' % (exportReq, prodSortedFile, preSortedFile)
    cmplines(prodSorted, preSorted, fields, keyIndex)

def getSortedFile(originFile, index):

    filename = originFile.rsplit('.',1)[0]
    sortedfilename = filename + "_sorted.csv"
    cmd = 'sort -k %d %s > %s' % (index+1, originFile, sortedfilename)
    os.system(cmd)
    return sortedfilename


def sendmail(text):
    sender = '[email protected]'
    receivers = ['[email protected]']

    message = MIMEMultipart()
    message['From'] = Header("導出對比工具", 'utf-8')
    message['To'] =  Header("導出對比工具", 'utf-8')
    subject = '導出對比結果'
    message['Subject'] = Header(subject, 'utf-8')

    message.attach(MIMEText('導出對比結果如附件所示', 'plain', 'utf-8'))

    att1 = MIMEText(open(resultfile, 'rb').read(), 'base64', 'utf-8')
    att1["Content-Type"] = 'application/octet-stream'
    att1["Content-Disposition"] = 'attachment; filename="export_cmp_result.txt"'
    message.attach(att1)

    try:
        smtpObj = smtplib.SMTP('smtp.exmail.qq.com', 25)
        smtpObj.login('[email protected]', 'YxtkETvis7Ck3UYZ')
        smtpObj.sendmail(sender, receivers, message.as_string())
        print "Email Send success!"
    except smtplib.SMTPException:
        print "Email Send failed!"


if __name__ == '__main__':

    savedStdout = sys.stdout

    mkdir(filedir)
    f_result = open(resultfile, 'w')
    sys.stdout = f_result

    allreqs = []
    for func in caseGenerateFuncs:
        allreqs.extend(func(startTimeParam, endTimeParam))
    for req in allreqs:
        cmpExport(req)
        print '\n'

    allOldReqs = []
    for func in oldExportInstGenerateFuncs:
        allOldReqs.extend(func(startTimeParam, endTimeParam))
    for req in allOldReqs:
        #cmpByOldExportReq(req)
        print '\n'

    print 'success done !'

    sendmail('export cmp result')

    f_result.close()

    sys.stdout = savedStdout

cases.py : 導出對比的測試用例配置

#!/usr/bin/env python
# -*- coding: utf-8 -*-
#-------------------------------------------------------------------------------
# Name:        cases.py
# Purpose:     Provides cases of exports
#
# Author:      qin.shuq
#
# Created:     12/22/2017
# Copyright:   (c) qin.shuq 2017
# Licence:     <your licence>
#-------------------------------------------------------------------------------

import time
import math
import json

kdtId = 63077
parts = 2

endTime = math.floor(time.time()) - 300
startTime = endTime - 600

baseExportReqStr = '{"biz_type":"order", "export_type":"default", "request_id": "123456", "source":"testsource", "order_by":"book_time", "order":"desc", "order_search_param": {"kdt_id":%d, "start_time":%d, "end_time":%d, "must_not": {"is_visible":0}}, "template_id":1}' % (kdtId, startTime, endTime)

def divideNParts(total, N):
    '''
       divide [0, total) into N parts:
        return [(0, total/N), (total/N, 2*total/N), ((N-1)*total/N, total)]
    '''

    each = total / N
    parts = []
    for index in range(N):
        begin = index*each
        if index == N-1:
            end = total
        else:
            end = begin + each
        parts.append((begin, end))
    return parts

def commonGenerateReqByTime(startTime, endTime, kdtId=xxx, templateId=1):
    def generateReqByTimeInner(startTime, endTime):
        totalInterval = endTime-startTime
        timeparts = divideNParts(totalInterval, parts)
        timeparts = map(lambda t: (t[0]+startTime, t[1]+startTime), timeparts)
        reqs = []
        for time in timeparts:
            baseReq = buildReq(baseExportReqStr, time[0], time[1], kdtId, templateId)
            reqs.append(json.dumps(baseReq))
        return reqs
    return generateReqByTimeInner

def commonGenerator(startTime, endTime, kdtId=xxx, templateId=1, field='', values=[]):
    def generateReqInner(startTime, endTime):
        reqs = []
        for val in values:
            baseReq = buildReq(baseExportReqStr, startTime, endTime, kdtId, templateId, field, val)
            reqs.append(json.dumps(baseReq))
        return reqs
    return generateReqInner

def buildReq(baseExportReqTemplate, startTime, endTime, kdtId=xxx, templateId=1, field=None, value=None):
    requestId = str(startTime) + "_" + str(endTime) + "_" + str(kdtId) + "_" + str(templateId)
    baseReq = json.loads(baseExportReqTemplate)
    baseReq['order_search_param']['start_time'] = startTime
    baseReq['order_search_param']['end_time'] = endTime
    if field and value:
        baseReq['order_search_param'][field] = value
    baseReq['order_search_param']['kdt_id'] = kdtId
    #baseReq['order_search_param']['order_nos'] = ['E2017**********00001', 'E2017**********0887']
    baseReq['request_id'] = requestId
    baseReq['template_id'] = templateId
    return baseReq

def generateGenerators(startTime, endTime, configs):
    gvars = globals()
    for (templateId,kdtId) in kdtIdTemplateIdMap.iteritems():
        if len(configs) == 0:
            funcName = 'generateReqByTime_' + str(kdtId) + "_" + str(templateId)
            gvars[funcName] = commonGenerateReqByTime(startTime, endTime, kdtId, templateId)
        else:
            for (field, values) in configs.iteritems():
                funcName = 'generateReqBy_' + str(kdtId) + "_" + str(templateId) + "_" + field
                gvars[funcName] = commonGenerator(startTime, endTime, kdtId, templateId, field, values)

#templateId=1,7,8 wsc export ;  =2,3 canyin ; =4,5 retail
kdtIdTemplateIdMap = {"1": xxx, "2":yyy, "3":yyy, "4":zzz, "5":zzz, "7": xxx, "8": xxx}
#kdtIdTemplateIdMap = {"1": xxx}

#如果改動了搜索相關，則需要測試訂單搜索，使用該配置
searchconfigs = {"order_state": [[],["TOPAY"], ["CONFIRM"], ["PAID"], ["SENT"], ["SUCCESS"], ["CLOSE"]],                  "order_type": [[],["NORMAL"], ["GROUP"], ["HOTEL"]],                  "express_type": [[],["EXPRESS"], ["SELF_FETCH"], ["LOCAL_DELIVERY"]],                  "feedback": [[],["SAFE_NEW"], ["SAFE_FINISHED"]],                  "buy_way":[[],["WXPAY", "WXPAY_DAIXIAO", "WXPAY_SHOULI", "WX_APPPAY", "WX_WAPPAY"], ["ALIWAP", "ALIPAY"], ["UMPAY", "UNIONPAY", "UNIONWAP", "BAIDUWAP", "YZPAY"]]
                 }

# 如果只改動了詳情，不需要測試訂單搜索，只需要按照時間段來導出預發線上數據進行比較即可。
detailconfigs = {}

def buildOldExportInst(kdtId,startTime,endTime):
    def buildOldExportInstInner(startTime, endTime):
        totalInterval = endTime-startTime
        timeparts = divideNParts(totalInterval, parts)
        timeparts = map(lambda t: (t[0]+startTime, t[1]+startTime), timeparts)
        reqs = []
        for time in timeparts:
            req = {}
            req['kdt_id']=kdtId
            req['start_time']=time[0]
            req['end_time']=time[1]
            req['export_type']='default';
            req['biz_type']='order';
            req['export_id']= 'xxxxxx';   # 借殼生蛋
            req['param_hash']= '7295**********************e3c1';
            reqs.append(json.dumps(req))
        return reqs
    return buildOldExportInstInner

vipKdtIdConfigs = [xxx]

def buildAllOldExportInstGenerator(startTime, endTime, vipKdtIdConfigs):
    gvars = globals()
    for config in vipKdtIdConfigs:
        funcName = 'buildOldExportInstInner_' + str(config)
        gvars[funcName] = buildOldExportInst(config, startTime, endTime)

def getGenerateFuncs():
    gvars = globals()
    caseGenerators = [ gvars[var] for var in gvars if var.startswith('generateReq')  ]
    print 'case generators: ', [ var for var in gvars if var.startswith('generateReq') ]
    return caseGenerators

def getOldExportGenerateFuncs():
    gvars = globals()
    oldExportCaseGenerators = [ gvars[var] for var in gvars if var.startswith('buildOldExportInstInner')  ]
    print 'old export case generators: ', [ var for var in gvars if var.startswith('buildOldExportInstInner') ]

    return oldExportCaseGenerators

generateGenerators(startTime, endTime, detailconfigs)
caseGenerateFuncs = getGenerateFuncs()

buildAllOldExportInstGenerator(startTime, endTime, vipKdtIdConfigs)
oldExportInstGenerateFuncs = getOldExportGenerateFuncs()

# 5.14 - 5.22 has dirty data, should be excluded
startTimeParam = 1506787200
endTimeParam = 1530288000

小結

無論大改還是小改，通過運行這個預發和線上對比工具，很大程度上增強了成功發布的信心。可見，預發和線上的自動化對比工具，確實是發布前的最後一道防線。

訂單導出的預發和線上的自動化對比工具