BugStory

• Software Fault : A static defect in thesoftware (i.e., defect)，靜態存在於軟體中的缺陷，如code寫錯了

• Software Error : An incorrect internalstate that is the manifestation of some fault，軟體執行時，執行到fault觸發產生錯誤的中間狀態。

• Software Failure : External, incorrectbehavior with respect to the requirements or other description of the expected behavior，Error傳到軟體外部，使得使用者或測試人員觀測到失效的行為。

Fault：

可能導致系統或功能失效的異常條件（Abnormal condition thatcan cause an element or an item to fail.），可譯為“故障”。

Error：

計算、觀察或測量值或條件，與真實、規定或理論上正確的值或條件之間的差異（Discrepancybetween a computed, observed or measured value or condition and the true,specified, or theoretically correct value or condition.），可譯為“錯誤”。Error是能夠導致系統出現Failure的系統內部狀態。

Failure：

當一個系統不能執行所要求的功能時，即為Failure，可譯為“失效”。（Termination ofthe ability of an element or an item to perform a function as required.）

PIEModel

1. Execution/Reachability : The location orlocations in the program that contain the fault must be reached，執行必須通過錯誤的程式碼

2. Infection : The state of the programmust be incorrect ，在執行錯誤程式碼時必須觸發一個錯誤的中間狀態

3. Propagation : The infected state mustpropagate to cause some output of the program to be incorrect，錯誤的中間狀態必須傳播到最後輸出，使得觀測到的輸出結果和預期結果不一致，即失效。

產生fault的程式，可能在測試時不會觸發錯誤的中間狀態；同理，觸發錯誤的中間狀態可能不會使測試人員觀察到失效的行為。

有沒有一個fault，任何測試都不會將其測試出來？這樣還能稱之為fault嗎？

有這樣的faul

Terminology

測試是為了發現bug，通過實施測試和觀察failure

debug是為了解決bug，通過定位，理解和修復fault

Validation

“The assurance that a product, service, orsystem meets the needs of the customer and other identified stakeholders. Itoften involves acceptance and suitability with external customers.”

Verification

“The evaluation of whether or not aproduct, service, or system complies with a regulation, requirement,specification, or imposed condition. It is often an

internal process.”

Verification是驗證,是通過提供客觀證據證明規定的要求是否得到滿足,也就是說,輸入與輸出比較.

Validation是確認,是在驗證好的基礎上,對預期的使用和應用要求是否得到滿足,也就是說,在確認時,應考慮使用和應用的條件範圍要遠遠大於輸入時確定的範圍.一般是由客戶或代表客戶的人執行.

Verification也就是說要做正確、而Validation是看經過Verification是否是我們想要的。

而Validation首先前提是經過Verification，重要的是做的是否是customer需要的。

黑盒測試，通過測試來檢測每個功能是否都能正常使用。在測試中，把程式看作一個不能開啟的黑盒子，在完全不考慮程式內部結構和內部特性的情況下，在程式介面進行測試，它只檢查程式功能是否按照需求規格說明書的規定正常使用，程式是否能適當地接收輸入資料而產生正確的輸出資訊。黑盒測試著眼於程式外部結構，不考慮內部邏輯結構，主要針對軟體介面和軟體功能進行測試。

白盒測試，又稱結構測試、透明盒測試、邏輯驅動測試或基於程式碼的測試。白盒測試是一種測試用例設計方法，盒子指的是被測試的軟體，白盒指的是盒子是可視的，你清楚盒子內部的東西以及裡面是如何運作的。"白盒"法全面瞭解程式內部邏輯結構、對所有邏輯路徑進行測試。"白盒"法是窮舉路徑測試。在使用這一方案時，測試者必須檢查程式的內部結構，從檢查程式的邏輯著手，得出測試資料。貫穿程式的獨立路徑數是天文數字。

灰盒測試，是介於白盒測試與黑盒測試之間的一種測試，灰盒測試多用於整合測試階段，不僅關注輸出、輸入的正確性，同時也關注程式內部的情況。灰盒測試不像白盒那樣詳細、完整，但又比黑盒測試更關注程式的內部邏輯，常常是通過一些表徵性的現象、事件、標誌來判斷內部的執行狀態。

測試等級：

系統測試,System testing

整合測試,Integration testing

模組測試,Module testing

單元測試,Unit testing

FaultRevisit

Fault is defined by testing! 故障是通過測試定義的

自動化軟體測試 Automatic software testing

測試用例生成

測試資料生成

測試用例選擇

•Regression Test Selection

•針對修改部分的測試

• Test Suite Reduction

•針對測試需求的用例集約簡

• Test Case Prioritization

•針對測試需求的用例排序

軟體行為學習

測試用例修復

測試用例演化

Junit

Junit測試時程式設計師測試，即所謂白盒測試

整合TestCase類即可利用junit測試

需要知道程式結構以及邏輯

設計不同的測試輸入以及輸出語言

儘可能覆蓋多的語句以及分支

測試的技巧

1.     大量測試資料

2.     典型錯誤型別

3.     強制產生錯誤條件

CFG Generator 觀測結構

CFG——Control FlowGraph 控制流圖

觀察程式的控制流走向

CodeCover

用來統計測試覆蓋率，支援多種覆蓋（包括語句覆蓋，分支覆蓋，MCDC覆蓋等）

CFG

Test Path: A path that starts at an initialvertex and ends at a final vertex

Each test executes one and only one testpath

GraphCoverage Criteria 圖覆蓋準則

• Syntactic reach : A path exists in thegraph

• Semantic reach : A test exists that canexecute that path

語義可行的一定語法可行，但語法可行的不一定語義可行

Test Criteria

• Test Requirements (TR) :

Describe properties of test paths

• Test Criterion :

Rules that define test requirements

• Satisfaction:

Given a set TR of test requirements for acriterion C, a set of tests T satisfies C on a graph if and only if for everytest requirement in TR, there is a test path in path(T) that meets the testrequirement tr

StructuralCoverage 結構覆蓋

Vertex Coverage (VC) 點覆蓋

Test set T satisfies vertex coverage ongraph G if and only if for every syntactically reachable vertex v in V, thereis a path p in path(T) such that p covers v.

Edge Coverage (EC) 邊覆蓋 邊覆蓋一定滿足點覆蓋，反過來就不一定了

Test set T satisfies vertex coverage ongraph G if and only if for every syntactically reachable edge e in E, there isa path p in path(T) such that p covers e.

Edge-Pair Coverage (EPC) 邊對覆蓋

TR contains each reachable path of lengthto up 2, inclusive, in G.

Complete Path Coverage (CPC) : TR containsall paths in G.

n-Path Coverage (nPC) : TR contains eachreachable path of length up to n, inclusive, in G.

ControlFlow Graph

•A control flow graph (CFG) is a representation,using graph notation, of all paths that might be traversed through a programduring its execution.

if , if-return , while, do , for , breakand continue , switch

PrimePath Coverage

如果一個圖含有一個迴圈，那麼它就有無限條路徑，這時，CPC是不可行的

Simple Path : A path from node ni to nj issimple if no node appears more than once, except possibly the first and lastnodes are the same 路徑中沒有相同的結點，除了第一個和最後一個可以相同

• No internal loops

• A loop is a simple path

Prime Path : A simple path that does notappear as a proper subpath of any other simple path

是簡單路徑，且不是其他簡單路徑的子路徑

Prime Path Coverage (PPC) : TR containseach prime path in G.包括點覆蓋，路徑覆蓋，但不包括邊對覆蓋

Round-Trip Path : A prime path that startsand ends at the same node

Simple Round Trip Coverage (SRTC) : TRcontains at least one round-trip path for each reachable vertex in G thatbegins and ends a round-trip path.

Complete Round Trip Coverage (CRTC) : TRcontains all roundtrip paths for each reachable vertex in G.

Prime path: the maximal-long simple path

BasicPath Coverage

介於分支覆蓋和全路徑覆蓋中

Testing that fulfills the requirements ofbranch testing & also tests all of the independent paths that could be usedto construct any arbitrary path through the computer program.

Independent Path

•A path through the system is independentfrom other paths only if it includes some vertices or edges that are notcovered in the other path.

From linear algebra, it is known that eachmatrix has a unique rank (number of linearly independent rows) that is lessthan or equal to the number of columns.

•The number of linearly independent pathsis the rank of this matrix.

•The rank of this matrix is exactly the cyclomaticcomplexity of the graph. 圈複雜度

CC=E-V+2

McCabe’s Basic Path Testing

1. Generate control flow graph

2. Compute cyclomatic complexity

3. Select a set of basic paths

4. Generate tests for the basic paths

EventFlow Graph 事件流圖

Definition: An Event Flow Graph (EFG) is a3-tuple M = ⟨V, I, E⟩, where:

1. V is a set of vertices representing allevents of objects.

2. I ⊆V is a set of initial vertices.

3. E ⊆V ×Vis a set of edges between vertices. (vi, vj) ∈E if vjmay be executed immediately after vi.

RandomTesting

Test cases are generated purely at random[SWEBOK v3.0]

–Input domain must be known

–Pick random points within input domain

–Automation

Problems in RT

• Define input domain

• Random mechanism 隨機機制

• Randomness and IntegrityService(random.org) 隨機性和完整性

ART Algorithm

randomly generate an input t, run t, add tto T

while (stop criteria not reached)

randomlygenerate k candidate input c1, … ck

for eachcandidate ci

compute min distance di to T

end for

select onecandidate t with min distance

run t, add t toT

end while

Problems in ART

• Distance

• Overhead 開銷

• Curse of Dimensionality 維度災難

Anti-Random Testing Process:

1 . 選擇一條測試用例

2. 從所有可能的測試用例中，選擇一條使其與已有測試用例的海明距離之和最大

3. 重複第二步，直至測試用例總量滿足需求

DataFlow Coverage 資料流覆蓋

Sets of Def and Use

• def (n) or def (e) :

The set of variables that are defined bynode n or edge e

• use (n) or use (e) :

The set of variables that are used by noden or edge e

DU pair :

A pair of locations (li, lj) such that avariable v is defined at li and used at lj

• Def-clear :

A path from li to lj is def-clear withrespect to variable v if v is not given another value on any of the nodes oredges in the path

• Reach :

If there is a def-clear path from li to ljwith respect to v, the def of v at li reaches the use at lj

• du-path :

A simple subpath that is def-clear withrespect to v from a defof v to a use of v

• du (ni, nj, v)

the set of du-paths from ni to nj

• du (ni, v)

the set of du-paths that start at ni

All-defs coverage (ADC) :

For each set of du-paths S = du (n, v), TRcontains at least one path d in S.

All-uses coverage (AUC) :

For each set of du-paths to uses S = du(ni, nj, v), TR contains at least one path d in S.

All-du-paths coverage (ADUPC) :

For each set S = du (ni, nj, v), TRcontains every path d in S.

EquivalencePartitioning 等價劃分

• Can be equally applied at several levelsof testing

 –Unit

– Integration

– System

• Relatively easy to apply with noautomation

• Easy to adjust the procedure to get moreor fewer tests

Input Domains

• The input domain for a program containsall the possible inputs to that program

• For even small programs, the input domainis so large that it might as well be infinite

• Testing is fundamentally about choosingfinite sets of values from the input domain

• Input parameters define the scope of theinput domain

– Parameters to a method

– Data read from a file

– Global variables

– User level inputs

• Domain for each input parameter ispartitioned into regions

• At least one value is chosen from eachregion

Partitioning Domains

• Domain D

• Partition scheme p of D

• The partition p defines a set of blocks,b1 , b2 , … bn

• The partition must satisfy two properties:

1. blocks must be pairwise disjoint (nooverlap) 劃分的塊之間不能有重疊

2. together the blocks cover the domain D(complete) 所有的塊和在一起完全覆蓋D

• 將不能窮舉的測試過程進行分類

• 把輸入域劃分為若干子集，然後從每一個子集中選取少數具有代表性的資料作為測試用例

• 等價類即子集合，其中的資料在某種意義上等價

有效與無效等價類

(1) 有效等價類 – 有意義的、合理的輸入資料

– 檢驗程式是否實現了規格說明中預先規定的功能和效能

(2) 無效等價類

– 是無意義的、不合理的輸入資料所構成的集合

– 鑑別程式異常處理的情況，檢查功能和效能的實現是否有不符合規格說明要求的地方

等價類的劃分原則

• 完備性

–等價類的並集應涵蓋整個輸入域

• 無冗餘

–等價類之間互不相交

Two Approaches

1. Interface-based approach 基於介面

– Develops characteristics directly fromindividual input parameters

– Simplest application

– Can be partially automated in somesituations

2. Functionality-based approach 基於功能

– Develops characteristics from abehavioral view of the program under test

– Harder to develop—requires more designeffort

– May result in better tests, or fewertests that are as effective

Interface-Based Approach

• Mechanically consider each parameter inisolation

• This is an easy modeling technique andrelies mostly on syntax

• Ignores relationships among parameters

Functionality-Based Approach

• Identify characteristics that correspondto the intended functionality

• Requires more design effort from tester

• Can incorporate domain and semanticknowledge

• Can use relationships among parameters

• Modeling can be based on requirements,not implementation

• The same parameter may appear in multiplecharacteristics, so it’s harder to translate values to test cases

等價類的劃分方法

（1）按照區間劃分 在輸入條件規定了取值範圍或值的個數的情況下，可以確定一個有效等價類和兩個無效等價類。

（2）按照數值劃分 在規定了一組輸入資料 （假設包括n個輸入值），並且程式要對每一 個輸入值分別進行處理的情況下，可確定n個有效等價類（每個值確定一個有效等價類）和一個無效等價類（所有不允許的輸入值的集合）。

（3）按照數值集合劃分 在輸入條件規定了輸入值的集合或規定了“必須如何”的條件下，可以確定一個有效等價類和一個無效等價類（該集合有效值之外）

（4）按照限制條件或規則劃分 在規定了輸 入資料必須遵守的規則或限制條件的情況下可確定一個有效等價類（符合規則）和若干個無效等價類（從不同角度違反規則）。

（5）細分等價類 在確知已劃分的等價類中 各元素在程式中的處理方式不同的情況下，則應再將該等價類進一步劃分為更小的等價類，並建立等價類表。

LogicCoverage 邏輯覆蓋

Decision Coverage (DC)： Executing true and false of decision.

Condition Coverage (CC): Executing true andfalse of each condition

Condition/Decision Coverage (C/DC): CombingDC and CC.

Multiple Condition Coverage

•Multiple condition coverage (MCC) reportswhether every possible combination of Boolean sub-expressions occurs.

•The test cases required for full multiplecondition coverage of a condition are essentially given by the logical operatortruth table for the condition.

Boundary-ValueAnalysis 邊界值分析

An input variable with equivalent class[Min, Max]

Select

– Min

– Min+

– Nom

– Max-

– Max

Weak Robust-Boundary Analysis

Strong Robust-Boundary Analysis

MC/DC

•MC/DC or MCDC:

Executing the independent true and falseoutcomes of each condition.

•MC/DC ≥ C/DC

MC/DC: Discussion

• Modified condition/decision coverage wasdesigned for languages containing logical operators that do not short circuit. 短路

• The short circuit logical operators in C,C++ and Java only evaluate conditions when their result can affect theencompassing decision.

• MC/DC will be affected by the structuresof decisions in program.

CombinatorialTesting 組合測試

Weak Equivalence Class Testing 弱等價類測試，根據類別最多的輸入來劃分

Strong Equivalence Class Testing 強等價類測試，根據所有輸入的類別來劃分

Cost Effective Testing

Pair-wise Testing

t-wise/t-ways Combinatorial Testing

Variable strength combinatorial testing

MutationTesting 變異測試

• Faults are introduced into the program bycreating many versions of the program called mutants.

• Each mutant contains a single fault.

• Test cases are applied to the originalprogram and to the mutant program.

• The goal is to cause the mutant programto fail, thus demonstrating the effectiveness of the test case.

一種在細節方面改程序序原始碼的軟體測試方法。這些所謂的變異，是基於良好定義的變異操作，這些操作或者是模擬典型應用錯誤（例如：使用錯誤的操作符或者變數名字），或者是強制產生有效地測試（例如使得每個表示式都等於0）。目的是幫助測試者發現有效地測試，或者定位測試資料的弱點，或者是在執行中很少（或從不）使用的程式碼的弱點。

• Mutation Testing is a method of insertingfaults into programs to test whether the tests pick them up, thereby validatingor invalidating the tests.

• Mutation Testing is a testing techniquethat focuses on measuring the adequacy of test cases. 充分性

• Mutation Testing should be used inconjunction with traditional testing techniques, not instead of them.

• Mutation testing involves the creation ofa set of mutant programs of the program being tested.

• Each mutant differs from the originalprogram by one mutation.

• A mutation is a single syntactic changethat is made to a program statement.

Categories of Mutation Operators

Operand Mutation Operators: 運算元變異

• Replace a single operand with anotheroperand or constant.

Expression Mutation Operators: 表示式變異

• Replace an operator or insert newoperators.

Equivalent Mutant 等價變異體

• There may be surviving mutants thatcannot be killed, these are called Equivalent Mutants

• Although syntactically different, thesemutants are indistinguishable through testing.

• They therefore have to be checked ‘byhand’

一些不能被殺死的變異，即使語法上不通，通過測試是不能分辨的，只能手動檢測

Mutation Score

• The mutation score for a set of testcases is the percentage of non-equivalent mutants killed by the test data.

• Mutation Score = 100 * K / (T - E)

• K= Number of killed mutants

• T= Number of total mutants

• E = Number of equivalent mutants

• A set of test cases is mutation adequateif its mutation score is 100%.

Fault-basedTesting 基於fault的測試

變異測試是運用最廣泛的

• Operator Reference Fault (ORF)

• An occurrenceof a logical connective “&” is replaced by “|” or vice versa.

• Expression Negation Fault (ENF)

• Asubexpression (except conditions) is replaced by its negation.

• Variable Negation Fault (V NF)

• An occurrenceof a condition is replaced by its negation.

• Associative Shift Fault (ASF)

• ASF is causedby omission of the brackets because of themisunderstanding about operatorevaluation priorities.

• Missing Variable Fault (MVF)

• An occurrenceof a condition is omitted.

• Variable Reference Fault (VRF)

• An occurrenceof a condition is replaced by another possible condition .

• Clause Conjunction Fault (CCF)

• An occurrenceof condition c is replaced by c & c’

• Clause Disjunction Fault (CDF)

• Stuck-At-0 Fault (SA0)

• An occurrenceof a condition is replaced by 0

• Stuck-At-1 Fault (SA1)

AdvancedLogic Coverage 增強邏輯覆蓋

DecisionTables 決策表

Decision tables represent logicalrelationships between conditions (roughly, inputs) and actions (roughly,outputs). Test cases are systematically derived … [SWEBOK v3.0]

• 決策表的優點：能夠將複雜的問題按照各種可能的情況全部列舉出來，簡明並避免 遺漏。因此，利用決策表能夠設計出完整的測試用例集合。

• 在一些資料處理問題當中，某些操作的實施依賴於多個邏輯條件的組合，即：針對不同邏輯條件的組合值，分別執行不同的操作。決策表很適合於處理這類問題。

決策表的組成

• 決策表通常由以下4部分組成：

– 條件樁—列出問題的所有條件

– 條件項—針對條件樁中條件列出所有可能的取值

– 動作樁—列出問題規定的可能採取的操作

– 動作項—指出條件項各取值情況下應採取的動作

條件樁 條件項

動作樁 動作項

任何一個條件組合的特定取值及相應要執行的動作為一條規則。在決策表中貫穿條件項和動作項的一列是一條規則。

決策表的生成

構造決策表的5個步驟：

（1）確定規則的個數（每個條件取真、假值）

（2）列出所有的條件樁和動作樁

（3）填入條件項

（4）填入動作項，得到初始決策表

（5）簡化決策表，合併相似規則

若表中有兩條以上規則具有相同的動作，並且在條件項之間存在極為相似的關係，便可以合併

合併後的條件項用符號“-”表示，說明執行的動作與該條件的取值無關，稱為無關條件

Diversity

Graph-based Test Design-CFG

Graph-based Test Design-EFG

Adaptive Random Test Design

Equivalence Partition Test Design

Combinatorial Test Design

TestSuite Reduction 測試集簡化

TestCase Prioritization 測試用例優先順序技術

TestOracle

FaultLocalization

功能測試

功能測試 Functional Testing

根據產品特性和設 需求，驗證一個產品的特性和行為是否滿足設計需求

功能測試常用步驟

1. 根據需求來細分功能點

2. 根據功能點派生測試需求

3. 根據測試需求設計功能測試用例

4. 逐項執行功能測試用例驗證產品

相關的測試型別

•正確性

•產品功能是否與需求和設計文件一致

•可靠性

•使用者互動是否引發軟體崩潰和其它異常

•易用性

•軟體產品完成特定任務的難易程度

探索式測試

效能測試

•效能測試 Performance Testing

•驗證產品的效能在特定負載和環境條件下使用是否滿足效能指標

•進一步發現系統中存在的效能瓶頸，優化系統

效能測試度量方法

•不同的關注物件採用不同的效能的度量方法

•服務端效能採用CPU、記憶體等使用率來度量

•客戶端效能通常根據系統處理特定使用者請求的響應時間來度量

移動應用測試