標籤： machine_learning

bias & variance

Estimator（估計量）: a function of the data that is used to infer the value of an unknown parameter in a statistical model,can be writed like θ^(X).”估計量”是樣本空間對映到樣本估計值的一個函式 (Then an “estimator” is a function that maps the sample space to a set of sample estimates

.)估計量用來估計未知總體的引數，它有時也被稱為估計子；一次估計是指把這個函式應用在一組已知的資料集上，求函式的結果。對於給定的引數，可以有許多不同的估計量。
Estimand:The parameter being estimated,like θ.
Estimate: a particular realization of this random variable θ^(X) is called the “estimate”,like θ^(x).
Bias: The bias of θˆ is defined as B(θˆ) = E(θˆ)−θ. It is the distance between the average of the collection of estimates, and the single parameter being estimated. It also is the expected value of the error, since E

(θˆ)−θ=E(θˆ−θ) .The estimator θˆ is an unbiased estimator of θ if and only if B(θˆ)=0.example: If the parameter is the bull’s-eye of a target, and the arrows are estimates, then a relatively high absolute value for the bias means the average position of the arrows is off-target, and a relatively low absolute bias means the average position of the arrows is on target. They may be dispersed, or may be clustered.
Variance(方差):The variance of θˆ is simply the expected value of the squared sampling deviations; that is, var(θˆ)=E[(θˆ−E(θˆ))2]. It is used to indicate how far, on average, the collection of estimates are from the expected value(期望) of the estimates.example :If the parameter is the bull’s-eye of a target, and the arrows are estimates, then a relatively high variance means the arrows are dispersed, and a relatively low variance means the arrows are clustered. Some things to note: even if the variance is low, the cluster of arrows may still be far off-target, and even if the variance is high, the diffuse collection of arrows may still be unbiased. Finally, note that even if all arrows grossly miss the target, if they nevertheless all hit the same point, the variance is zero.
The relationship between bias and variance is analogous to the relationship between accuracy and precision.
從以上描述可以看出，bias表示預測值的均值與實際值的差值；而variance表示預測結果作為一個隨機變數時的方差，其描述中類比靶心的例子較為明瞭。http://blog.csdn.net/ywl22/article/details/8606166。

Bias、variance與複雜度的關係


note:the sample mean

X¯¯¯=1N∑i=1NXi is an unbiased estimator of μ,and the sample variance s2=1n−1∑i=1n(Xi−X¯¯¯)2 is an unbiased estimator of σ2(not theS2=1n∑i=1n(Xi−X¯¯¯)2that is a biased estimator of σ^2,proof is here)
樣本均值是總體均值的無偏估計，而樣本方差卻不是總體方差的無偏估計，其小於總體方差。

Mean squared error

In statistics, the mean squared error (MSE) of an estimator measures the average of the squares of the “errors”, that is, the difference between the estimator and what is estimated.MSE is a risk function, corresponding to the expected value of the squared error loss or quadratic loss.(損失函式or代價函式？)
MSE(θ^)=Var(θ^)+(Bias(θ^,θ))2
=E[(θˆ−E(θˆ))2]+(E(θˆ)−θ)2
proof
ps:
In statistics, the bias (or bias function) of an estimator is the difference between this estimator’s expected value and the true value of the parameter being estimated. An estimator or decision rule with zero bias is called unbiased. Otherwise the estimator is said to be biased.