Gradient Checking

Welcome to the final assignment for this week! In this assignment you will learn to implement and use gradient checking.

You are part of a team working to make mobile payments available globally, and are asked to build a deep learning model to detect fraud–whenever someone makes a payment, you want to see if the payment might be fraudulent, such as if the user’s account has been taken over by a hacker.

But backpropagation is quite challenging to implement, and sometimes has bugs. Because this is a mission-critical application, your company’s CEO wants to be really certain that your implementation of backpropagation is correct. Your CEO says, “Give me a proof that your backpropagation is actually working!” To give this reassurance, you are going to use “gradient checking”.

Let’s do it!

# Packages
import numpy as np
from testCases import *
from gc_utils import sigmoid, relu, dictionary_to_vector, vector_to_dictionary, gradients_to_vector

1) How does gradient checking work?

Backpropagation computes the gradients ∂J∂θ, where θ denotes the parameters of the model. J

is computed using forward propagation and your loss function.

Because forward propagation is relatively easy to implement, you’re confident you got that right, and so you’re almost 100% sure that you’re computing the cost J correctly. Thus, you can use your code for computing J to verify the code for computing ∂J∂θ.

Let’s look back at the definition of a derivative (or gradient):

∂J∂θ=limε→0J(θ+ε)−J(θ−ε)2ε(1)

If you’re not familiar with the “limε→0” notation, it’s just a way of saying “when ε is really really small.”

We know the following:

• ∂J∂θ is what you want to make sure you’re computing correctly.
• You can compute J(θ+ε) and J(θ−ε) (in the case that θ is a real number), since you’re confident your implementation for J is correct.

Lets use equation (1) and a small value for ε to convince your CEO that your code for computing ∂J∂θ is correct!

2) 1-dimensional gradient checking

Consider a 1D linear function J(θ)=θx. The model contains only a single real-valued parameter θ, and takes x as input.

You will implement code to compute J(.) and its derivative ∂J∂θ. You will then use gradient checking to make sure your derivative computation for J is correct.

Figure 1 : 1D linear model

The diagram above shows the key computation steps: First start with x, then evaluate the function J(x) (“forward propagation”). Then compute the derivative ∂J∂θ (“backward propagation”).

Exercise: implement “forward propagation” and “backward propagation” for this simple function. I.e., compute both J(.) (“forward propagation”) and its derivative with respect to θ (“backward propagation”), in two separate functions.

# GRADED FUNCTION: forward_propagation

def forward_propagation(x, theta):
    """
    Implement the linear forward propagation (compute J) presented in Figure 1 (J(theta) = theta * x)

    Arguments:
    x -- a real-valued input
    theta -- our parameter, a real number as well

    Returns:
    J -- the value of function J, computed using the formula J(theta) = theta * x
    """

    ### START CODE HERE ### (approx. 1 line)
    J = theta * x
    ### END CODE HERE ###

    return J

x, theta = 2, 4
J = forward_propagation(x, theta)
print ("J = " + str(J))

J = 8

Expected Output:

# GRADED FUNCTION: backward_propagation

def backward_propagation(x, theta):
    """
    Computes the derivative of J with respect to theta (see Figure 1).

    Arguments:
    x -- a real-valued input
    theta -- our parameter, a real number as well

    Returns:
    dtheta -- the gradient of the cost with respect to theta
    """

    ### START CODE HERE ### (approx. 1 line)
    dtheta = x
    ### END CODE HERE ###

    return dtheta

x, theta = 2, 4
dtheta = backward_propagation(x, theta)
print ("dtheta = " + str(dtheta))

dtheta = 2

Expected Output:

** dtheta **

2

Exercise: To show that the backward_propagation() function is correctly computing the gradient ∂J∂θ, let’s implement gradient checking.

Instructions:
- First compute “gradapprox” using the formula above (1) and a small value of ε. Here are the Steps to follow:
1. θ+=θ+ε
2. θ−=θ−ε
3. J+=J(θ+)
4. J−=J(θ−)
5. gradapprox=J+−J−2ε
- Then compute the gradient using backward propagation, and store the result in a variable “grad”
- Finally, compute the relative difference between “gradapprox” and the “grad” using the following formula:

difference=∣∣grad−gradapprox∣∣2∣∣grad∣∣2+∣∣gradapprox∣∣2(2)
You will need 3 Steps to compute this formula:
- 1’. compute the numerator using np.linalg.norm(…)
- 2’. compute the denominator. You will need to call np.linalg.norm(…) twice.
- 3’. divide them.
- If this difference is small (say less than 10−7), you can be quite confident that you have computed your gradient correctly. Otherwise, there may be a mistake in the gradient computation.

# GRADED FUNCTION: gradient_check

def gradient_check(x, theta, epsilon = 1e-7):
    """
    Implement the backward propagation presented in Figure 1.

    Arguments:
    x -- a real-valued input
    theta -- our parameter, a real number as well
    epsilon -- tiny shift to the input to compute approximated gradient with formula(1)

    Returns:
    difference -- difference (2) between the approximated gradient and the backward propagation gradient
    """

    # Compute gradapprox using left side of formula (1). epsilon is small enough, you don't need to worry about the limit.
    ### START CODE HERE ### (approx. 5 lines)
    thetaplus = theta+epsilon                      # Step 1
    thetaminus = theta-epsilon                     # Step 2
    J_plus = forward_propagation(x,thetaplus)        # Step 3
    J_minus = forward_propagation(x,thetaminus)       # Step 4
    gradapprox = (J_plus-J_minus)/2/epsilon        # Step 5
    ### END CODE HERE ###

    # Check if gradapprox is close enough to the output of backward_propagation()
    ### START CODE HERE ### (approx. 1 line)
    grad = backward_propagation(x,theta)
    ### END CODE HERE ###

    ### START CODE HERE ### (approx. 1 line)
    numerator = np.linalg.norm(grad-gradapprox)    # Step 1'
    denominator = np.linalg.norm(grad)+np.linalg.norm(gradapprox)                             # Step 2'
    difference = numerator/denominator             # Step 3'
    ### END CODE HERE ###

    if difference < 1e-7:
        print ("The gradient is correct!")
    else:
        print ("The gradient is wrong!")

    return difference

x, theta = 2, 4
difference = gradient_check(x, theta)
print("difference = " + str(difference))

The gradient is correct!
difference = 2.91933588329e-10

Expected Output:
The gradient is correct!

** difference **

2.9193358103083e-10

Congrats, the difference is smaller than the 10−7 threshold. So you can have high confidence that you’ve correctly computed the gradient in backward_propagation().

Now, in the more general case, your cost function J has more than a single 1D input. When you are training a neural network, θ actually consists of multiple matrices W

相關推薦

Improving Deep Neural Networks Gradient Checking Homework

Gradient Checking Welcome to the final assignment for this week! In this assignment you will learn to implement and use gradient c

Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization - week1

圖片 .com arr neu regular img family nts radi Normalizing input Vanishing/Exploding gradients deep neural network suffer from t

Coursera 吳恩達 Deep Learning 第2課 Improving Deep Neural Networks 第一週 程式設計作業程式碼 Regularization

2 - L2 Regularization # GRADED FUNCTION: compute_cost_with_regularization def compute_cost_with_reg

Coursera 吳恩達 Deep Learning 第二課 改善神經網路 Improving Deep Neural Networks 第二週 程式設計作業程式碼Optimization methods

Optimization Methods Until now, you’ve always used Gradient Descent to update the parameters and minimize the cost. In this notebo

Coursera 吳恩達 Deep Learning 第2課 Improving Deep Neural Networks 第一週 程式設計作業程式碼 Initialization

2 - Zero initialization # GRADED FUNCTION: initialize_parameters_zeros def initialize_parameters_z

Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization（第二週作業）

1 - Non-regularized model You will use the following neural network (already implemented for you below). This model can be used: in regularization

Mastering the game of Go with deep neural networks and tree search

深度 策略 參數初始化 技術 以及 -1 簡單 cpu 網絡 Silver, David, et al. "Mastering the game of Go with deep neural networks and tree search." Nature 529.758

[譯]深度神經網絡的多任務學習概覽(An Overview of Multi-task Learning in Deep Neural Networks)

noi 使用方式 stats 基於 共享 process machines 嬰兒 sdro 譯自：http://sebastianruder.com/multi-task/ 1. 前言 在機器學習中，我們通常關心優化某一特定指標，不管這個指標是一個標準值，還是企業KPI。為

DeepEyes: 用於深度神經網絡設計的遞進式可視分析系統 (DeepEyes: Progressive Visual Analytics for Designing Deep Neural Networks)

失誤 min 使用 包括 系統 所有 訓練 如果 blog 深度神經網絡，在模式識別問題上，取得非常不錯的效果。但設計一個性能好的神經網絡，需要反復嘗試，是個非常耗時的過程。這個工作[1]實現了用於深度神經網絡設計的可視分析系統，DeepEyes。該系統可以在DNNs訓練過

YouTube推薦系統（下）：Deep Neural Networks for YouTube Recommendations

咳，還是要說說這篇文章，雖然講它的人已經很多了。畢竟作為深度學習模型大規模應用於工業界推薦系統的標誌，這篇文章是繞不過去的。原文來自Deep Neural Networks for YouTube Recommendations，是YouTube2016年發表於Recosys的文章。 這篇文章的結構依然很經

深度神經網路的多工學習概覽(An Overview of Multi-task Learning in Deep Neural Networks)

譯自：http://sebastianruder.com/multi-task/ 1. 前言 在機器學習中，我們通常關心優化某一特定指標，不管這個指標是一個標準值，還是企業KPI。為了達到這個目標，我們訓練單一模型或多個模型集合來完成指定得任務。然後，我們通過精細調參，來改進模型直至效能不再

論文翻譯——Scalable Object Detection using Deep Neural Networks

Scalable Object Detection using Deep Neural Networks 作者：Dumitru Erhan,Christian Szegedy, Alexander Toshev等 發表時間

DeepPose: Human Pose Estimation via Deep Neural Networks論文翻譯

翻譯點選連結獲取 基本思想 級聯網路架構：在第一階段將影象輸入後得到大致位置，在之後的階段利用相同的網路架構得到更精細的結果。對級聯的所有階段使用相同的網路架構，但學習不同的網路引數。其中網路架構使用的是Alex，所不同的是loss函式，AlexNet是用於分類的，而本文的架構是用於迴

《2017-Aggregated Residual Transformations for Deep Neural Networks》論文閱讀

如何評價谷歌的xception網路？ 動機 傳統的要提高模型的準確率，都是加深或加寬網路，但是隨著超引數數量的增加（比如channels數，filter size等等），網路設計的難度和計算開銷也會增加。 本文提出的 ResNeXt 結構可以在不增加引數複

CS229 6.3 Neurons Networks Gradient Checking

BP演算法很難除錯，一般情況下會隱隱存在一些小問題，比如（off-by-one error），即只有部分層的權重得到訓練，或者忘記計算bais unit，這雖然會得到一個正確的結果，但效果差於準確BP得到的結果。 有了cost function，目標是求出一組引數W，b，這裡以表示，cost functio

14.On the Decision Boundary of Deep Neural Networks

關於深度神經網路的決策邊界 摘要 雖然深度學習模型和技術取得了很大的經驗成功，但我們對許多方面成功來源的理解仍然非常有限。為了縮小差距，我們對訓練資料和模型進行了微弱的假設,產生深度學習架構的決策邊界。我們在理論上和經驗上證明，對於二元情形和具有常用交叉熵的多類情況，神經網路的最後權重層收斂

Deep Neural Networks for Object Detection

zhuanzii 採用的是AlexNet,不過稍作修改。 原AlexNet網路： 具體改進： 1. 把最後一個層softmax改成a regession layer. predict a mask of a fixed size. 1代表this pixel l

Channel Pruning for Accelerating Very Deep Neural Networks

https://github.com/yihui-he/channel-pruning   ICCV 2017, by Yihui He, Xiangyu Zhang and Jian Sun Please have a look at&

DeepLearning.ai筆記:(1-4)-- 深層神經網路（Deep neural networks）

這一週主要講了深層的神經網路搭建。 深層神經網路的符號表示 在深層的神經網路中， LL表示神經網路的層數 L=4L=4 n[l]n[l]表示第ll層的神經網路個數 W[l]:(n[l],nl−1)W[l]:(n[l],nl−1) dW[l

DeepLearning.ai作業:(1-4)-- 深層神經網路（Deep neural networks）

不要抄作業！ 我只是把思路整理了，供個人學習。 不要抄作業！ 本週的作業分了兩個部分，第一部分先構建神經網路的基本函式，第二部分才是構建出模型並預測。 Part1 構建的函式有： Initialize the parameters t