deeplearning.ai課程作業：Course 2 Week 2

Optimization Methods

Until now, you’ve always used Gradient Descent to update the parameters and minimize the cost. In this notebook, you will learn more advanced optimization methods that can speed up learning and perhaps even get you to a better final value for the cost function. Having a good optimization algorithm can be the difference between waiting days vs. just a few hours to get a good result.

Gradient descent goes “downhill” on a cost function $J$. Think of it as trying to do this:

Notations: As usual, $\frac{\partial J}{\partial a } =$

To get started, run the following code to import the libraries you will need.

import numpy as np
import matplotlib.pyplot as plt
import scipy.io
import math
import sklearn
import sklearn.datasets

from opt_utils import load_params_and_grads, initialize_parameters, forward_propagation, backward_propagation
from opt_utils import compute_cost, predict, predict_dec, plot_decision_boundary, load_dataset
from testCases import *

%matplotlib inline
plt.rcParams['figure.figsize'] = (7.0, 4.0) # set default size of plots
plt.rcParams['image.interpolation'] = 'nearest'
plt.rcParams['image.cmap'] = 'gray'

1 - Gradient Descent

A simple optimization method in machine learning is gradient descent (GD). When you take gradient steps with respect to all $m$ examples on each step, it is also called Batch Gradient Descent.

Warm-up exercise: Implement the gradient descent update rule. The gradient descent rule is, for $l = 1, ..., L$:
$W^{[l]} = W^{[l]} - \alpha \text{ } dW^{[l]} \tag{1}$
$b^{[l]} = b^{[l]} - \alpha \text{ } db^{[l]} \tag{2}$

where L is the number of layers and $\alpha$ is the learning rate. All parameters should be stored in the parameters dictionary. Note that the iterator l starts at 0 in the for loop while the first parameters are $W^{[1]}$ and $b^{[1]}$. You need to shift l to l+1 when coding.

# GRADED FUNCTION: update_parameters_with_gd

def update_parameters_with_gd(parameters, grads, learning_rate):
    """
    Update parameters using one step of gradient descent
    
    Arguments:
    parameters -- python dictionary containing your parameters to be updated:
                    parameters['W' + str(l)] = Wl
                    parameters['b' + str(l)] = bl
    grads -- python dictionary containing your gradients to update each parameters:
                    grads['dW' + str(l)] = dWl
                    grads['db' + str(l)] = dbl
    learning_rate -- the learning rate, scalar.
    
    Returns:
    parameters -- python dictionary containing your updated parameters 
    """

    L = len(parameters) // 2 # number of layers in the neural networks

    # Update rule for each parameter
    for l in range(L):
        ### START CODE HERE ### (approx. 2 lines)
        parameters["W" + str(l+1)] = parameters["W"+str(l+1)]-learning_rate*grads["dW"+str(l+1)]
        parameters["b" + str(l+1)] = parameters["b"+str(l+1)]-learning_rate*grads["db"+str(l+1)]
        ### END CODE HERE ###
        
    return parameters

parameters, grads, learning_rate = update_parameters_with_gd_test_case()

parameters = update_parameters_with_gd(parameters, grads, learning_rate)
print("W1 = " + str(parameters["W1"]))
print("b1 = " + str(parameters["b1"]))
print("W2 = " + str(parameters["W2"]))
print("b2 = " + str(parameters["b2"]))

W1 = [[ 1.63535156 -0.62320365 -0.53718766]
 [-1.07799357  0.85639907 -2.29470142]]
b1 = [[ 1.74604067]
 [-0.75184921]]
W2 = [[ 0.32171798 -0.25467393  1.46902454]
 [-2.05617317 -0.31554548 -0.3756023 ]
 [ 1.1404819  -1.09976462 -0.1612551 ]]
b2 = [[-0.88020257]
 [ 0.02561572]
 [ 0.57539477]]

Expected Output:

A variant of this is Stochastic Gradient Descent (SGD), which is equivalent to mini-batch gradient descent where each mini-batch has just 1 example. The update rule that you have just implemented does not change. What changes is that you would be computing gradients on just one training example at a time, rather than on the whole training set. The code examples below illustrate the difference between stochastic gradient descent and (batch) gradient descent.

• (Batch) Gradient Descent:

X = data_input
Y = labels
parameters = initialize_parameters(layers_dims)
for i in range(0, num_iterations):
    # Forward propagation
    a, caches = forward_propagation(X, parameters)
    # Compute cost.
    cost = compute_cost(a, Y)
    # Backward propagation.
    grads = backward_propagation(a, caches, parameters)
    # Update parameters.
    parameters = update_parameters(parameters, grads)
        

• Stochastic Gradient Descent:

X = data_input
Y = labels
parameters = initialize_parameters(layers_dims)
for i in range(0, num_iterations):
    for j in range(0, m):
        # Forward propagation
        a, caches = forward_propagation(X[:,j], parameters)
        # Compute cost
        cost = compute_cost(a, Y[:,j])
        # Backward propagation
        grads = backward_propagation(a, caches, parameters)
        # Update parameters.
        parameters = update_parameters(parameters, grads)

In Stochastic Gradient Descent, you use only 1 training example before updating the gradients. When the training set is large, SGD can be faster. But the parameters will “oscillate” toward the minimum rather than converge smoothly. Here is an illustration of this:

Note also that implementing SGD requires 3 for-loops in total:

1. Over the number of iterations
2. Over the $m$ training examples
3. Over the layers (to update all parameters, from $(W^{[1]},b^{[1]})$ to $(W^{[L]},b^{[L]})$)

In practice, you’ll often get faster results if you do not use neither the whole training set, nor only one training example, to perform each update. Mini-batch gradient descent uses an intermediate number of examples for each step. With mini-batch gradient descent, you loop over the mini-batches instead of looping over individual training examples.

What you should remember:

• The difference between gradient descent, mini-batch gradient descent and stochastic gradient descent is the number of examples you use to perform one update step.
• You have to tune a learning rate hyperparameter $\alpha$.
• With a well-turned mini-batch size, usually it outperforms either gradient descent or stochastic gradient descent (particularly when the training set is large).

2 - Mini-Batch Gradient descent

Let’s learn how to build mini-batches from the training set (X, Y).

There are two steps:

• Shuffle: Create a shuffled version of the training set (X, Y) as shown below. Each column of X and Y represents a training example. Note that the random shuffling is done synchronously between X and Y. Such that after the shuffling the $i^{th}$ column of X is the example corresponding to the $i^{th}$ label in Y. The shuffling step ensures that examples will be split randomly into different mini-batches.

• Partition: Partition the shuffled (X, Y) into mini-batches of size mini_batch_size (here 64). Note that the number of training examples is not always divisible by mini_batch_size. The last mini batch might be smaller, but you don’t need to worry about this. When the final mini-batch is smaller than the full mini_batch_size, it will look like this:
  
  Exercise: Implement random_mini_batches. We coded the shuffling part for you. To help you with the partitioning step, we give you the following code that selects the indexes for the $1^{st}$ and $2^{nd}$ mini-batches:

first_mini_batch_X = shuffled_X[:, 0 : mini_batch_size]
second_mini_batch_X = shuffled_X[:, mini_batch_size : 2 * mini_batch_size]
...

Note that the last mini-batch might end up smaller than mini_batch_size=64. Let $\lfloor s \rfloor$ represents $s$ rounded down to the nearest integer (this is math.floor(s) in Python). If the total number of examples is not a multiple of mini_batch_size=64 then there will be $\lfloor \frac{m}{mini\_batch\_size}\rfloor$ mini-batches with a full 64 examples, and the number of examples in the final mini-batch will be ( $m-mini_\_batch_\_size \times \lfloor \frac{m}{mini\_batch\_size}\rfloor$).

# GRADED FUNCTION: random_mini_batches

def random_mini_batches(X, Y, mini_batch_size = 64, seed = 0):
    """
    Creates a list of random minibatches from (X, Y)
    
    Arguments:
    X -- input data, of shape (input size, number of examples)
    Y -- true "label" vector (1 for blue dot / 0 for red dot), of shape (1, number of examples)
    mini_batch_size -- size of the mini-batches, integer
    
    Returns:
    mini_batches -- list of synchronous (mini_batch_X, mini_batch_Y)
    """
    
    np.random.seed(seed)            # To make your "random" minibatches the same as ours
    m = X.shape[1]                  # number of training examples
    mini_batches = []
        
    # Step 1: Shuffle (X, Y)
    permutation = list(np.random.permutation(m))
    shuffled_X = X[:, permutation]
    shuffled_Y = Y[:, permutation].reshape((1,m))

    # Step 2: Partition (shuffled_X, shuffled_Y). Minus the end case.
    num_complete_minibatches = math.floor(m/mini_batch_size) # number of mini batches of size mini_batch_size in your partitionning
    for k in range(0, num_complete_minibatches):
        ### START CODE HERE ### (approx. 2 lines)
        mini_batch_X = shuffled_X[:, mini_batch_size*(k) : mini_batch_size*(k+1)]
        mini_batch_Y = shuffled_Y[:, mini_batch_size*(k) : mini_batch_size*(k+1)]
        ### END CODE HERE ###
        mini_batch = (mini_batch_X, mini_batch_Y)
        mini_batches.append(mini_batch)
    
    # Handling the end case (last mini-batch < mini_batch_size)
    if m % mini_batch_size != 0:
        ### START CODE HERE ### (approx. 2 lines)
        mini_batch_X = shuffled_X[:, mini_batch_size*num_complete_minibatches : m]
        mini_batch_Y = shuffled_Y[:, mini_batch_size*num_complete_minibatches : m]
        ### END CODE HERE ###
        mini_batch = (mini_batch_X, mini_batch_Y)
        mini_batches.append(mini_batch)
    
    return mini_batches

X_assess, Y_assess, mini_batch_size = random_mini_batches_test_case()
mini_batches = random_mini_batches(X_assess, Y_assess, mini_batch_size)

print ("shape of the 1st mini_batch_X: " + str(mini_batches[0][0].shape))
print ("shape of the 2nd mini_batch_X: " + str(mini_batches[1][0].shape))
print ("shape of the 3rd mini_batch_X: " + str(mini_batches[2][0].shape))
print ("shape of the 1st mini_batch_Y: " + str(mini_batches[0][1].shape))
print ("shape of the 2nd mini_batch_Y: " + str(mini_batches[1][1].shape)) 
print ("shape of the 3rd mini_batch_Y: " + str(mini_batches[2][1].shape))
print ("mini batch sanity check: " + str(mini_batches[0][0][0][0:3]))

shape of the 1st mini_batch_X: (12288, 64)
shape of the 2nd mini_batch_X: (12288, 64)
shape of the 3rd mini_batch_X: (12288, 20)
shape of the 1st mini_batch_Y: (1, 64)
shape of the 2nd mini_batch_Y: (1, 64)
shape of the 3rd mini_batch_Y: (1, 20)
mini batch sanity check: [ 0.90085595 -0.7612069   0.2344157 ]

Expected Output:

What you should remember:

• Shuffling and Partitioning are the two steps required to build mini-batches
• Powers of two are often chosen to be the mini-batch size, e.g., 16, 32, 64, 128.

3 - Momentum

Because mini-batch gradient descent makes a parameter update after seeing just a subset of examples, the direction of the update has some variance, and so the path taken by mini-batch gradient descent will “oscillate” toward convergence. Using momentum can reduce these oscillations.

Momentum takes into account the past gradients to smooth out the update. We will store the ‘direction’ of the previous gradients in the variable $v$. Formally, this will be the exponentially weighted average of the gradient on previous steps. You can also think of $v$ as the “velocity” of a ball rolling downhill, building up speed (and momentum) according to the direction of the gradient/slope of the hill.
Exercise: Initialize the velocity. The velocity, $v$, is a python dictionary that needs to be initialized with arrays of zeros. Its keys are the same as those in the grads dictionary, that is:
for $l =1,...,L$:

v["dW" + str(l+1)] = ... #(numpy array of zeros with the same shape as parameters["W" + str(l+1)])
v["db" + str(l+1)] = ... #(numpy array of zeros with the same shape as parameters["b" + str(l+1)])

Note that the iterator l starts at 0 in the for loop while the first parameters are v[“dW1”] and v[“db1”] (that’s a “one” on the superscript). This is why we are shifting l to l+1 in the for loop.

# GRADED FUNCTION: initialize_velocity

def initialize_velocity(parameters):
    """
    Initializes the velocity as a python dictionary with:
                - keys: "dW1", "db1", ..., "dWL", "dbL" 
                - values: numpy arrays of zeros of the same shape as the corresponding gradients/parameters.
    Arguments:
    parameters -- python dictionary containing your parameters.
                    parameters['W' + str(l)] = Wl
                    parameters['b' + str(l)] = bl
    
    Returns:
    v -- python dictionary containing the current velocity.
                    v['dW' + str(l)] = velocity of dWl
                    v['db' + str(l)] = velocity of dbl
    """
    
    L = len(parameters) // 2 # number of layers in the neural networks
    v = {}
    
    # Initialize velocity
    for l in range(L):
        ### START CODE HERE ### (approx. 2 lines)
        v["dW" + str(l+1)] = np.zeros((parameters["W" + str(l+1)].shape[0],parameters["W" + str(l+1)].shape[1]))
        v["db" + str(l+1)] = np.zeros((parameters["b" + str(l+1)].shape[0],parameters["b" + str(l+1)].shape[1]))
        ### END CODE HERE ###
        
    return v

parameters = initialize_velocity_test_case()

v = initialize_velocity(parameters)
print("v[\"dW1\"] = " + str(v["dW1"]))
print("v[\"db1\"] = " + str(v["db1"]))
print("v[\"dW2\"] = " + str(v["dW2"]))
print("v[\"db2\"] = " + str(v["db2"]))

v["dW1"] = [[ 0.  0.  0.]
 [ 0.  0.  0.]]
v["db1"] = [[ 0.]
 [ 0.]]
v["dW2"] = [[ 0.  0.  0.]
 [ 0.  0.  0.]
 [ 0.  0.  0.]]
v["db2"] = [[ 0.]
 [ 0.]
 [ 0.]]

Expected Output:

Exercise: Now, implement the parameters update with momentum. The momentum update rule is, for $l = 1, ..., L$:

$ (3) { v d W [ l ] = β v d W [ l ] + ( 相關推薦 deeplearning.ai課程作業：Course 1 Week 2 deeplearning.ai課程作業：Course 1 Week 2 原始作業在GitHub上下載，本文僅作為本人學習過程的記錄，含答案，不喜勿看。全部自己跑過，保證可行。 Part 1：Python Basics with Numpy (optional assignment) deeplearning.ai課程作業：Course 1 Week 4 deeplearning.ai課程作業：Course 1 Week 4 含答案，不喜勿看！謝謝 Part1： Building your Deep Neural Network: Step by Step Welcome to your week 4 assignment (pa deeplearning.ai課程作業：Course 1 Week 3 deeplearning.ai課程作業：Course 1 Week 3 第三週的作業，親測在Coursera上可行，把資料集下載到自己電腦上單獨啟動jupyter來測試就 可行不怎麼可行，問題主要出在scatter函式裡面的引數c=Y在Coursera上是沒問題的，在自己電腦上跑就有問題 吳恩達深度學習課程deeplearning.ai課程作業：Class 4 Week 2 Residual Networks 吳恩達deeplearning.ai課程作業，自己寫的答案。 補充說明： 1. 評論中總有人問為什麼直接複製這些notebook執行不了？請不要直接複製貼上，不可能執行通過的，這個只是notebook中我們要自己寫的那部分，要正確執行還需要其他py檔案，請 吳恩達深度學習課程deeplearning.ai課程作業：Class 4 Week 2 Keras 吳恩達deeplearning.ai課程作業，自己寫的答案。 補充說明： 1. 評論中總有人問為什麼直接複製這些notebook執行不了？請不要直接複製貼上，不可能執行通過的，這個只是notebook中我們要自己寫的那部分，要正確執行還需要其他py檔案，請 deeplearning.ai課程作業：Course 2 Week 3 deeplearning.ai課程作業：Course 2 Week 3 TensorFlow Tutorial Welcome to this week’s programming assignment. Until now, you’ve always used numpy to deeplearning.ai課程作業：Course 2 Week 2 deeplearning.ai課程作業：Course 2 Week 2 Optimization Methods Until now, you’ve always used Gradient Descent to update the parameters and minimize 吳恩達深度學習課程deeplearning.ai課程作業：Class 1 Week 4 assignment4_2 吳恩達deeplearning.ai課程作業，自己寫的答案。 補充說明： 1. 評論中總有人問為什麼直接複製這些notebook執行不了？請不要直接複製貼上，不可能執行通過的，這個只是notebook中我們要自己寫的那部分，要正確執行還需要其他py檔案，請 吳恩達深度學習課程deeplearning.ai課程作業：Class 4 Week 3 Car detection 吳恩達deeplearning.ai課程作業，自己寫的答案。 補充說明： 1. 評論中總有人問為什麼直接複製這些notebook執行不了？請不要直接複製貼上，不可能執行通過的，這個只是notebook中我們要自己寫的那部分，要正確執行還需要其他py檔案， 吳恩達深度學習課程deeplearning.ai課程作業：Class 1 Week 3 assignment3 吳恩達deeplearning.ai課程作業，自己寫的答案。 補充說明： 1. 評論中總有人問為什麼直接複製這些notebook執行不了？請不要直接複製貼上，不可能執行通過的，這個只是notebook中我們要自己寫的那部分，要正確執行還需要其他py檔案，請 吳恩達深度學習課程deeplearning.ai課程作業：Class 1 Week 4 assignment4_1 吳恩達deeplearning.ai課程作業，自己寫的答案。 補充說明： 1. 評論中總有人問為什麼直接複製這些notebook執行不了？請不要直接複製貼上，不可能執行通過的，這個只是notebook中我們要自己寫的那部分，要正確執行還需要其他py檔案，請 deeplearning.ai課程作業：Convolutional Neural Networks- Course 4 Week 1 deeplearning.ai課程作業：Convolutional Neural Networks- Course 4 Week 1 Part 1 Convolutional Neural Networks: Step by Step Welcome to Course 4’s 吳恩達深度學習課程deeplearning.ai課程作業：Class 2 Week 3 TensorFlow Tutorial 吳恩達deeplearning.ai課程作業，自己寫的答案。 補充說明： 1. 評論中總有人問為什麼直接複製這些notebook執行不了？請不要直接複製貼上，不可能執行通過的，這個只是notebook中我們要自己寫的那部分，要正確執行還需要其他py檔案，請 deeplearning.ai課程作業：Convolutional Neural Networks- Course 4 Week4 deeplearning.ai課程作業：Convolutional Neural Networks- Course 4 Week4 Part 1 Face Recognition for the Happy House Welcome to the first assignmen deeplearning.ai課程作業：Convolutional Neural Networks- Course 4 Week3 deeplearning.ai課程作業：Convolutional Neural Networks- Course 4 Week3 Autonomous driving - Car detection Welcome to your week 3 programming assign 吳恩達deep learning課程作業：Class 4 Week 3 Car detection 作業題是基於YOLO框架做的一個簡單的車輛檢測網路。從課程的學習和練習可以讓我們基本掌握YOLO框架的思路和程式碼的大致編碼，想遷移YOLO框架做自己目標檢測任務的同學可以嘗試自己寫一下作業。在這裡記錄一下練習的作業吧。 1、下載作業題，並將h5檔案放置到到model_data中。h5檔案是包含 吳恩達Coursera深度學習課程 DeepLearning.ai 程式設計作業——Regularization（2-1.2） 如果資料集沒有很大，同時在訓練集上又擬合得很好，但是在測試集的效果卻不是很好，這時候就要使用正則化來使得其擬合能力不會那麼強。 import numpy as np import sklearn import matplotlib.pyplot as plt 吳恩達Coursera深度學習課程 DeepLearning.ai 程式設計作業——Optimization Methods（2-2） Optimization Methods Until now, you’ve always used Gradient Descent to update the parameters and minimize the cost. In this noteboo 吳恩達 DeepLearning.ai 課程提煉筆記（4-2）卷積神經網絡 --- 深度卷積模型 mes and shift abd actual 應用 比賽 特征 通道 1. 經典的卷積網絡 介紹幾種經典的卷積神經網絡結構，分別是LeNet、AlexNet、VGGNet。 LeNet-5： LeNet-5主要是針對灰度設計的，所以其輸入較小，為 ，其結構如下： deeplearning.ai課程學習（2） 第二週：神經網路的程式設計基礎(Basics of Neural Network programming) 1、邏輯迴歸的代價函式（Logistic Regression Cost Function） 邏輯迴歸需要注意的兩個點是，sigmoid函式和log損失函式。   sigmoid函 $