tensorflow使用BN—Batch Normalization
阿新 • • 發佈:2019-01-01
你要的答案或許都在這裡:
自己搭建的一個框架,包含模型有:vgg(vgg16,vgg19), resnet(resnet_v2_50,resnet_v2_101,resnet_v2_152), inception_v4, inception_resnet_v2等。
注意:不要隨便加BN,有些問題加了後會導致loss變大。
(1)
訓練的時候:is_training為True。
import tensorflow as tf import numpy as np from tensorflow.python.ops import control_flow_ops from tensorflow.python.training import moving_averages def bn(x, is_training): x_shape = x.get_shape() params_shape = x_shape[-1:] axis = list(range(len(x_shape) - 1)) beta = _get_variable('beta', params_shape, initializer=tf.zeros_initializer()) gamma = _get_variable('gamma', params_shape, initializer=tf.ones_initializer()) moving_mean = _get_variable('moving_mean', params_shape, initializer=tf.zeros_initializer(), trainable=False) moving_variance = _get_variable('moving_variance', params_shape, initializer=tf.ones_initializer(), trainable=False) # These ops will only be preformed when training. mean, variance = tf.nn.moments(x, axis) update_moving_mean = moving_averages.assign_moving_average(moving_mean, mean, BN_DECAY) update_moving_variance = moving_averages.assign_moving_average(moving_variance, variance, BN_DECAY) tf.add_to_collection(UPDATE_OPS_COLLECTION, update_moving_mean) tf.add_to_collection(UPDATE_OPS_COLLECTION, update_moving_variance) mean, variance = control_flow_ops.cond( is_training, lambda: (mean, variance), lambda: (moving_mean, moving_variance)) return tf.nn.batch_normalization(x, mean, variance, beta, gamma, BN_EPSILON)
函式:
tf.nn.batch_normalization()
def batch_normalization(x,
mean,
variance,
offset,
scale,
variance_epsilon,
name=None):
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 1
- 2
- 3
- 4
- 5
- 6
- 7
Args:
- x: Input
Tensor
of arbitrary dimensionality. - mean: A mean
Tensor
. - variance: A variance
Tensor
. - offset: An offset
Tensor
, often denoted β in equations, or None. If present, will be added to the normalized tensor. - scale: A scale
Tensor
, often denoted γ in equations, orNone
. If present, the scale is applied to the normalized tensor. - variance_epsilon: A small float number to avoid dividing by 0.
- name: A name for this operation (optional).
- Returns: the normalized, scaled, offset tensor.
對於卷積,x:[bathc,height,width,depth]
對於卷積,我們要feature map中共享 γi 和 βi ,所以 γ,β的維度是[depth]