中drop用法_深度學習中“drop out”的一些trick
阿新 • • 發佈:2021-01-16
技術標籤:中drop用法
1、dropout用法
def dropout(x, keep_prob, noise_shape=None, seed=None, name=None)
其中:
x 為神經元輸出結果
keep_prob 為被保留神經元佔的比重
tensorflow原始碼:
def dropout(x, keep_prob, noise_shape=None, seed=None, name=None): # pylint: disable=invalid-name """Computes dropout. With probability `keep_prob`, outputs the input element scaled up by `1 / keep_prob`, otherwise outputs `0`. The scaling is so that the expected sum is unchanged. By default, each element is kept or dropped independently. If `noise_shape` is specified, it must be [broadcastable](http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html) to the shape of `x`, and only dimensions with `noise_shape[i] == shape(x)[i]` will make independent decisions. For example, if `shape(x) = [k, l, m, n]` and `noise_shape = [k, 1, 1, n]`, each batch and channel component will be kept independently and each row and column will be kept or not kept together. Args: x: A floating point tensor. keep_prob: A scalar `Tensor` with the same type as x. The probability that each element is kept. noise_shape: A 1-D `Tensor` of type `int32`, representing the shape for randomly generated keep/drop flags. seed: A Python integer. Used to create random seeds. See `tf.set_random_seed` for behavior. name: A name for this operation (optional). Returns: A Tensor of the same shape of `x`. Raises: ValueError: If `keep_prob` is not in `(0, 1]` or if `x` is not a floating point tensor. """ with ops.name_scope(name, "dropout", [x]) as name: x = ops.convert_to_tensor(x, name="x") if not x.dtype.is_floating: raise ValueError("x has to be a floating point tensor since it's going to" " be scaled. Got a %s tensor instead." % x.dtype) if isinstance(keep_prob, numbers.Real) and not 0 < keep_prob <= 1: raise ValueError("keep_prob must be a scalar tensor or a float in the " "range (0, 1], got %g" % keep_prob) # Early return if nothing needs to be dropped. if isinstance(keep_prob, float) and keep_prob == 1: return x if context.executing_eagerly(): if isinstance(keep_prob, ops.EagerTensor): if keep_prob.numpy() == 1: return x else: keep_prob = ops.convert_to_tensor( keep_prob, dtype=x.dtype, name="keep_prob") keep_prob.get_shape().assert_is_compatible_with(tensor_shape.scalar()) # Do nothing if we know keep_prob == 1 if tensor_util.constant_value(keep_prob) == 1: return x noise_shape = _get_noise_shape(x, noise_shape) # uniform [keep_prob, 1.0 + keep_prob) random_tensor = keep_prob random_tensor += random_ops.random_uniform( noise_shape, seed=seed, dtype=x.dtype) # 0. if [keep_prob, 1.0) and 1. if [1.0, 1.0 + keep_prob) binary_tensor = math_ops.floor(random_tensor) ret = math_ops.div(x, keep_prob) * binary_tensor if not context.executing_eagerly(): ret.set_shape(x.get_shape()) return ret
依據tensorflow原始碼分析dropout的原理:
1)keep_prob為神經元輸出保留的概率,若keep_prob=1,則神經元輸出全部保留,具體見程式碼如下:
# Early return if nothing needs to be dropped. if isinstance(keep_prob, float) and keep_prob == 1: return x if context.executing_eagerly(): if isinstance(keep_prob, ops.EagerTensor): if keep_prob.numpy() == 1: return x else: keep_prob = ops.convert_to_tensor( keep_prob, dtype=x.dtype, name="keep_prob") keep_prob.get_shape().assert_is_compatible_with(tensor_shape.scalar()) # Do nothing if we know keep_prob == 1 if tensor_util.constant_value(keep_prob) == 1: return x
2)若keepprov不等於0, 則有一些神經元將會被淘汰,但是為了保證整個網路輸出不受影響,我們只將保留的神經元作為輸出均值,再利用保留概率,算出等價的網路總輸出,進而保證訓練與測試結果的一致性。即y = y/keepprob,具體見程式碼如下:
# 0. if [keep_prob, 1.0) and 1. if [1.0, 1.0 + keep_prob) binary_tensor = math_ops.floor(random_tensor) ret = math_ops.div(x, keep_prob) * binary_tensor if not context.executing_eagerly(): ret.set_shape(x.get_shape()) return ret
3)drop_out使用時一定要區分訓練與測試過程,因為訓練是為了得到種類多的小規模特徵提取方法,而測試需要結合全部小規模特徵提取方法,得到一些綜合特徵。
定義place_holder
keep_prob = tf.placeholder(tf.float32)
呼叫訓練優化器時:
sess.run(train_step, feed_dict={xs: X_train, ys: y_train, keep_prob: 0.5})
執行前向計算,而不優化引數時:
train_result = sess.run(merged, feed_dict={xs: X_train, ys: y_train, keep_prob: 1})
test_result = sess.run(merged, feed_dict={xs: X_test, ys: y_test, keep_prob: 1})