tf.train.exponential_decay(
    learning_rate,
    global_step,
    decay_steps,
    decay_rate,
    staircase=False,
    name=None
)

一句話描述：對學習率learning_rate應用指數衰減。
多說點：固定的學習率總是顯得笨拙：太小速度太慢，太大又擔心得不到最優解。一個很直接的想法就是隨著訓練的進行，動態設定學習率——隨著訓練次數增加，學習率逐步減小。而tf.train.exponential_decay()就是tf內建的一個生成動態減小學習率的函式。
它的公式如下

decayed_learning_rate = learning_rate *
                        decay_rate ^ (global_step / decay_steps)

也即
$decayed\_learning\_rate = learning\_rate*\frac{global\_step}{decay\_step}$
舉個例子：
初始學習率LEARNING_RATE_BASE = 0.1

import tensorflow as tf
import matplotlib.pyplot as plt

LEARNING_RATE_BASE = 0.1
DECAY_RATE = 0.9
GLOBAL_STEPS = 1000
DECAY_STEPS = 100

global_ = tf.Variable(tf.constant(0))
learning_rate_1 = tf.train.exponential_decay(LEARNING_RATE_BASE,
 
 global_, DECAY_STEPS, DECAY_RATE, staircase=True)
learning_rate_2 = tf.train.exponential_decay(LEARNING_RATE_BASE, global_, DECAY_STEPS, DECAY_RATE, staircase=False)

LR1 = []
LR2 = []

with tf.Session() as sess:
    for i in range(GLOBAL_STEPS):
        lr1 = sess.run(learning_rate_1, feed_dict={global_: i})
        LR1.append(lr1)
        lr2 = sess.run(learning_rate_2, feed_dict={global_: i})
        LR2.append(lr2)

plt.figure(1)
plt.plot(range(GLOBAL_STEPS), LR2, 'r-')
plt.plot(range(GLOBAL_STEPS), LR1, 'b-')

plt.show()