最近在學習用TensorFlow開發CNN的時候，看到官方的一篇教程Creating Estimators in tf.contrib.learn，介紹了使用封裝好的一些工具開發訓練程式的方法，覺得很不錯，在這裡記錄一下學習的心得。
tf.contrib下面的東西很多，google自己沒有稱之為框架，但我感覺藉助於tf.contrib.learn.Estimator，也等於是定義了一個框架，因為好多訓練程式的流程是差不多的。

文件中建議的流程如下:

• Instantiate an Estimator 建立Estimator
• Construct a custom model function 自定義模型函式
• Configure a neural network using tf.contrib.layers 配置用到的Layers
• Choose an appropriate loss function from tf.contrib.losses 定義計算loss的方法
• Define a training op for your model 定義訓練（即收斂loss）方法
• Generate and return predictions （我怎麼覺得計算prediction應該放到loss前面）

建立Estimator

系統中已經有一些現成的，在learn.estimator目錄下，比如：

• LinearClassifier: Constructs a linear classification model.
• LinearRegressor: Constructs a linear regression model.
• DNNClassifier: Construct a neural network classification model.
• DNNRegressor: Construct a neural network regressions model.

看樣子應該是可以直接就用了，但是暫時還沒有去研究。

自定義Estimator

nn = tf.contrib.learn.Estimator(model_fn=model_fn, model_dir=None
 
, config=None, params=model_params, feature_engineering_fn=None)

• model_fn: 模型函式，具體後面再討論
• model_dir: 指定log和引數儲存檔案的位置
• config: 待研究
• params: 傳遞給model_fn的自定義引數，dict形式
• * feature_engineering_fn*: 待研究

原文的例子：

# Learning rate for the model
LEARNING_RATE = 0.001

# Set model params
model_params = {"learning_rate": LEARNING_RATE}

nn = tf.contrib.learn.Estimator(
    model_fn=model_fn, params=model_params)

定義模型函式(model_fn)

model_fn的引數是：

def model_fn(features, targets, mode, params):
   # Logic to do the following:
   # 1. Configure the model via TensorFlow operations
   # 2. Define the loss function for training/evaluation
   # 3. Define the training operation/optimizer
   # 4. Generate predictions
   return predictions, loss, train_op

• features: 訓練資料，也就是x，格式沒有統一定義，取決於演算法的需要；由呼叫fit(), evaluate(), or predict()的時候傳遞進來；
• targets: 訓練資料的介面，也就是y；fit，evaluate的時候會傳遞；
• mode： 用來區分fit/evaluate/predict
  
  • tf.contrib.learn.ModeKeys.TRAIN
  • tf.contrib.learn.ModeKeys.EVAL
  • tf.contrib.learn.ModeKeys.INFER
• params: 自定義引數，沒有可以不傳

返回結果是一個三元組：

• predictions (required in INFER and EVAL modes)
  predictions = {"results": tensor_of_predictions}
  In INFER mode, the dict that you return from model_fn will then be returned by predict(), so you can construct it in the format in which you’d like to consume it. 也就是說這個prediction dict的格式沒有要求，根據需要來定義。
  In EVAL mode, the dict is used by metric functions to compute metrics. Any MetricSpec objects passed to the metrics argument of evaluate() must have a prediction_key that matches the key name of the corresponding predictions in predictions. 看這個意思，是說evaluate的時候，要指定要用predictions中的哪個key來做比較(是不是這樣，待驗證)

• loss (required in EVAL and TRAIN mode). 這個就是通常意義的loss；

• train_op (required only in TRAIN mode). An Op that runs one step of training.

定義Hidden Layers

對於卷積神經網路來說，一般就是卷積層和池化層，tf.contrib.layers下有一些定義好的類：

• tf.contrib.layers.convolution2d
• tf.contrib.layers.max_pool2d
• tf.contrib.layers.avg_pool2d
• ….

定義全連線層

系統有一些預定義的類：

• relu(inputs, num_outputs). Create a layer of num_outputs nodes fully connected to the previous layer inputs with a ReLu activation function (tf.nn.relu):
  hidden_layer = tf.contrib.layers.relu(inputs=input_layer, num_outputs=10)

• relu6(inputs, num_outputs). Create a layer of num_outputs nodes fully connected to the previous layer hidden_layer with a ReLu 6 activation function (tf.nn.relu6):
  second_hidden_layer = tf.contrib.layers.relu6(inputs=hidden_layer, num_outputs=20)

• linear(inputs, num_outputs). Create a layer of num_outputs nodes fully connected to the previous layer second_hidden_layer with no activation function, just a linear transformation:
  output_layer = tf.contrib.layers.linear(inputs=second_hidden_layer, num_outputs=3)

當然還可以通過定製fully_connected的引數來定義自己需要的全連線層：

output_layer = tf.contrib.layers.fully_connected(inputs=second_hidden_layer, num_outputs=10, activation_fn=tf.sigmoid)


定義loss函式

tf.contrib.losses 下包含一些預定義的loss, 比如：

• absolute_difference(predictions, targets). Calculates loss using the absolute-difference formula (also known as L1 loss).
• log_loss(predictions, targets). Calculates loss using the logistic loss forumula (typically used in logistic regression).
• mean_squared_error(predictions, targets). Calculates loss using the mean squared error (MSE; also known as L2 loss).

定義train_op

一般按這樣來定義：

train_op = tf.contrib.layers.optimize_loss(
loss=loss,
global_step=tf.contrib.framework.get_global_step(),
learning_rate=params["learning_rate"],
optimizer="SGD")


• loss: 就是前面定義的loss函式
• global_step: An integer Variable representing the step counter to increment for each model training run. Can easily be created/incremented in TensorFlow via the get_global_step() function (具體作用待研究)
• learning_rate: 應該就是類似於梯度下降的步長之類的意思，比如0.01

• optimizer: 優化方法，指定在tf.contrib.layers.optimizers中定義的方法，比如

  • SGD. Implementation of gradient descent (tf.train.GradientDescentOptimizer) 梯度下降
  • Adagrad. Implementation of the AdaGrad optimization algorithm (tf.train.AdagradOptimizer)
  • Adam. Implementation of the Adam optimization algorithm (tf.train.AdamOptimizer)
  • Ftrl. Implementation of the FTRL-Proximal (“Follow The (Proximally) Regularized Leader”) algorithm (tf.train.FtrlOptimizer)
  • Momentum. Implementation of stochastic gradient descent with momentum (tf.train.MomentumOptimizer)

  • RMSProp. Implementation of the RMSprop algorithm (tf.train.RMSPropOptimizer)

    除了名稱之外，還可以有多種方法指定optimizer，參見文件。

綜合以上的內容，一個完整的model_fn的例子：

def model_fn(features, targets, mode, params):
“”“Model function for Estimator.”“”

# Connect the first hidden layer to input layer
# (features) with relu activation
first_hidden_layer = tf.contrib.layers.relu(features, 10)

#Connect the second hidden layer to first hidden layer with relu
second_hidden_layer = tf.contrib.layers.relu(first_hidden_layer, 10)

# Connect the output layer to second hidden layer (no activation fn)
output_layer = tf.contrib.layers.linear(second_hidden_layer, 1)

# Reshape output layer to 1-dim Tensor to return predictions
predictions = tf.reshape(output_layer, [-1])
predictions_dict = {“ages”: predictions}

# Calculate loss using mean squared error
loss = tf.contrib.losses.mean_squared_error(predictions, targets)

train_op = tf.contrib.layers.optimize_loss(
loss=loss,
global_step=tf.contrib.framework.get_global_step(),
learning_rate=params[“learning_rate”],
optimizer=”SGD”)

return predictions_dict, loss, train_op

訓練，評估和預測

# Fit
nn.fit(x=training_set.data, y=training_set.target, steps=5000)

# Score accuracy
ev = nn.evaluate(x=test_set.data, y=test_set.target, steps=1)
loss_score = ev["loss"]
print("Loss: %s" % loss_score) 

# Print out predictions
predictions = nn.predict(x=prediction_set.data,
                         as_iterable=True)
for i, p in enumerate(predictions):
  print("Prediction %s: %s" % (i + 1, p["ages"]))