雙向RNN實際上僅僅是兩個獨立的RNN放在一起, 本博文將介紹如何在tensorflow中實現雙向rnn

單層雙向rnn

tensorflow中已經提供了雙向rnn的介面,它就是tf.nn.bidirectional_dynamic_rnn(). 我們先來看一下這個介面怎麼用.

bidirectional_dynamic_rnn(
    cell_fw, #前向 rnn cell
    cell_bw, #反向 rnn cell
    inputs, #輸入序列.
    sequence_length=None,# 序列長度
    initial_state_fw=
 
None,#前向rnn_cell的初始狀態
    initial_state_bw=None,#反向rnn_cell的初始狀態
    dtype=None,#資料型別
    parallel_iterations=None,
    swap_memory=False,
    time_major=False,
    scope=None
)

返回值:一個tuple(outputs, outputs_states), 其中,outputs是一個tuple(outputs_fw, outputs_bw). 關於outputs_fw和outputs_bw,如果time_major=True

如何使用:
bidirectional_dynamic_rnn 在使用上和 dynamic_rnn是非常相似的.

1. 定義前向和反向rnn_cell
2. 定義前向和反向rnn_cell的初始狀態
3. 準備好序列
4. 呼叫bidirectional_dynamic_rnn

import tensorflow as tf
from tensorflow.contrib import rnn
cell_fw = rnn.LSTMCell(10)
cell_bw =
 
 rnn.LSTMCell(10)
initial_state_fw = cell_fw.zero_state(batch_size)
initial_state_bw = cell_bw.zero_state(batch_size)
seq = ...
seq_length = ...
(outputs, states)=tf.nn.bidirectional_dynamic_rnn(cell_fw, cell_bw, seq,
 seq_length, initial_state_fw,initial_state_bw)
out = tf.concat(outputs, 2)
# ....

多層雙向rnn

單層雙向rnn可以通過上述方法簡單的實現,但是多層的雙向rnn就不能簡單的將MultiRNNCell傳給bidirectional_dynamic_rnn了.
想要知道為什麼,我們需要看一下bidirectional_dynamic_rnn的原始碼片段.

with vs.variable_scope(scope or "bidirectional_rnn"):
  # Forward direction
  with vs.variable_scope("fw") as fw_scope:
    output_fw, output_state_fw = dynamic_rnn(
        cell=cell_fw, inputs=inputs, sequence_length=sequence_length,
        initial_state=initial_state_fw, dtype=dtype,
        parallel_iterations=parallel_iterations, swap_memory=swap_memory,
        time_major=time_major, scope=fw_scope)

這只是一小部分程式碼,但足以看出,bi-rnn實際上是依靠dynamic-rnn實現的,如果我們使用MuitiRNNCell的話,那麼每層之間不同方向之間互動就被忽略了.所以我們可以自己實現一個工具函式,通過多次呼叫bidirectional_dynamic_rnn來實現多層的雙向RNN 這是我對多層雙向RNN的一個精簡版的實現,如有錯誤,歡迎指出

bidirectional_dynamic_rnn原始碼一探

上面我們已經看到了正向過程的程式碼實現,下面來看一下剩下的反向部分的實現.
其實反向的過程就是做了兩次reverse

1. 第一次reverse:將輸入序列進行reverse,然後送入dynamic_rnn做一次運算.
2. 第二次reverse:將上面dynamic_rnn返回的outputs進行reverse,保證正向和反向輸出 對應位置的 輸入是一致的 是對上的.

def _reverse(input_, seq_lengths, seq_dim, batch_dim):
  if seq_lengths is not None:
    return array_ops.reverse_sequence(
        input=input_, seq_lengths=seq_lengths,
        seq_dim=seq_dim, batch_dim=batch_dim)
  else:
    return array_ops.reverse(input_, axis=[seq_dim])

with vs.variable_scope("bw") as bw_scope:
  inputs_reverse = _reverse(
      inputs, seq_lengths=sequence_length,
      seq_dim=time_dim, batch_dim=batch_dim)
  tmp, output_state_bw = dynamic_rnn(
      cell=cell_bw, inputs=inputs_reverse, sequence_length=sequence_length,
      initial_state=initial_state_bw, dtype=dtype,
      parallel_iterations=parallel_iterations, swap_memory=swap_memory,
      time_major=time_major, scope=bw_scope)

output_bw = _reverse(
  tmp, seq_lengths=sequence_length,
  seq_dim=time_dim, batch_dim=batch_dim)

outputs = (output_fw, output_bw)
output_states = (output_state_fw, output_state_bw)

return (outputs, output_states)

tf.reverse_sequence

對序列中某一部分進行反轉

reverse_sequence(
    input,#輸入序列,將被reverse的序列
    seq_lengths,#1Dtensor,表示輸入序列長度
    seq_axis=None,# 哪維代表序列
    batch_axis=None, #哪維代表 batch
    name=None,
    seq_dim=None,
    batch_dim=None
)

官網上的例子給的非常好,這裡就直接貼上過來:

# Given this:
batch_dim = 0
seq_dim = 1
input.dims = (4, 8, ...)
seq_lengths = [7, 2, 3, 5]

# then slices of input are reversed on seq_dim, but only up to seq_lengths:
output[0, 0:7, :, ...] = input[0, 7:0:-1, :, ...]
output[1, 0:2, :, ...] = input[1, 2:0:-1, :, ...]
output[2, 0:3, :, ...] = input[2, 3:0:-1, :, ...]
output[3, 0:5, :, ...] = input[3, 5:0:-1, :, ...]

# while entries past seq_lens are copied through:
output[0, 7:, :, ...] = input[0, 7:, :, ...]
output[1, 2:, :, ...] = input[1, 2:, :, ...]
output[2, 3:, :, ...] = input[2, 3:, :, ...]
output[3, 2:, :, ...] = input[3, 2:, :, ...]

例二:

# Given this:
batch_dim = 2
seq_dim = 0
input.dims = (8, ?, 4, ...)
seq_lengths = [7, 2, 3, 5]

# then slices of input are reversed on seq_dim, but only up to seq_lengths:
output[0:7, :, 0, :, ...] = input[7:0:-1, :, 0, :, ...]
output[0:2, :, 1, :, ...] = input[2:0:-1, :, 1, :, ...]
output[0:3, :, 2, :, ...] = input[3:0:-1, :, 2, :, ...]
output[0:5, :, 3, :, ...] = input[5:0:-1, :, 3, :, ...]

# while entries past seq_lens are copied through:
output[7:, :, 0, :, ...] = input[7:, :, 0, :, ...]
output[2:, :, 1, :, ...] = input[2:, :, 1, :, ...]
output[3:, :, 2, :, ...] = input[3:, :, 2, :, ...]
output[2:, :, 3, :, ...] = input[2:, :, 3, :, ...]

參考資料