神經機器翻譯工具Nematus

1、資料預處理 ./preprocess.sh

主要流程包括：

• tokenization（符號化處理）
  

  This means that spaces have to be inserted between (e.g.) words and punctuation.

#!/bin/sh

# suffix of source language files
SRC=en
SRCTAG=en
# suffix of target language files
TRG=es
TRGTAG=es
# number of merge operations. Network vocabulary should be slightly larger (to include characters),
 

# or smaller if the operations are learned on the joint vocabulary
#bpe_operations=89500
bpe_operations=45000

tools=tools
# path to moses decoder: https://github.com/moses-smt/mosesdecoder
mosesdecoder=$tools/moses
# path to subword segmentation scripts: https://github.com/rsennrich/subword-nmt
subword_nmt=$tools/subword-nmt 

# tokenize
 

for prefix in train dev
 do
   cat data/$prefix.$SRCTAG | \
   $mosesdecoder/scripts/tokenizer/normalize-punctuation.perl -l $SRC | \
   $tools/normalise-romanian.py | \
   $tools/remove-diacritics.py | \
   $tools/tokenizer.perl -a -l $SRC > data/$prefix.tok.$SRCTAG

   cat data/$prefix.$TRGTAG | \
   $mosesdecoder/scripts/tokenizer/normalize-punctuation.perl -l $TRG | \
   $tools/tokenizer.perl -a -l $TRG > data/$prefix.tok.$TRGTAG

 done
 

• clean(去除長度大於50的句子)
  

  Long sentences and empty sentences are removed as they can cause problems with the training pipeline, and obviously mis-aligned sentences are removed.

#############################################
# clean empty and long sentences, and sentences with high source-target ratio (training corpus only)
$mosesdecoder/scripts/training/clean-corpus-n.perl data/train.tok $SRCTAG $TRGTAG data/train.tok.clean 1 50 #80

• truecase（起始詞首字母小寫,名字除外）
  

  The initial words in each sentence are converted to their most probable casing. This helps reduce data sparsity.

# train truecaser
$mosesdecoder/scripts/recaser/train-truecaser.perl -corpus data/train.tok.clean.$SRCTAG -model model/truecase-model.$SRCTAG
$mosesdecoder/scripts/recaser/train-truecaser.perl -corpus data/train.tok.clean.$TRGTAG -model model/truecase-model.$TRGTAG

# apply truecaser (cleaned training corpus)
for prefix in train
 do
  $tools/scripts/truecase.perl -model model/truecase-model.$SRCTAG < data/$prefix.tok.clean.$SRCTAG > data/$prefix.tc.$SRCTAG
  $tools/scripts/truecase.perl -model model/truecase-model.$TRGTAG < data/$prefix.tok.clean.$TRGTAG > data/$prefix.tc.$TRGTAG
 done

# apply truecaser (dev/test files)
for prefix in dev
 do
  $tools/scripts/truecase.perl -model model/truecase-model.$SRCTAG < data/$prefix.tok.$SRCTAG > data/$prefix.tc.$SRCTAG
  $tools/scripts/truecase.perl -model model/truecase-model.$TRGTAG < data/$prefix.tok.$TRGTAG > data/$prefix.tc.$TRGTAG
 done

• bpe（使用BPE演算法對單詞進行編碼）
  
  • 處理前：Lea Michele ’ s On Her Way To The Top Of The Charts With This Song !
  • 處理後：[email protected]@ a Michele ’ s On Her Way To The Top Of The [email protected]@ ts With This Song !
    

# train BPE   源語言和目標語言分別單獨訓練 bpe
cat data/train.tc.$SRCTAG | $subword_nmt/learn_bpe.py -s $bpe_operations > model/$SRCTAG.bpe
cat data/train.tc.$TRGTAG | $subword_nmt/learn_bpe.py -s $bpe_operations > model/$TRGTAG.bpe

# apply BPE

for prefix in train dev
 do
  $tools/scripts/apply_bpe.py -c model/$SRCTAG.bpe < data/$prefix.tc.$SRCTAG > data/$prefix.bpe.$SRCTAG
  $tools/scripts/apply_bpe.py -c model/$TRGTAG.bpe < data/$prefix.tc.$TRGTAG > data/$prefix.bpe.$TRGTAG
 done

• build vocabulary（建立詞彙表）

# build network dictionary
python $tools/build_dictionary.py data/train.bpe.$SRCTAG data/train.bpe.$TRGTAG

2、配置網路的超引數（Hypeparameter） ./config.py

主要引數包括：

• reload: 是否過載模型，隔一段時間儲存模型防止程式異常中斷，比如停電等。
• dim_word: 詞向量維數
• dim: 隱狀態維數，或者稱之為隱狀態大小（hidden unit size）
• n_words_src: 源語言詞彙表大小
• n_words_tgt: 目標語言詞彙表大小
• decay_c: 正則化因子λ$\lambda$
• patience: 用於 early stop
• lrate: 初始學習率
• optimizer: 優化器
• maxlen: 訓練語料句子最大長度
• batch_size: 批量大小，若GPU視訊記憶體很小應該設定儘量小的值，防止出現out of memory
• datasets: 訓練集
• valid_datasets: 驗證集
• dictionaries: 詞彙表存放的檔案
• validFreq: 驗證的頻率
• dispFreq: 輸出提示資訊的頻率
• saveFreq: 儲存模型的頻率
• sampleFreq: 驗證集輸出頻率（即輸出一些源語言句子、目標語言句子、參考譯文的一些示例）
• use_dropout: 是否使用dropout
• overwrite: 是否儲存模型的中間結果，False儲存，True不儲存
• external_validation_script: 驗證時呼叫的指令碼程式

import numpy
import os
import sys

#此處根據需要修改，看詞彙表的個數多少
VOCAB_SIZE_SRC = 40000
VOCAB_SIZE_TGT = 40000
SRCTAG = "en"
TRGTAG = "es"
DATA_DIR = "data"
TUNING_DIR = "tuning"
model = "model_en_es"

from nematus.nmt import train


if __name__ == '__main__':
    validerr = train(saveto=model+'/model.npz',
                    reload_=True,
                    dim_word=500,
                    dim=1024,
                    n_words_tgt=VOCAB_SIZE_TGT,
                    n_words_src=VOCAB_SIZE_SRC,
                    decay_c=0.,
                    clip_c=1.,
                    patience=10, #early stop patience
                    lrate=0.0001,
                    optimizer='adam', #adam,adadelta
                    maxlen=50,
                    batch_size=80,
                    valid_batch_size=80,
                    datasets=[DATA_DIR + '/train.bpe.' + SRCTAG, DATA_DIR + '/train.bpe.' + TRGTAG],
                    valid_datasets=[DATA_DIR + '/dev.bpe.' + SRCTAG, DATA_DIR + '/dev.bpe.' + TRGTAG],
                    dictionaries=[DATA_DIR + '/train.bpe.' + SRCTAG + '.json',DATA_DIR + '/train.bpe.' + TRGTAG + '.json'],
                    validFreq=10000, #10000,3000
                    dispFreq=1000,  #1000,100
                    saveFreq=30000, #30000,10000
                    #sampleFreq=10000,
                    sampleFreq=0,  #不產生樣本
                    use_dropout=True,
                    dropout_embedding=0.2, # dropout for input embeddings (0: no dropout)
                    dropout_hidden=0.2, # dropout for hidden layers (0: no dropout)
                    dropout_source=0.1, # dropout source words (0: no dropout)
                    dropout_target=0.1, # dropout target words (0: no dropout)
                    overwrite=False,
                    external_validation_script='./validate.sh')
    print validerr