1. 程式人生 > >Tensorflow lstm實現的小說撰寫預測

Tensorflow lstm實現的小說撰寫預測

最近,在研究深度學習方面的知識,結合Tensorflow,完成了基於lstm的小說預測程式demo。

lstm是改進的RNN,具有長期記憶功能,相對於RNN,增加了多個門來控制輸入與輸出。原理方面的知識網上很多,在此,我只是將我短暫學習的tensorflow寫一個預測小說的demo,如果有錯誤,還望大家指出。

1、將小說進行分詞,去除空格,建立詞彙表與id的字典,生成初始輸入模型的x與y

def readfile(file_path):
    f = codecs.open(file_path, 'r', 'utf-8')
    alltext = f.read()
    alltext = re.sub(r'\s','', alltext)
    seglist = list(jieba.cut(alltext, cut_all = False))
    return seglist
    
def _build_vocab(filename):
    data = readfile(filename)
    counter = collections.Counter(data)
    count_pairs = sorted(counter.items(), key=lambda x: (-x[1], x[0]))


    words, _ = list(zip(*count_pairs))
    word_to_id = dict(zip(words, range(len(words))))
    id_to_word = dict(zip(range(len(words)),words))
    dataids = []
    for w in data:
        dataids.append(word_to_id[w])
    return word_to_id, id_to_word,dataids


def dataproducer(batch_size, num_steps):
    word_to_id, id_to_word, data = _build_vocab('F:\\ml\\code\\lstm\\1.txt')
    datalen = len(data)
    batchlen = datalen//batch_size
    epcho_size = (batchlen - 1)//num_steps


    data = tf.reshape(data[0: batchlen*batch_size], [batch_size,batchlen])
    i = tf.train.range_input_producer(epcho_size, shuffle=False).dequeue()
    x = tf.slice(data, [0,i*num_steps],[batch_size, num_steps])
    y = tf.slice(data, [0,i*num_steps+1],[batch_size, num_steps])
    x.set_shape([batch_size, num_steps])
    y.set_shape([batch_size, num_steps])
    return x,y,id_to_word

2、建立lstm模型:

lstm_cell = tf.nn.rnn_cell.BasicLSTMCell(size, forget_bias = 0.5)
lstm_cell = tf.nn.rnn_cell.DropoutWrapper(lstm_cell, output_keep_prob = keep_prob)
cell = tf.nn.rnn_cell.MultiRNNCell([lstm_cell], num_layers)

3、根據訓練資料輸出誤差反向調整模型

with tf.variable_scope("Model", reuse = None, initializer = initializer):#tensorflow主要通過變數空間來實現共享變數
    with tf.variable_scope("r", reuse = None, initializer = initializer):
        softmax_w = tf.get_variable('softmax_w', [size, vocab_size])
        softmax_b = tf.get_variable('softmax_b', [vocab_size])
    with tf.variable_scope("RNN", reuse = None, initializer = initializer):
        for time_step in range(num_steps):
            if time_step > 0: tf.get_variable_scope().reuse_variables()
            (cell_output, state) = cell(inputs[:, time_step, :], state,)
            outputs.append(cell_output)
            
        output = tf.reshape(outputs, [-1,size])
        
        logits = tf.matmul(output, softmax_w) + softmax_b
        loss = tf.nn.seq2seq.sequence_loss_by_example([logits], [tf.reshape(targets,[-1])], [tf.ones([batch_size*num_steps])])
        
        global_step = tf.Variable(0)
        learning_rate = tf.train.exponential_decay(
        10.0, global_step, 5000, 0.1, staircase=True)
        optimizer = tf.train.GradientDescentOptimizer(learning_rate)
        gradients, v = zip(*optimizer.compute_gradients(loss))
        gradients, _ = tf.clip_by_global_norm(gradients, 1.25)
        optimizer = optimizer.apply_gradients(zip(gradients, v), global_step=global_step)

4、預測新一輪輸出

teststate = test_initial_state
        (celloutput,teststate)= cell(test_inputs, teststate)
        partial_logits = tf.matmul(celloutput, softmax_w) + softmax_b
        partial_logits = tf.nn.softmax(partial_logits)

5、根據之前建立的操作,執行tensorflow會話

sv = tf.train.Supervisor(logdir=None)
with sv.managed_session() as session:
    costs = 0
    iters = 0
    for i in range(1000):
        _,l= session.run([optimizer, cost])
        costs += l
        iters +=num_steps
        perplextity = np.exp(costs / iters)
        if i%20 == 0:
            print(perplextity)
        if i%100 == 0:
            p = random_distribution()
            b = sample(p)
            sentence = id_to_word[b[0]]
            for j in range(200):
                test_output = session.run(partial_logits, feed_dict={test_input:b})
                b = sample(test_output)
                sentence += id_to_word[b[0]]
            print(sentence)    

其中,使用sv.managed_session()後,在此會話間,將不能修改graph。如果採用普通的session,程式將會阻塞於session.run(),對於這個問題,我還是很疑惑,希望理解的人幫忙解答下。

程式碼地址位於https://github.com/summersunshine1/datamining/tree/master/lstm,執行時只需將readdata中檔案路徑修改即可。作為深度學習的入門小白,希望大家多多指點。

執行結果如下: