激勵函數的目的是為了調節權重和誤差。

relu

　　max(0，x)

relu6

　　min(max(0，x)，6)

sigmoid

　　1/(1+exp(-x))

tanh

　　((exp(x)-exp(-x))/(exp(x)+exp(-x))

　　雙曲正切函數的值域是(-1,1)

softsign

　　x/(abs(x)+1)

softplus

　　log(exp(x)+1)

elu

　　(exp(x)+1)if x<0 else x

import math
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

x = np.linspace(-10, 10, 500)

relu = list(map(lambda m: max(0, m), x))
relu6 = list(map(lambda m: min(max(0, m), 6), x))
sigmoid = 1 / (np.exp(-x) + 1)
tanh = (np.exp(x) - np.exp(-x)) / (np.exp(x) + np.exp(-x))
softsign = x / (np.abs(x) + 1)
softplus = np.log(np.exp(x) + 1)
elu = list(map(lambda m: math.exp(m) + 1 if m < 0 else m, x))


data = {
    ‘relu‘: relu,
    ‘relu6‘: relu6,
    ‘sigmoid‘: sigmoid,
    ‘tanh‘: tanh,
    ‘softsign‘: softsign,
    ‘softplus‘: softplus,
    ‘elu‘: elu
}
df = pd.DataFrame(data, index=x)
# print(df)


df[["relu", "relu6"]].plot(
    kind="line", grid=True,
    style={"relu": "y-", "relu6": "r:"},
    yticks=np.linspace(-1, 8, 10),
    xlim=[-10, 10], ylim=[-1, 8])

df[["softplus", "elu"]].plot(
    kind="line", grid=True,
    style={"softplus": "y-", "elu": "m:"},
    yticks=np.linspace(-1, 8, 10),
    xlim=[-10, 10], ylim=[-1, 8])

df[["sigmoid", "tanh", "softsign"]].plot(
    kind="line", grid=True,
    yticks=np.linspace(-1.5, 1.5, 7),
    xlim=[-10, 10], ylim=[-1.5, 1.5])

plt.show()

Tensorflow中神經網絡的激活函數