When the network's evaluation indicators have not improved, we can improve the network' performance by reducing the learning rate. Class used as follow:

class torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode='min', factor=0.1, patience=10, verbose=False, threshold=0.0001, threshold_mode='rel', cooldown=0, min_lr=0, eps=1e-08)

Where: 

· optimizer: the optimizer of the network.

· mode: the type of data is str, and the parameter can choose 'min' or 'max'. The 'min' indicates that the learning rate will decrease when the monitoring amount stops decreasing, and the 'max' indicates that the learning rate will decrease when the monitoring amount stop increasing. Default is 'min'.

· factor: how much to decrease the learning rate each time, new_lr = old_lr * factor.

· patience: tolerate the number of times that the performance of the network does not improve, and reduce the learning rate above this number of times.

· verbose: if 'True', output a message to stdout for each update. Default: False.

· threshold: measure the new best value's threshold, focusing only on significant changes. Default: 1e-4.

· threshold_mode: the parameter can choose 'rel' or 'abs'. If 'rel', dynamic_threshold = best * ( 1 + threshold ) when mode is 'max' and dynamic_threshold = best * ( 1 - threshold ) when mode is 'min'. If 'abs', dynamic_threshold = best + threshold when mode is 'max' and dynamic_threshold = best - threshold when mode is 'min'. Default: 'rel'.

· cooldown: the number of epochs to wait before resuming normal operation after reducing learning rate. Default: 0.

· min_lr: the lower bound of the learning rate.

· eps: minimum attenuation to apply to learning rate. If the difference between the old and new learning is less than eps, the update is ignored. Default: 1e-8.

Notice: 

When using torch.optim.lr_scheduler.ReduceLROnPlateau(), you need to select the metrics of the network, and use the step method of the following class to achieve. For example: 

optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
scheduler = ReduceLROnPlateau(optimizer, 'min',factor=0.5, patience=4, verbose=True)
.....
scheduler.step(train_loss)
# scheduler.step(val_loss)

Reference: 

https://blog.csdn.net/weixin_40100431/article/details/84311430