Online and Offline Reinforcement Learning by Planning with a Learned Model

阿新 • • 發佈：2021-08-07

發表時間：2021
文章要點：文章接著muzero做的，當時muzero裡面提出了一個MuZero Reanalyze（Reanalyse）的方式，這篇文章提出的MuZero Unplugged演算法其實就是把MuZero Reanalyse用到offline RL裡面。作者想說的就是這個方法不僅可以用在online RL上，在offline RL上同樣表現很好，相當於一個演算法對著這兩種設定都適用（Reanalyse fraction of 0% refers to training by only interacting with the environment, no Reanalyse of stored data, whereas a fraction of 100% refers to the fully offline case with no environment interaction at all）。MuZero Unplugged的主要想說的是，由於muzero是off-policy演算法，那麼replay buffer裡的trajectory肯定是很多不同的policy產生的，由於policy一直在被更新，那麼那些儲存較早的trajectory的policy和n-step return就不準了，所以就要用最新的網路引數去重新做樹搜尋更新policy以及target value來估n-step return（所以叫Reanalyze）。這個原理在muzero的文章中也有說，如下

然後這篇文章就是把這個方式和offline RL的演算法作對比，比如BC，DQN，IQN，BCQ，REM，CRR等。
總結：

感覺muzero確實還挺通用，而且因為它還要學model，相當於給訓練多加了很多正則項，再加上還有MCTS的planning，確實效果會robust很多。但是也能想象，這麼多網路一起訓練，算力要求不會低的。而且這個方式確實就和DQN一樣，對於offline RL來說就是把replay buffer拿來繼續train，所以通用性是可以理解的。另一個感悟是，原來deepmind自己接著自己的演算法做，也不是直接拿來用，還是要自己復現一遍啊，666。
疑問：這篇文章主要其實在講實驗，各個部分具體是咋實現的其實沒細說，從給的演算法虛擬碼也能看出來，寫的非常籠統，需要靠猜。

Online and Offline Reinforcement Learning by Planning with a Learned Model

Online and Offline Reinforcement Learning by Planning with a Learned Model

HIERARCHICAL REINFORCEMENT LEARNING BY DISCOVERING INTRINSIC OPTIONS

Offline Evaluation of Online Reinforcement Learning Algorithms

Search on the Replay Buffer: Bridging Planning and Reinforcement Learning

強化學習論文研讀（四）——Deep Reinforcement Learning with Double Q-Learning

Decoupling Value and Policy for Generalization in Reinforcement Learning

Game Theory and Multi-agent Reinforcement Learning筆記上

Improving Generalization in Reinforcement Learning with Mixture Regularization

ON THE ROLE OF PLANNING IN MODEL-BASED DEEP REINFORCEMENT LEARNING

Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning

Reinforcement Learning, Spike-Time-Dependent Plasticity, and the BCM Rule

Reinforcement Learning with Long Short-Term Memory

TREEQN AND ATREEC: DIFFERENTIABLE TREE-STRUCTURED MODELS FOR DEEP REINFORCEMENT LEARNING

lec-1-Deep Reinforcement Learning, Decision Making, and Control

Natural and Realistic Single Image Super-Resolution with Explicit Natural Manifold Discrimination

[Mysql] GROUP BY （WITH ROLLUP）& ORDER BY (IF)資料分組&條件排序

Reinforcement Learning (DQN) 中經驗池詳細解釋

論文記載： Deep Reinforcement Learning for Traffic LightControl in Vehicular Networks

MFMARL(Mean Field Multi-Agent Reinforcement Learning)實現

CLion編寫Qt報錯：could not find a package configuration file provide by “QT“ with any of the following

Online and Offline Reinforcement Learning by Planning with a Learned Model

相關推薦