經驗分享(9)yarn重要配置yarn.nodemanager.local-dirs
yarn中有一個比較重要的配置yarn.nodemanager.local-dirs,如果配置的不好,在飽和狀態執行下叢集會出現很多問題:
1 預設配置${hadoop.tmp.dir}/nm-local-dir,系統盤通常只有幾百G,配置在這裡會經常報磁碟空間不足的錯誤;
2 配置到其中1個數據盤,比如/data0/,這個資料盤會經常沒有響應,或者直接壞掉;
3 正確的配置是配置到所有的資料盤,通常是12個,其實yarn的description寫的很清楚
<property>
<description>List of directories to store localized files in. An
application's localized file directory will be found in:
${yarn.nodemanager.local-dirs}/usercache/${user}/appcache/application_${appid}.
Individual containers' work directories, called container_${contid}, will
be subdirectories of this.
</description>
<name>yarn.nodemanager.local-dirs</name>
<value>${hadoop.tmp.dir}/nm-local-dir</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/tmp/hadoop-tmp</value>
<description>A base for other temporary directories.</description>
</property>