https://hadoop.apache.org
A framework for job scheduling and cluster resource management.

一 部署

角色

ResourceManager、NodeManager

客戶端

Job、AppMaster、Container

AppMaster是一個YARN job執行時第一個由ResourceManager啟動的container，然後負責整個job的執行，包括container的申請、啟動、kill、狀態檢查等。
每種型別的job都需要提供自己的AppMaster。
YARN只提供了MapReduce型別job的AppMaster。

Spark on YARN的AppMaster：

org.apache.spark.deploy.yarn.ApplicationMaster

HA

zk節點

/yarn-leader-election/${yarn.resourcemanager.cluster-id}/ActiveBreadCrumb
/yarn-leader-election/${yarn.resourcemanager.cluster-id}/ActiveStandbyElectorLock

程式碼：

org.apache.hadoop.yarn.server.resourcemanager.EmbeddedElectorService

二 常用命令

yarn node
yarn queue -status root
yarn top
yarn application -list -appStates ALL -appTypes MapReduce|more
yarn rmadmin
yarn logs -applicationid=$applicationId

三 資源管理

計算資源

CPU、記憶體

管理

queue
scheduler

• FIFO：先進先出
• Capacity：按百分比配置
• Fair：按具體的核數和記憶體數配置

四 API

http://:8088$ip/ws/v1/cluster/metrics
http://:8088$ip/ws/v1/cluster/scheduler

http://:8088$ip/ws/v1/cluster/apps

通過API獲取指標之後對接ELK

五 資料本地性 Data Locality

Moving Computation is Cheaper than Moving Data

Spark Task分配過程

job->stage-task
resourceOffer會呼叫getAllowedLocalityLevel來獲取當前允許啟動任務的locality級別，getAllowedLocalityLevel內部通過時間delay以及是否有更多task來控制允許哪些級別，比如一開始只允許PROCESS_LOCAL級別，delay一段時間之後，再允許啟動PROCESS_LOCAL和NODE_LOCAL級別的task

Data Locality等級：PROCESS_LOCAL, NODE_LOCAL, NO_PREF, RACK_LOCAL, ANY
配置：spark.locality.wait

Impala

Efficiently retrieving data from HDFS is a challenge for all SQL-on-Hadoop systems. In order to perform data scans from both disk and memory at or near hardware speed, Impala uses an HDFS feature called short-circuit local reads to bypass the DataNode protocol when reading from local disk. Impala can read at almost disk bandwidth (approx.100MB/s per disk) and is typically able to saturate all available disks. We have measured that with 12 disks, Impala is capable of sustaining I/O at 1.2GB/sec.