SpirngXD on YARN模式部署說明
阿新 • • 發佈:2018-12-23
Spring XD可以on yarn執行。
下載on yarn執行包並解壓
wget http://repo.spring.io/release/org/springframework/xd/spring-xd/1.3.1.RELEASE/spring-xd-1.3.1.RELEASE-yarn.zip
unzip spring-xd-1.3.1.RELEASE-yarn.zip
部署本地Hadoop環境
要on yarn執行需要依賴本地的Hadoop環境。下載Hadoop安裝包並解壓。筆者試驗環境中已經提供了針對Hadoop的自動化配置指令碼,這裡不再贅述。
xd on yarn配置
修改servers.yml
spring:
yarn:
siteYarnAppClasspath: "$HADOOP_HOME/etc/hahoop,$HADOOP_HOME/share/hadoop/common/*,$HADOOP_HOME/share/hadoop/common/lib/*,$HADOOP_HOME/share/hadoop/hdfs/*,$HADOOP_HOME/share/hadoop/hdfs/lib/*,$HADOOP_HOME/share/hadoop/yarn/*,$HADOOP_HOME/share/hadoop/yarn/lib/*"
siteMapreduceAppClasspath: "$HADOOP_HOME/share/hadoop/mapreduce/*,$HADOOP_HOME/share/hadoop/mapreduce/lib/*"
配置xd相關選項
可配置在yarn上執行幾個admin幾個container等,這裡採用預設配置:
xd:
appmasterMemory: 512M
adminServers: 1
adminMemory: 512M
adminJavaOpts: -XX:MaxPermSize=128m
adminLocality: false
containers: 3
containerMemory: 512M
containerJavaOpts: -XX:MaxPermSize=128m
containerLocality: false
配置yarn環境地址
spring:
hadoop:
fsUri: hdfs://10.200.48.52:8020
resourceManagerHost: 10.200.48.52
resourceManagerPort: 8032
配置Zookeeper地址:
zk:
namespace: xdonyarn
client:
connect: 10.200.48.66:2181,10.200.48.67:2181,10.200.48.68:2181
與分散式環境相同,配置redis,jdbc等資訊,這裡略過。
部署到yarn
bin/xd-yarn push
列出已安裝的應用
bin/xd-yarn pushed
NAME PATH
---- --------------------
app hdfs://node1:8020/xd
啟動應用
bin/xd-yarn submit
檢視狀態資訊
bin/xd-yarn submitted
APPLICATION ID USER NAME QUEUE TYPE STARTTIME FINISHTIME STATE FINALSTATUS ORIGINAL TRACKING URL
------------------------------ ---- --------- ------- ---- ---------------- ---------- ------- ----------- -------------------------
application_1472450074786_0046 rc spring-xd default XD 10/28/16 2:46 PM N/A RUNNING UNDEFINED http://10.200.48.54:46604
檢視admin-ui地址
bin/xd-yarn admininfo
Admins: [http://10.200.48.55:47832]
通過瀏覽器訪問
用瀏覽器開啟地址http://10.200.48.55:47832/admin-ui 即可訪問到springxd的ui介面。至此,部署完成。
通過xd-shell連線到管理節點
bin/xd-shell
然後在xd-shell下執行
xd:>admin config server --uri http://10.200.48.55:47832
Successfully targeted http://10.200.48.55:47832
即可像分散式模式一樣執行操作。