1. 程式人生 > >SpirngXD on YARN模式部署說明

SpirngXD on YARN模式部署說明

Spring XD可以on yarn執行。

下載on yarn執行包並解壓

wget http://repo.spring.io/release/org/springframework/xd/spring-xd/1.3.1.RELEASE/spring-xd-1.3.1.RELEASE-yarn.zip
unzip spring-xd-1.3.1.RELEASE-yarn.zip

部署本地Hadoop環境

要on yarn執行需要依賴本地的Hadoop環境。下載Hadoop安裝包並解壓。筆者試驗環境中已經提供了針對Hadoop的自動化配置指令碼,這裡不再贅述。

xd on yarn配置

修改servers.yml

配置檔案,配置siteYarnAppClasspathsiteMapreduceAppClasspath

spring:
  yarn:
    siteYarnAppClasspath: "$HADOOP_HOME/etc/hahoop,$HADOOP_HOME/share/hadoop/common/*,$HADOOP_HOME/share/hadoop/common/lib/*,$HADOOP_HOME/share/hadoop/hdfs/*,$HADOOP_HOME/share/hadoop/hdfs/lib/*,$HADOOP_HOME/share/hadoop/yarn/*,$HADOOP_HOME/share/hadoop/yarn/lib/*"
siteMapreduceAppClasspath: "$HADOOP_HOME/share/hadoop/mapreduce/*,$HADOOP_HOME/share/hadoop/mapreduce/lib/*"

配置xd相關選項

可配置在yarn上執行幾個admin幾個container等,這裡採用預設配置:

xd:
  appmasterMemory: 512M
  adminServers: 1
  adminMemory: 512M
  adminJavaOpts: -XX:MaxPermSize=128m
  adminLocality: false
  containers: 3
containerMemory: 512M containerJavaOpts: -XX:MaxPermSize=128m containerLocality: false

配置yarn環境地址

spring:
  hadoop:
    fsUri: hdfs://10.200.48.52:8020
    resourceManagerHost: 10.200.48.52
    resourceManagerPort: 8032

配置Zookeeper地址:

zk:
  namespace: xdonyarn
  client:
    connect: 10.200.48.66:2181,10.200.48.67:2181,10.200.48.68:2181

與分散式環境相同,配置redis,jdbc等資訊,這裡略過。

部署到yarn

bin/xd-yarn push

列出已安裝的應用

bin/xd-yarn pushed
  NAME  PATH
  ----  --------------------
  app   hdfs://node1:8020/xd

啟動應用

bin/xd-yarn submit

檢視狀態資訊

bin/xd-yarn submitted
  APPLICATION ID                  USER  NAME       QUEUE    TYPE  STARTTIME         FINISHTIME  STATE    FINALSTATUS  ORIGINAL TRACKING URL
  ------------------------------  ----  ---------  -------  ----  ----------------  ----------  -------  -----------  -------------------------
  application_1472450074786_0046  rc    spring-xd  default  XD    10/28/16 2:46 PM  N/A         RUNNING  UNDEFINED    http://10.200.48.54:46604

檢視admin-ui地址

bin/xd-yarn admininfo
Admins: [http://10.200.48.55:47832]

通過瀏覽器訪問

用瀏覽器開啟地址http://10.200.48.55:47832/admin-ui 即可訪問到springxd的ui介面。至此,部署完成。

通過xd-shell連線到管理節點

bin/xd-shell

然後在xd-shell下執行

xd:>admin config server --uri http://10.200.48.55:47832
Successfully targeted http://10.200.48.55:47832

即可像分散式模式一樣執行操作。