1. 程式人生 > >Ansible入門篇(七):Hadoop自動化安裝

Ansible入門篇(七):Hadoop自動化安裝

Hadoop是一個由Apache基金會所開發的分散式系統基礎架構,主要用於儲存和計算。

Hadoop安裝包下載

Hadoop官網下載對應版本的安裝包,本文以hadoop 2.7.5為例。

將下載好的hadoop-2.7.5.tar.gz上傳到/opt/ansible/roles/files目錄下,本文將所有tar安裝包都進行了重新命名操作,比如將hadoop-2.7.5.tar.gz解壓後重命名為hadoop並打成hadoop.tar.gz包。

 

編寫主機hosts

編輯/opt/ansible/hosts檔案新增部署Zookeeper機器IP

[hadoop_servers]
172.18.18.120 namenode_active=true namenode_standby=false datanode=true
172.18.18.121 namenode_active=false namenode_standby=true datanode=true
172.18.18.122 namenode_active=false namenode_standby=false datanode=true 

 

編寫Hadoop配置檔案

將hadoop-2.7.5.tar.gz解壓後的配置檔案hadoop-env.sh、core-site.xml、hdfs-site.xml及slaves修改成自己的配置並重命名為hadoop-env.sh.j2、core-site.xml.j2、hdfs-site.xml.j2及slaves.hadoop.j2,然後上傳到/opt/ansible/roles/templates/ 目錄下。由於配置檔案較多,本文不進行內容展示。

 

編寫引數配置main.yml 

編輯/opt/ansible/roles/vars/main.yml檔案,新增變數  

BigdataDir: /opt/hzgc  

AnsibleDir: /opt/ansible

 

編寫執行檔案hadoop.yml 

 編輯新建/opt/ansible/yml/hadoop.yml檔案,新增內容如下: 

- hosts: hadoop_servers
  remote_user: root
  roles:
  - roles

 

  tasks:
   - name: install dependency package
     yum: name={{ item }} state=present
     with_items:
     - openssh
     - rsync
   - name: delete {{BigdataDir}}/hadoop
     file: path={{BigdataDir}}/hadoop/ state=absent
   - name: copy and unzip hadoop
     unarchive: src={{AnsibleDir}}/roles/files/hadoop.tar.gz dest={{BigdataDir}}
   - name: create hadoop logs directory
     file: dest={{BigdataDir}}/hadoop/logs mode=0775 state=directory
   - name: install configuration file hadoop-env.sh.j2 for hadoop
     template: src={{AnsibleDir}}/roles/templates/hadoop-env.sh.j2 dest={{BigdataDir}}/hadoop/etc/hadoop/hadoop-env.sh
   - name: install configuration file core-site.xml.j2 for hadoop
     template: src={{AnsibleDir}}/roles/templates/core-site.xml.j2 dest={{BigdataDir}}/hadoop/etc/hadoop/core-site.xml
   - name: install configuration file hdfs-site.xml.j2 for hadoop
     template: src={{AnsibleDir}}/roles/templates/hdfs-site.xml.j2 dest={{BigdataDir}}/hadoop/etc/hadoop/hdfs-site.xml
   - name: install configuration file slaves.j2 for hadoop
     template: src={{AnsibleDir}}/roles/templates/slaves.hadoop.j2 dest={{BigdataDir}}/hadoop/etc/hadoop/slaves
   - name: change shell sbin file
     file: dest={{BigdataDir}}/hadoop/sbin mode=0755 recurse=yes
   - name: change shell bin file
     file: dest={{BigdataDir}}/hadoop/bin mode=0755 recurse=yes
   - name: start journalnode
     shell: sh {{BigdataDir}}/hadoop/sbin/hadoop-daemon.sh start journalnode
     become: true
     become_method: su
     become_user: root
     when: datanode == "true"
   - name: format active namenode hdfs
     shell: sh {{BigdataDir}}/hadoop/bin/hdfs namenode -format -force
     become: true
     become_method: su
     become_user: root
     when: namenode_active == "true"
   - name: start active namenode hdfs
     shell: sh {{BigdataDir}}/hadoop/sbin/hadoop-daemon.sh start namenode
     become: true
     become_method: su
     become_user: root
     when: namenode_active == "true"
   - name: format standby namenode hdfs
     shell: sh {{BigdataDir}}/hadoop/bin/hdfs namenode -bootstrapStandby -force
     become: true
     become_method: su
     become_user: root
     when: namenode_standby == "true"
   - name: stop active namenode hdfs
     shell: sh {{BigdataDir}}/hadoop/sbin/hadoop-daemon.sh stop namenode
     become: true
     become_method: su
     become_user: root
     when: namenode_active == "true"
   - name: format ZKFC
     shell: sh {{BigdataDir}}/hadoop/bin/hdfs zkfc -formatZK -force
     become: true
     become_method: su
     become_user: root
     when: namenode_active == "true"
   - name: stop journalnode
     shell: sh {{BigdataDir}}/hadoop/sbin/hadoop-daemon.sh stop journalnode
     become: true
     become_method: su
     become_user: root
     when: datanode == "true"
   - name: start namenode
     shell: sh {{BigdataDir}}/hadoop/sbin/start-dfs.sh
     become: true
     become_method: su
     become_user: root
     when: namenode_active == "true"
     tags:
      - start hadoop

 

執行Hadoop安裝部署 及啟動  

ansible-playbook -i hosts  yml/hadoop.yml  

檢視是否安裝成功:jps 

注:因為hadoop依賴zookeeper通訊,所以在安裝啟動hadoop的過程中要保證zookeeper正常啟動,zookeeper的自動化部署參考Zookeeper自動化安裝