Hadoop生態圈-使用Ganglia監控flume中間件
阿新 • • 發佈:2018-10-16
作品 code ystemd png 節點 grep 開源 aci nsa
Hadoop生態圈-使用Ganglia監控flume中間件
作者:尹正傑
版權聲明:原創作品,謝絕轉載!否則將追究法律責任。
一.Ganglia監控簡介
加州伯克利大學千禧計劃的其中一個開源項目.是一個集群匯總監控用的的軟件,和Cacti不同,cacti是詳細監控集群中每臺服務器的運行狀態,而Ganglia是將集群中的服務器數據進行匯總然後監控。有時通過cacti或者zabbix看不出來的集群總體負載問題,卻能夠在Ganglia中體現。被監控的主機(即client)安裝ganglia-gmond並啟動該進程。服務器端需要安裝gmetad和web程序。大致大構圖如下:
二.部署Ganglia監控軟件
1>.安裝依賴包
[root@yinzhengjie ~]# yum -y install rrdtool perl-rrdtool rrdtool-devel apr-devel2>.安裝httpd服務與php
[root@yinzhengjie ~]# yum -y install httpd php
3>.升級yum源
[root@yinzhengjie ~]# rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm4>.安裝ganglia
[root@yinzhengjie ~]# yum -y install ganglia-gmetad ganglia-web ganglia-gmond
5>.編輯gmetad.conf配置文件
[root@yinzhengjie ~]# more /etc/ganglia/gmetad.conf | grep data_source | grep -v ^# data_source "flume120.aggrx" 10.1.2.120 [root@yinzhengjie ~]#
6>.編輯 /etc/ganglia/gmond.conf配置文件,具體修改如下圖所示。
7>.關閉防火墻和selinux
[root@yinzhengjie ~]# systemctl stop firewalld [root@yinzhengjie ~]# systemctl disable firewalld Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service. Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service. [root@yinzhengjie ~]# [root@yinzhengjie ~]# sed -i ‘s#SELINUX=enforcing#SELINUX=disabled#‘ /etc/selinux/config [root@yinzhengjie ~]# [root@yinzhengjie ~]# cat /etc/selinux/config | grep SELINUX= | grep -v ^# SELINUX=disabled [root@yinzhengjie ~]# [root@yinzhengjie ~]# getenforce Disabled [root@yinzhengjie ~]# [root@yinzhengjie ~]#
8>.啟動ganglia
[root@yinzhengjie ~]# systemctl start httpd [root@yinzhengjie ~]# [root@yinzhengjie ~]# systemctl enable httpd Created symlink from /etc/systemd/system/multi-user.target.wants/httpd.service to /usr/lib/systemd/system/httpd.service. [root@yinzhengjie ~]# [root@yinzhengjie ~]# [root@yinzhengjie ~]# systemctl start gmetad [root@yinzhengjie ~]# [root@yinzhengjie ~]# systemctl enable gmetad Created symlink from /etc/systemd/system/multi-user.target.wants/gmetad.service to /usr/lib/systemd/system/gmetad.service. [root@yinzhengjie ~]# [root@yinzhengjie ~]# [root@yinzhengjie ~]# systemctl start gmond [root@yinzhengjie ~]# [root@yinzhengjie ~]# systemctl enable gmond Created symlink from /etc/systemd/system/multi-user.target.wants/gmond.service to /usr/lib/systemd/system/gmond.service. [root@yinzhengjie ~]#
9>.創建ganglia的web UI目錄的軟連接
[root@yinzhengjie ~]# ln -s /usr/share/ganglia/ /var/www/html/ganglia [root@yinzhengjie ~]# [root@yinzhengjie ~]# ll /var/www/html/ total 0 lrwxrwxrwx 1 root root 19 Oct 16 13:49 ganglia -> /usr/share/ganglia/ [root@yinzhengjie ~]#
10>.登陸webUI(如果能看到下圖就OK啦!)
三.使用Ganglia監控flume(註意,flume-agent端的節點可以不和Ganglia不是同一臺機器喲!)
1>.編寫flume的配置文件
[root@node106 ~]# cat /yinzhengjie/data/flume/job/flume-telnet.conf # 這裏的“yinzhengjie”是agent的名稱,它是我們自定義的。我們分別給“yinzhengjie”的sources,sinks,channels的別名分別為r1,k1和c1。 yinzhengjie.sources = r1 yinzhengjie.sinks = k1 yinzhengjie.channels = c1 # 指定source的類型為netcat(source的type有很多,咱們不用記住它,需要用的時候去官網查就好),綁定source的主機是“node106.yinzhengjie.org.cn”,綁 定的端口為“8888” yinzhengjie.sources.r1.type = netcat yinzhengjie.sources.r1.bind = node106.yinzhengjie.org.cn yinzhengjie.sources.r1.port = 8888 # 指定sink的類型,我們這裏指定的為logger,即控制臺輸出。 yinzhengjie.sinks.k1.type = logger # 指定channel的類型為memory,指定channel的容量是1000,每次傳輸的容量是100 yinzhengjie.channels.c1.type = memory yinzhengjie.channels.c1.capacity = 1000 yinzhengjie.channels.c1.transactionCapacity = 100 # 綁定source和sink yinzhengjie.sources.r1.channels = c1 yinzhengjie.sinks.k1.channel = c1 [root@node106 ~]#
2>.
3>.
4>.
5>.
Hadoop生態圈-使用Ganglia監控flume中間件