Heimdall線下測試環境--CDH優化&服務安裝
阿新 • • 發佈：2019-01-24
CDH優化&服務安裝
MySQL
MySQL安裝包下載 選擇Linux - Generic平臺和5.6版本下載
或者直接下載5.6.34版本解除安裝舊mysql
rpm -qa | grep mysql
rpm -e --nodeps mysql-libs-5.1.73-5.el6_6.x86_64- 安裝
##解壓到/usr/local/mysql
cd /usr/local/mysql
rpm -ivh MySQL-server-5.6.34-1.linux_glibc2.5.x86_64.rpm
rpm -ivh MySQL-client-5.6.34-1. - 啟動mysql並修改密碼
檢視初始密碼
service mysql start
netstat -ntlp|grep 3306
cat /root/.mysql_secret
#登入
mysql -u root -p mysql
#使用以下命令設定root的新密碼
mysql> SET PASSWORD = PASSWORD('root');
mysql> quit- 建立資料庫並授權
#hive
mysql>create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
#activity monitor
mysql>create - mysql-connect.jar放到下面目錄下
/opt/cloudera/parcels/CDH/lib/hive/lib
/opt/cloudera/parcels/CDH-5.8.2-1.cdh5.8.2.p0.3-el6/lib/hive/lib
/usr/share/java/HDFS
- 最大傳輸執行緒數：8192
- DataNode等日誌目錄：放到掛在的1T硬碟上/data/var/log/hadoop-hdfs
- DataNode等最大日誌檔案備份：3
- Balancer的java堆疊大小：512M
- DateNode的java堆疊大小：512M
- 用於快取的最大記憶體：1G
客戶端java堆大小：512M
所有的日誌目錄均放在掛載盤（/data）
最大日誌檔案備份數：3- hdfs檔案目錄許可權設定
for host in `cat h.slaves`; do
ssh $host groupadd supergroup
ssh $host usermod -g supergroup mapred
ssh $host usermod -g supergroup hbase
ssh $host usermod -g supergroup yarn
done
##hdfs啟動使用者即為超級使用者
sudo -u hdfs hadoop fs -chmod 775 /
sudo -u hdfs hadoop fs -chown heimdall /user
...YARN
- ApplicationMaster Java最大堆疊：820M
- Map任務記憶體：512M
- Reduce任務記憶體：1G
- Map任務的最大堆疊：410M
- Reduce任務的最大堆疊：820M
- JobHistory Server的Java堆疊大小：1G
- 容器虛擬CPU核心：8
- ResourceManager的Java堆疊大小：1G
- 容器記憶體增量：512M
- 最大容器記憶體：2G
- 最大容器虛擬CPU核心數：32
Zookeeper
Hive
Gateway就是客戶端的意思
- 資料庫修改為MySQL
- 配置hive
[hive-site.xml 的 Hive 客戶端高階配置程式碼段（安全閥）]
<property>
<name>hive.cli.print.header</name>
<value>true</value>
<description>Whether to print the names of the columns</description>
</property>
<property>
<name>hive.cli.print.current.db</name>
<value>true</value>
<description>Whether to include the current database in the Hive prompt</description>
</property>Flume
a1.sources=r1
a1.channels=c1 c3
a1.sinks=k1 k3
a1.sources.r1.type=avro
a1.sources.r1.bind=0.0.0.0
a1.sources.r1.port=41414
a1.sources.r1.channels=c1 c3
a1.sources.r1.selector.type=multiplexing
a1.sources.r1.selector.header=state
a1.sources.r1.selector.mapping.normal=c1
a1.sources.r1.selector.mapping.sdk=c3
a1.sources.r1.selector.default = c1
a1.channels.c1.type=file
a1.channels.c1.checkpointDir=/data/flume/file-channel/checkpoint/normal
a1.channels.c1.dataDirs=/data/flume/file-channel/data/normal1,/data/flume/file-channel/data/normal2
a1.channels.c1.capacity=200000
a1.channels.c1.transactionCapacity=1000
a1.channels.c1.checkpointInterval=10000
a1.channels.c1.keep-alive=30
a1.channels.c3.type=file
a1.channels.c3.checkpointDir=/data/flume/file-channel/checkpoint/sdk
a1.channels.c3.dataDirs=/data/flume/file-channel/data/sdk1,/data/flume/file-channel/data/sdk2
a1.channels.c3.capacity=200000
a1.channels.c3.transactionCapacity=1000
a1.channels.c3.checkpointInterval=10000
a1.channels.c3.keep-alive=30
a1.sinks.k1.channel = c1
a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
a1.sinks.k1.brokerList=heimdall05.test.hadoop.com:9092,heimdall06.test.hadoop.com:9092,heimdall07.test.hadoop.com:9092
a1.sinks.k1.topic = mytopic
a1.sinks.k1.requiredAcks = -1
a1.sinks.k1.batchSize = 1000
a1.sinks.k1.producer.type=sync
a1.sinks.k1.custom.encoding=UTF-8
a1.sinks.k3.channel = c3
a1.sinks.k3.type = org.apache.flume.sink.kafka.KafkaSink
a1.sinks.k3.brokerList=heimdall05.test.hadoop.com:9092,heimdall06.test.hadoop.com:9092,heimdall07.test.hadoop.com:9092
a1.sinks.k3.topic = sdkTopic
a1.sinks.k3.requiredAcks = -1
a1.sinks.k3.batchSize = 1000
a1.sinks.k3.producer.type=sync
a1.sinks.k3.custom.encoding=UTF-8
Kafka
kafka安裝
- Java Heap Size of Broker:1G
安裝Kafka Manager
nohup bin/kafka-manager -Dconfig.file=/usr/kafka-manager/kafka-manager-1.3.0.7/conf/application.conf -Dhttp.port=7778 &
開啟Kafka Manager 管理頁面，新增Kafla Cluster。
新增完成後，點選列表中的Cluster，檢視topic和broker情況。
Storm
strom 分為2個主要版本0.x和1.x,安裝0.x版本需要安裝zeromq，此次安裝的是1.0.2版本，安裝過程參考Storm 1.0.2叢集安裝指導 ，0.x版本安裝請參考 centos6.4 下storm-0.8.1 安裝記錄
- 進入storm/conf目錄，修改storm.yaml配置檔案
storm.zookeeper.servers:
- "heimdall02.test.hadoop.com"
- "heimdall03.test.hadoop.com"
- "heimdall04.test.hadoop.com"
nimbus.seeds: ["heimdall05.test.hadoop.com"]
storm.local.dir: "/data/storm/live"
supervisor.slots.ports:
- 6700
- 6701- 啟動
進入storm資料夾
啟動主控節點
bin/storm nimbus
啟動工作節點 需要分別到每個節點啟動服務。
bin/storm supervisor
啟動管理頁面
bin/storm ui
-------------------
後臺啟動
nohup bin/storm nimbus &
nohup bin/storm supervisor &
nohup bin/storm ui&
ElasticSearch
- 所有節點配置elk啟
[elasticsearch.in.sh]
ES_MIN_MEM=2g
ES_MAX_MEM=2g- 所有節點配置叢集模式，配置詳解
cluster.name: elasticsearch
node.name: es01
node.cluster: true //指定master伺服器
network.host: 192.168.1.205
iscovery.zen.ping.unicast.hosts:["192.168.1.205::9300", "192.168.1.206:9300","192.168.1.207:9300"] # ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
# Before you set out to tweak and tune the configuration, make sure you
# understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please see the documentation for further information on configuration options:
# <http://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration.html>
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: heimdall
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: es01
#
# Add custom attributes to the node:
#
node.rack: r1
#
# define cluster master
node.cluster: true
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /data/es/data
#
# Path to log files:
#
path.logs: /data/es/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
bootstrap.mlockall: true
#
# Make sure that the `ES_HEAP_SIZE` environment variable is set to about half the memory
# available on the system and that the owner of the process is allowed to use this limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 172.16.0.205
#
# Set a custom port for HTTP:
#
# http.port: 9200
#
# For more information, see the documentation at:
# <http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html>
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.zen.ping.unicast.hosts: ["172.16.0.205:9300", "172.16.0.206:9300","172.16.0.207:9300"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of nodes / 2 + 1):
#
discovery.zen.minimum_master_nodes: 2
#
# For more information, see the documentation at:
# <http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery.html>
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
# gateway.recover_after_nodes: 3
#
# For more information, see the documentation at:
# <http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-gateway.html>
#
# ---------------------------------- Various -----------------------------------
#
# Disable starting multiple nodes on a single system:
#
# node.max_local_storage_nodes: 1
#
# Require explicit names when deleting indices:
#
# action.destructive_requires_name: true
#
script.inline: true
script.indexed: true
script.engine.groovy.file.aggs: true
script.engine.groovy.file.mapping: true
script.engine.groovy.file.search: true
script.engine.groovy.file.update: true
script.engine.groovy.file.plugin: true
script.engine.groovy.indexed.aggs: true
script.engine.groovy.indexed.mapping: false
script.engine.groovy.indexed.search: true
script.engine.groovy.indexed.update: false
script.engine.groovy.indexed.plugin: false
script.engine.groovy.inline.aggs: true
script.engine.groovy.inline.mapping: false
script.engine.groovy.inline.search: true
script.engine.groovy.inline.update: false
script.engine.groovy.inline.plugin: false - 安裝 elasticsearch-head 外掛
bin/plugin install mobz/elasticsearch-head /bin/elasticsearch -d Tomcat
- 配置JDK
- 下載tomcat
使用的是apache-tomcat-7.0.69.tar.gz，解壓到/usr/local/tomcat/下
-配置tomcat
在catalina.sh 第一行新增
JAVA_OPTS="-XX:PermSize=512m -XX:MaxPermSize=512m"
啟動
/bin/startup.sh
停止
ps -ef|grep tomcat
kill -9 xxx
檢視日誌
tail -200f logs/catalina.outMemcached
yum -y install gcc
yum -y install gcc-c++ # tar xzvf libevent-2.0.22-stable.tar.gz ##解壓
# cd libevent-2.0.22-stable
# ./configure --prefix=/usr
# make
# make install 測試ls -al /usr/lib | grep libevent-
cd /tmp
# tar xzvf memcached-1.4.34.tar.gz
# cd memcached-1.4.34
# ./configure --with-libevent=/usr --enable-64bit --enable-threads
# make
# make install 測試ls –al /usr/local/bin/memcached
預設memcached單個程序只支援到2G記憶體，需要更大記憶體支援的話，需要開啟64位支援，編譯的時候加引數：
–enable-64bit
- 啟動memcached
直接啟動
#/usr/local/bin/memcached -d -m 1024 -u root -p 11211 -P /tmp/memcached.pid 或者
#建立啟動使用者
groupadd memcached
useradd -g memcached memcached
#建立PID程序目錄
cd /var/run
mkdir memcached
cp /tmp/memcached.pid /var/run/memcached/
chown –R memcached.memcached memcached
#啟動服務：
/usr/local/bin/memcached –d –m 1024 –u memcached –l 172.16.0.208 –p 11211 –c 1024 –P /var/run/memcached/memcached.pid
#關閉服務：
kill `cat /var/run/memcached/memcached.pid`- 安裝telnet
yum install telnet-server
yum install telnet - 啟用telnet
修改 /etc/xinetd.d/telnet：disable=no
重啟服務service xinetd restart - 測試memcached：
telnet 127.0.0.1 11211
stats
set var 32 0 4
java
get var
quit