CDH5.3.3安裝(hadoop叢集)
Cloudera Manager提供兩種軟體包安裝源,Package 和 Parcel:
Package就是一個個rpm檔案,以yum的方式組織起來。
Parcel是rpm包的壓縮格式,以.parcel結尾,所有的rpm壓縮在一個檔案中,方便下載和分發,使用manifest.json檔案對parcel檔案進行描述,將parcel檔案儲存到區域網內的Web伺服器上,安裝過程中就不需要從網際網路上下載檔案了,實現了離線安裝。
(1) 版本含義介紹:
https://ccp.cloudera.com/display/DOC/CDH+Version+and+Packaging+Information
(2)各版本特性檢視:
https://ccp.cloudera.com/display/DOC/CDH+Packaging+Information+for+Previous+Releases
(3)Cloudera Manager下載地址
http://archive.cloudera.com/cm5/cm/5/cloudera-manager-el6-cm5.3.3_x86_64.tar.gz
(4)parcel包,manifest.json檔案下載地址
http://archive.cloudera.com/cdh5/parcels/latest/
(5)mysql安裝設定
http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cm_ig_mysql.html
(6)MySqlJDBC驅動
http://dev.mysql.com/downloads/connector/j/
Mysql的版本與Mysql JDBC驅動版本對應情況
Connector/J 5.1 支援Mysql 4.1、Mysql 5.0、Mysql 5.1、Mysql 6.0 alpha這些版本。
Connector/J 5.0 支援MySQL 4.1、MySQL 5.0 servers、distributed transaction (XA)。
Connector/J 3.1 支援MySQL 4.1、MySQL 5.0 servers、MySQL 5.0 except distributed transaction (XA) support。
Connector/J 3.0 支援MySQL 3.x or MySQL 4.1。
(7)jdk
http://www.oracle.com/technetwork/java/javase/downloads/index.html
環境:centos6.5
===============================================================================================================================
安裝VNC
rpm -ivh tigervnc-1.1.0-5.el6_4.1.x86_64.rpm
rpm -ivh tigervnc-server-1.1.0-5.el6_4.1.x86_64.rpm
vi /etc/sysconfig/vncservers
VNCSERVERS="1:root"
VNCSERVERARGS[1]="-geometry 800x600 -nolisten tcp -localhost"
vncserver
vi /root/.vnc/xstartup
#twm &
gnome-session &
===============================================================================================================================
1、配置網路(所有節點)
vi /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.10.100 hadoop1
192.168.10.101 hadoop2
192.168.10.102 hadoop3
===============================================================================================================================
2、SSH無密碼驗證配置(所有節點)
<hadoop1 node>
ssh-keygen -t rsa -P ''
cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys
chmod 600 $HOME/.ssh/authorized_keys
ssh localhost
ssh hadoop1
scp $HOME/.ssh/authorized_keys [email protected]:$HOME/.ssh/
<hadoop2 node>
ssh-keygen -t rsa -P ''
cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys
scp $HOME/.ssh/authorized_keys [email protected]:$HOME/.ssh/
<hadoop3 node>
ssh-keygen -t rsa -P ''
cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys
scp $HOME/.ssh/authorized_keys [email protected]:$HOME/.ssh/
scp $HOME/.ssh/authorized_keys [email protected]:$HOME/.ssh/
===============================================================================================================================
3、解除安裝系統自帶OPEN-JDK,安裝oracle的java(所有節點)
rpm -qa|grep java
rpm -e --nodeps java-1.6.0-openjdk-1.6.0.0-1.66.1.13.0.el6.x86_64
rpm -e --nodeps java-1.7.0-openjdk-1.7.0.45-2.4.3.3.el6.x86_64
rpm -e --allmatches --nodeps tzdata-java-2013g-1.el6.noarch
rpm -ivh jdk-7u75-linux-x64.rpm
vi /etc/profile
export JAVA_HOME=/usr/java/jdk1.7.0_75
export JRE_HOME=/usr/java/jdk1.7.0_75/jre
export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH
source /etc/profile
echo "JAVA_HOME=/usr/java/latest" >> /etc/environment
=============================================================================================================================
4、關閉防火牆(所有節點)
(臨時關閉)
service iptables stop
(重啟後生效)
chkconfig iptables off
SELINUX:
(臨時生效)
setenforce 0
修改/etc/selinux/config下的SELINUX=disabled (重啟後永久生效)
===========================================================================================================================
5、所有節點配置NTP服務,所有datanode節點以master節點為基礎同步時間
安裝包:
ntpdate-4.2.6p5-1.el6.centos.x86_64
ntp-4.2.6p5-1.el6.centos.x86_64
--------------------------------------------------------------------
<master>
vi /etc/ntp.conf
restrict 0.0.0.0 mask 0.0.0.0 nomodify notrap
server 127.127.1.0
fudge 127.127.1.0 stratum 10
driftfile /var/lib/ntp/drift
broadcastdelay 0.008
keys /etc/ntp/keys
includefile /etc/ntp/crypto/pw
restrict 127.0.0.1
restrict -6 ::1
---------------------------------------------------------------------
<datanode>
restrict 0.0.0.0 mask 0.0.0.0 nomodify notrap
restrict default kod nomodify notrap nopeernoquery
restrict -6 default kod nomodify notrapnopeer noquery
server hadoop1 prefer
fudge 127.127.1.0 stratum 10
driftfile /var/lib/ntp/drift
broadcastdelay 0.008
keys /etc/ntp/keys
restrict 127.0.0.1
restrict -6 ::1
===========================================================================================================================
6、mysql資料庫配置安裝(master節點),使用嵌入式資料庫存在效能問題
mysql5.5以後是通過cmake來編譯
tar -zxvf cmake-3.2.1.tar.gz
cd cmake-3.2.1
./configure
make
make install
rpm -qa|grep ncurses-devel
rpm -ivh ncurses-devel-5.7-3.20090208.el6.x86_64.rpm
groupadd mysql
useradd mysql -g mysql -M -s /sbin/nologin
mkdir -p /usr/local/mysql
mkdir -p /usr/local/mysql/data
cd /soft
tar -zxvf mysql-5.6.24.tar.gz
chown -R root:root mysql-5.6.24
cd mysql-5.6.24
cmake . \
-DCMAKE_INSTALL_PREFIX=/usr/local/mysql \
-DMYSQL_DATADIR=/usr/local/mysql/data \
-DSYSCONFDIR=/etc \
-DWITH_MYISAM_STORAGE_ENGINE=1 \
-DWITH_INNOBASE_STORAGE_ENGINE=1 \
-DWITH_MEMORY_STORAGE_ENGINE=1 \
-DWITH_FEDERATED_STORAGE_ENGINE=1 \
-DWITH_READLINE=1 \
-DMYSQL_UNIX_ADDR=/tmp/mysqld.sock \
-DMYSQL_TCP_PORT=3306 \
-DENABLED_LOCAL_INFILE=1 \
-DWITH_PARTITION_STORAGE_ENGINE=1 \
-DEXTRA_CHARSETS=all \
-DDEFAULT_CHARSET=utf8 \
-DDEFAULT_COLLATION=utf8_general_ci
make
make install
cd /usr/local/mysql
chown -R mysql:mysql .
chmod -R ug+rwx /usr/local/mysql/data
scripts/mysql_install_db --datadir=/usr/local/mysql/data --user=mysql
vi /root/.bash_profile
PATH=$PATH:$HOME/bin:/usr/local/mysql/bin:/usr/local/mysql/lib
source /root/.bash_profile
vi /etc/my.cnf
#[mysqld] 下面新增:
[mysqld]
port=3306
socket=/tmp/mysqld.sock
user=mysql
basedir=/usr/local/mysql
datadir=/usr/local/mysql/data
default-storage-engine=MyISAM
explicit_defaults_for_timestamp=true
federated
character-set-server=utf8
sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES
server-id=1
#log_bin=/tmp/1.000001
#slow_query_log=1
#slow_query_log_file=/tmp/mysqlslow.log1
#general_log=1
#general_log_file=/tmp/mysqlgen.log
innodb_log_files_in_group=1
[client]
port=3306
socket=/tmp/mysqld.sock
#user=root
#password=hadoop
cp support-files/mysql.server /etc/init.d/mysql
chmod 755 /etc/init.d/mysql
vi /etc/init.d/mysql
basedir=/usr/local/mysql
datadir=/usr/local/mysql/data
chkconfig --add mysql
chkconfig --list mysql
chkconfig mysql on
----------------------------------------------------------------------------------------------------------
官方建議設定:
Here is an option file with Cloudera recommended settings:
[mysqld]
transaction-isolation = READ-COMMITTED
# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
# symbolic-links = 0
key_buffer = 16M
key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1
max_connections = 550
#log_bin should be on a disk with enough free space. Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your system and chown the specified folder to the mysql user.
#log_bin=/var/lib/mysql/mysql_binary_log
#expire_logs_days = 10
#max_binlog_size = 100M
# For MySQL version 5.1.8 or later. Comment out binlog_format for older versions.
binlog_format = mixed
read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M
# InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
-----------------------------------------------------------------------------------------------
Note: If you already have the JDBC driver installed on the hosts that need it, you can skip this section. However, MySQL 5.6 requires a driver version 5.1.26 or higher.
===========================================================================================================================
7、建立資料庫(master節點)
--hive資料庫
create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
--叢集監控資料庫
create database amon DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
--hue資料庫
create database hue DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
給使用者授權(這裡密碼設為hadoop)
mysql>GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY 'hadoop'; //為root新增遠端連線的能力。
mysql>GRANT ALL PRIVILEGES ON *.* TO 'root'@'hadoop1' IDENTIFIED BY 'hadoop';
mysql>GRANT ALL PRIVILEGES ON *.* TO 'root'@'localhost' IDENTIFIED BY 'hadoop';
mysql>use mysql
mysql>update user set Password = password('hadoop') where User='root';
mysql>select Host,User,Password from user where User='root';
mysql>flush privileges;
===========================================================================================================================
8、安裝Cloudera Manager 5(cm5)和Agent,master節點安裝
根據自己的系統選擇相應的版本,本次安裝選用的是cloudera-manager-el6-cm5.3.3_x86_64.tar.gz。
下載完成後只上傳到master節點即可。然後解壓到/opt目錄下,不能解壓到其他地方,因為cdh5的源會預設在/opt/cloudera/parcel-repo尋找。給所有節點新增cloudera-scm使用者:
tar -zxvf /soft/cloudera-manager-el6-cm5.3.3_x86_64.tar.gz
解壓後將cm-5.3.3和cloudera資料夾轉移到/opt目錄
mv cm-5.3.3/ /opt
mv cloudera /opt/
chown -R root:root /opt/cm-5.3.3
chown -R root:root /opt/cloudera
為Cloudera Manager 5建立資料庫:
首先將mysql-connector-java-5.1.33.jar放到/opt/cm-5.3.3/share/cmf/lib/目錄下
cp /soft/mysql-connector-java-5.1.33.jar /opt/cm-5.3.3/share/cmf/lib/
給所有節點新增cloudera-scm使用者:
useradd --system --home=/opt/cm-5.3.3/run/cloudera-scm-server --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm
Agent配置:
修改/opt/cm-5.3.3/etc/cloudera-scm-agent/config.ini的server_host為主節點的主機名
server_host=hadoop1
準備Parcels,用以安裝CDH5(配置原始檔)
將CHD5相關的Parcel包放到主節點的/opt/cloudera/parcel-repo/目錄中(若無parcel-repo目錄需要手動建立)
cd /soft
mv manifest.json /opt/cloudera/parcel-repo/
mv CDH-5.3.3-1.cdh5.3.3.p0.5-el6.parcel.sha1 /opt/cloudera/parcel-repo/
mv CDH-5.3.3-1.cdh5.3.3.p0.5-el6.parcel /opt/cloudera/parcel-repo/
開啟manifest.json檔案,裡面是json格式的配置,我們需要的就是與我們系統版本相對應的hash碼,因為我們用的是Centos6.5,所以找到如下位置
"parcelName": "CDH-5.3.3-1.cdh5.3.3.p0.5-el6.parcel"
在這個大括號的最下面找到“hash”所對應的值
"hash": "d25348a155e18e554de70068a319bbd7930c070a"
將“hash”的值複製下來,然後建立一個檔案,檔名與你的parel包名一致,並加上.sha字尾。
將下載的CDH-5.3.3-1.cdh5.3.3.p0.5-el6.parcel.sha1,重新命名為CDH-5.3.3-1.cdh5.3.3.p0.5-el6.parcel.sha即可
cd /opt/cloudera/parcel-repo/
mv CDH-5.3.3-1.cdh5.3.3.p0.5-el6.parcel.sha1 CDH-5.3.3-1.cdh5.3.3.p0.5-el6.parcel.sha
同步Agent到其他節點
scp -r /opt/cm-5.3.3 [email protected]:/opt/
scp -r /opt/cm-5.3.3 [email protected]:/opt/
在主節點初始化CM5的資料庫:
/opt/cm-5.3.3/share/cmf/schema/scm_prepare_database.sh mysql cm -hlocalhost -uroot -phadoop --scm-host localhost scm scm scm
格式是:scm_prepare_database.sh 資料庫型別 資料庫 伺服器 使用者名稱 密碼 –scm-host Cloudera_Manager_Server所在的機器
相關啟動指令碼
通過/opt/cm-5.3.3/etc/init.d/cloudera-scm-server start啟動服務端。
通過/opt/cm-5.3.3/etc/init.d/cloudera-scm-agent start啟動Agent服務。
我們啟動的其實是個service指令碼,需要停止服務將以上的start引數改為stop就可以了,重啟是restart。
1. 在namenode啟動server和agent( 主節點也是要做到叢集中的, 所以主節點也要開啟Agent )
/opt/cm-5.3.3/etc/init.d/cloudera-scm-server start
/opt/cm-5.3.3/etc/init.d/cloudera-scm-agent start
2. 在所有datanode啟動agent
/opt/cm-5.3.3/etc/init.d/cloudera-scm-agent start
Cloudera Manager Server和Agent都啟動以後,就可以進行CDH5的安裝配置了。
這時可以通過瀏覽器訪問主節點的7180埠測試一下了(由於CM Server的啟動需要花點時間,這裡可能要等待一會才能訪問),預設的使用者名稱和密碼均為admin
http://192.168.10.100:7180
在區域網內訪問: http://192.168.10.100:7180/cmf/login
預設帳號和密碼:admin
如下三個版本:
Clouder Express:
免費版,長期使用
Clouder Enterprise試用版:
試用60天
Clouder Enterprise:
需要上傳授權許可
如上我們選擇的是Clouder Express版
選擇需要部署的機器
這裡需要說明的是指定主機安裝有多種方式:
1.直接列出ip或則host,多臺以逗號、分號、製表符、空格或放置在單獨的行。
2.指定ip的地址範圍例如:10.1.1.[1-4] 或則 host[1-3].company.com.3.記得指定的主機需要關閉防火牆,(如果遇到不能安裝,最好使用安裝cloudera manager虛擬機器進行復制)
選擇安裝包
部署完成
接下來是伺服器檢查,可能會遇到以下問題:
Cloudera 建議將 /proc/sys/vm/swappiness 設定為 0。當前設定為 60。使用 sysctl 命令在執行時更改該設定並編輯 /etc/sysctl.conf 以在重啟後儲存該設定。您可以繼續進行安裝,但可能會遇到問題,Cloudera Manager 報告您的主機由於交換執行狀況不佳。以下主機受到影響:
通過echo 0 > /proc/sys/vm/swappiness即可解決。
echo "vm.swappiness = 0" >> /etc/sysctl.conf
選擇安裝服務
服務配置,一般情況下保持預設就可以了(Cloudera Manager會根據機器的配置自動進行配置,如果需要特殊調整,自行進行設定就可以了)
hive元資料庫配置
配置引數
叢集安裝完成
附:中間安裝報錯解決
執行錯誤:
[[email protected] ~]# /opt/cm-5.3.3/share/cmf/schema/scm_prepare_database.sh mysql cm -hlocalhost -uroot -phadoop --scm-host localhost scm scm scm
JAVA_HOME=/usr/local/jdk1.7.0_67
Verifying that we can write to /opt/cm-5.3.3/etc/cloudera-scm-server
[ main] DbProvisioner ERROR Exception when creating/dropping database with user 'root' and jdbc url 'jdbc:mysql://localhost/?useUnicode=true&characterEncoding=UTF-8'
java.sql.SQLException: Access denied for user 'root'@'localhost' (using password: YES)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:996)[mysql-connector-java-5.1.33.jar:5.1.33]
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3887)[mysql-connector-java-5.1.33.jar:5.1.33]
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3823)[mysql-connector-java-5.1.33.jar:5.1.33]
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:870)[mysql-connector-java-5.1.33.jar:5.1.33]
at com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1659)[mysql-connector-java-5.1.33.jar:5.1.33]
at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1206)[mysql-connector-java-5.1.33.jar:5.1.33]
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2234)[mysql-connector-java-5.1.33.jar:5.1.33]
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2265)[mysql-connector-java-5.1.33.jar:5.1.33]
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2064)[mysql-connector-java-5.1.33.jar:5.1.33]
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:790)[mysql-connector-java-5.1.33.jar:5.1.33]
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:44)[mysql-connector-java-5.1.33.jar:5.1.33]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)[:1.7.0_67]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)[:1.7.0_67]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)[:1.7.0_67]
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)[:1.7.0_67]
at com.mysql.jdbc.Util.handleNewInstance(Util.java:377)[mysql-connector-java-5.1.33.jar:5.1.33]
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:395)[mysql-connector-java-5.1.33.jar:5.1.33]
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:325)[mysql-connector-java-5.1.33.jar:5.1.33]
at java.sql.DriverManager.getConnection(DriverManager.java:571)[:1.7.0_67]
at java.sql.DriverManager.getConnection(DriverManager.java:215)[:1.7.0_67]
at com.cloudera.enterprise.dbutil.DbProvisioner.executeSql(DbProvisioner.java:283)[db-common-5.3.3.jar:]
at com.cloudera.enterprise.dbutil.DbProvisioner.doMain(DbProvisioner.java:95)[db-common-5.3.3.jar:]
at com.cloudera.enterprise.dbutil.DbProvisioner.main(DbProvisioner.java:110)[db-common-5.3.3.jar:]
[ main] DbProvisioner ERROR Stack Trace:
java.sql.SQLException: Access denied for user 'root'@'localhost' (using password: YES)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:996)[mysql-connector-java-5.1.33.jar:5.1.33]
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3887)[mysql-connector-java-5.1.33.jar:5.1.33]
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3823)[mysql-connector-java-5.1.33.jar:5.1.33]
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:870)[mysql-connector-java-5.1.33.jar:5.1.33]
at com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1659)[mysql-connector-java-5.1.33.jar:5.1.33]
at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1206)[mysql-connector-java-5.1.33.jar:5.1.33]
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2234)[mysql-connector-java-5.1.33.jar:5.1.33]
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2265)[mysql-connector-java-5.1.33.jar:5.1.33]
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2064)[mysql-connector-java-5.1.33.jar:5.1.33]
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:790)[mysql-connector-java-5.1.33.jar:5.1.33]
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:44)[mysql-connector-java-5.1.33.jar:5.1.33]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)[:1.7.0_67]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)[:1.7.0_67]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)[:1.7.0_67]
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)[:1.7.0_67]
at com.mysql.jdbc.Util.handleNewInstance(Util.java:377)[mysql-connector-java-5.1.33.jar:5.1.33]
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:395)[mysql-connector-java-5.1.33.jar:5.1.33]
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:325)[mysql-connector-java-5.1.33.jar:5.1.33]
at java.sql.DriverManager.getConnection(DriverManager.java:571)[:1.7.0_67]
at java.sql.DriverManager.getConnection(DriverManager.java:215)[:1.7.0_67]
at com.cloudera.enterprise.dbutil.DbProvisioner.executeSql(DbProvisioner.java:283)[db-common-5.3.3.jar:]
at com.cloudera.enterprise.dbutil.DbProvisioner.doMain(DbProvisioner.java:95)[db-common-5.3.3.jar:]
at com.cloudera.enterprise.dbutil.DbProvisioner.main(DbProvisioner.java:110)[db-common-5.3.3.jar:]
--> Error 1, giving up (use --force if you wish to ignore the error)
需要修改mysql登入密碼:
解決:
修改MySQL的root使用者的密碼以及開啟遠端連線
mysql>use mysql;
mysql>desc user;
mysql>GRANT ALL PRIVILEGES ON *.* TO [email protected]"%" IDENTIFIED BY "hadoop"; //為root新增遠端連線的能力。
mysql>update user set Password = password('hadoop') where User='root';
mysql>select Host,User,Password from user where User='root';
mysql>flush privileges;
mysql>exit
[[email protected] ~]# /opt/cm-5.3.3/share/cmf/schema/scm_prepare_database.sh mysql cm -hlocalhost -uroot -phadoop --scm-host localhost scm scm scm
JAVA_HOME=/usr/local/jdk1.7.0_67
Verifying that we can write to /opt/cm-5.3.3/etc/cloudera-scm-server
Creating SCM configuration file in /opt/cm-5.3.3/etc/cloudera-scm-server
Executing: /usr/local/jdk1.7.0_67/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/opt/cm-5.3.3/share/cmf/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /opt/cm-5.3.3/etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
[ main] DbCommandExecutor INFO Successfully connected to database.
All done, your SCM database is configured correctly!
官方關於JDK版本的選擇建議:
Java Development Kit Installation
Some installation paths require that you install the Oracle Java Development Kit on hosts before deploying Cloudera Manager, CDH, and managed services. To install the Oracle JDK, follow the instructions in Installing the Oracle JDK. The completed installation, or any already existing installation, must meet the following requirements.
Requirements
•Install a supported version:◦CDH 5 - Supported JDK Versions
◦CDH 4 - Supported JDK Versions
•Install the same version of the Oracle JDK on each host.
•Install the JDK in /usr/java/jdk-version.
Important: •You cannot upgrade from JDK 1.7 to JDK 1.8 while upgrading to CDH 5.3. The cluster must already be running CDH 5.3 when you upgrade to JDK 1.8.
•On SLES 11 platforms, do not install or try to use the IBM Java version bundled with the SLES distribution. CDH does not run correctly with that version.
Installing the Oracle JDK
The Oracle JDK installer is available both as an RPM-based installer for RPM-based systems, and as a binary installer for other systems.
1.Download the .tar.gz file for one of the supported versions of the Oracle JDK from Java SE Downloads, Java SE 7 Downloads, or Java SE 6 Downloads. (These links are correct at the time of writing but change frequently.)
2.Extract the JDK to /usr/java/jdk-version; for example /usr/java/jdk.1.7.0_nn or /usr/java/jdk.1.8.0_nn, where nn is a supported version.
3.Set JAVA_HOME to the directory where the JDK is installed, for exampleexport JAVA_HOME=/usr/java/jdk.1.7.0_nn in the following files on Cloudera Manager Server and cluster hosts:◦Cloudera Manager Server - /etc/default/cloudera-scm-server
◦Cloudera Manager Agent - /etc/default/cloudera-scm-agent or follow the instructions in Configuring a Custom Java Home Location
◦Cloudera Manager package-based or unmanaged deployment - /etc/default/bigtop-utils
Supported JDK Versions
CDH 5 requires the Oracle JDK, and supports the versions shown in the table that follows.
Latest Certified Version
1.7.0_75
1.8.0_40