Hadoop 2.4.1 搭建Ha遇到問題記錄
阿新 • • 發佈：2019-02-12
15/06/19 17:12:44 ERROR namenode.FSNamesystem: FSNamesystem initialization failed. java.io.IOException: Invalid configuration: <span style="background-color: rgb(255, 102, 102);">a shared edits dir must not be specified if HA is not enabled</span>. // 明明我配置了HA，卻說沒有,下面的配置寫錯了 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:710) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:654) at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:892) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1310) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1424) 15/06/19 17:12:44 INFO namenode.FSNamesystem: Stopping services started for active state 15/06/19 17:12:44 FATAL namenode.NameNode: Exception in namenode join java.lang.NullPointerException at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.stopActiveServices(FSNamesystem.java:1118) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.close(FSNamesystem.java:1446) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:781) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:654) at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:892) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1310) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1424) 15/06/19 17:12:44 INFO util.ExitUtil: Exiting with status 1 15/06/19 17:12:44 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at v1/192.168.32.91 ************************************************************/
<strong>.寫成了-</strong> <property> <name>dfs.ha.automatic-failover-enabled</name> <value>true</value> </property> 正確如下： <property> <name>dfs.ha.automatic-failover.enabled</name> <value>true</value> </property>
2.journalnode的埠配置錯誤，導致不能格式化namenode,journalnode監聽的埠是：8485，而不是8445. 3.格式化standby namnenode時，有兩種方式 1.將active namenode格式化完成後，將整個hadoop.tmp.dir路徑全部拷貝到standby機器上。 2.在active namenode格式化完成後，並且格式化zkfc(只需要在active上格式化一次，standby上的zkfc不需要再次格式化)啟動好了namenode之後。 在standby namenode上執行:hdfs namenode -bootstrapStandby
[[email protected] hadoop]# hdfs namenode -bootstrapStandby
15/06/19 19:28:45 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = v2/192.168.32.92
STARTUP_MSG: args = [-bootstrapStandby]
STARTUP_MSG: version = 2.4.1
STARTUP_MSG: classpath = /root/hadoop/etc/hadoop:/root/hadoop/share/hadoop/common/lib/asm-3.2.jar:/root/hadoop/share/hadoop/common/lib/xz-1.0.jar:/root/hadoop/share/hadoop/common/lib/junit-4.8.2.jar:/root/hadoop/share/hadoop/common/lib/httpcore-4.2.5.jar:/root/hadoop/share/hadoop/common/lib/jetty-6.1.26.jar:/root/hadoop/share/hadoop/common/lib/commons-net-3.1.jar:/root/hadoop/share/hadoop/common/lib/jetty-util-6.1.26.jar:/root/hadoop/share/hadoop/common/lib/commons-compress-1.4.1.jar:/root/hadoop/share/hadoop/common/lib/log4j-1.2.17.jar:/root/hadoop/share/hadoop/common/lib/commons-io-2.4.jar:/root/hadoop/share/hadoop/common/lib/hadoop-auth-2.4.1.jar:/root/hadoop/share/hadoop/common/lib/stax-api-1.0-2.jar:/root/hadoop/share/hadoop/common/lib/jackson-xc-1.8.8.jar:/root/hadoop/share/hadoop/common/lib/netty-3.6.2.Final.jar:/root/hadoop/share/hadoop/common/lib/httpclient-4.2.5.jar:/root/hadoop/share/hadoop/common/lib/jsr305-1.3.9.jar:/root/hadoop/share/hadoop/common/lib/avro-1.7.4.jar:/root/hadoop/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/root/hadoop/share/hadoop/common/lib/jackson-mapper-asl-1.8.8.jar:/root/hadoop/share/hadoop/common/lib/commons-codec-1.4.jar:/root/hadoop/share/hadoop/common/lib/commons-el-1.0.jar:/root/hadoop/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/root/hadoop/share/hadoop/common/lib/guava-11.0.2.jar:/root/hadoop/share/hadoop/common/lib/xmlenc-0.52.jar:/root/hadoop/share/hadoop/common/lib/paranamer-2.3.jar:/root/hadoop/share/hadoop/common/lib/commons-math3-3.1.1.jar:/root/hadoop/share/hadoop/common/lib/mockito-all-1.8.5.jar:/root/hadoop/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/root/hadoop/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/root/hadoop/share/hadoop/common/lib/jsch-0.1.42.jar:/root/hadoop/share/hadoop/common/lib/hadoop-annotations-2.4.1.jar:/root/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/root/hadoop/share/hadoop/common/lib/jackson-core-asl-1.8.8.jar:/root/hadoop/share/hadoop/common/lib/jersey-core-1.9.jar:/root/hadoop/share/hadoop/common/lib/commons-logging-1.1.3.jar:/root/hadoop/share/hadoop/common/lib/jets3t-0.9.0.jar:/root/hadoop/share/hadoop/common/lib/jsp-api-2.1.jar:/root/hadoop/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/root/hadoop/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/root/hadoop/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/root/hadoop/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/root/hadoop/share/hadoop/common/lib/jersey-json-1.9.jar:/root/hadoop/share/hadoop/common/lib/commons-httpclient-3.1.jar:/root/hadoop/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/root/hadoop/share/hadoop/common/lib/commons-collections-3.2.1.jar:/root/hadoop/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/root/hadoop/share/hadoop/common/lib/activation-1.1.jar:/root/hadoop/share/hadoop/common/lib/jersey-server-1.9.jar:/root/hadoop/share/hadoop/common/lib/commons-cli-1.2.jar:/root/hadoop/share/hadoop/common/lib/commons-configuration-1.6.jar:/root/hadoop/share/hadoop/common/lib/commons-digester-1.8.jar:/root/hadoop/share/hadoop/common/lib/servlet-api-2.5.jar:/root/hadoop/share/hadoop/common/lib/commons-lang-2.6.jar:/root/hadoop/share/hadoop/common/lib/jettison-1.1.jar:/root/hadoop/share/hadoop/common/lib/zookeeper-3.4.5.jar:/root/hadoop/share/hadoop/common/lib/jackson-jaxrs-1.8.8.jar:/root/hadoop/share/hadoop/common/hadoop-nfs-2.4.1.jar:/root/hadoop/share/hadoop/common/hadoop-common-2.4.1-tests.jar:/root/hadoop/share/hadoop/common/hadoop-common-2.4.1.jar:/root/hadoop/share/hadoop/hdfs:/root/hadoop/share/hadoop/hdfs/lib/asm-3.2.jar:/root/hadoop/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/root/hadoop/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/root/hadoop/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/root/hadoop/share/hadoop/hdfs/lib/commons-io-2.4.jar:/root/hadoop/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/root/hadoop/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/root/hadoop/share/hadoop/hdfs/lib/jackson-mapper-asl-1.8.8.jar:/root/hadoop/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/root/hadoop/share/hadoop/hdfs/lib/commons-el-1.0.jar:/root/hadoop/share/hadoop/hdfs/lib/guava-11.0.2.jar:/root/hadoop/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/root/hadoop/share/hadoop/hdfs/lib/jackson-core-asl-1.8.8.jar:/root/hadoop/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/root/hadoop/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/root/hadoop/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/root/hadoop/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/root/hadoop/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/root/hadoop/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/root/hadoop/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/root/hadoop/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/root/hadoop/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/root/hadoop/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/root/hadoop/share/hadoop/hdfs/hadoop-hdfs-nfs-2.4.1.jar:/root/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.4.1.jar:/root/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.4.1-tests.jar:/root/hadoop/share/hadoop/yarn/lib/asm-3.2.jar:/root/hadoop/share/hadoop/yarn/lib/xz-1.0.jar:/root/hadoop/share/hadoop/yarn/lib/jetty-6.1.26.jar:/root/hadoop/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/root/hadoop/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/root/hadoop/share/hadoop/yarn/lib/log4j-1.2.17.jar:/root/hadoop/share/hadoop/yarn/lib/commons-io-2.4.jar:/root/hadoop/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/root/hadoop/share/hadoop/yarn/lib/jackson-xc-1.8.8.jar:/root/hadoop/share/hadoop/yarn/lib/jsr305-1.3.9.jar:/root/hadoop/share/hadoop/yarn/lib/jackson-mapper-asl-1.8.8.jar:/root/hadoop/share/hadoop/yarn/lib/commons-codec-1.4.jar:/root/hadoop/share/hadoop/yarn/lib/aopalliance-1.0.jar:/root/hadoop/share/hadoop/yarn/lib/jline-0.9.94.jar:/root/hadoop/share/hadoop/yarn/lib/guava-11.0.2.jar:/root/hadoop/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/root/hadoop/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/root/hadoop/share/hadoop/yarn/lib/javax.inject-1.jar:/root/hadoop/share/hadoop/yarn/lib/jackson-core-asl-1.8.8.jar:/root/hadoop/share/hadoop/yarn/lib/jersey-core-1.9.jar:/root/hadoop/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/root/hadoop/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/root/hadoop/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/root/hadoop/share/hadoop/yarn/lib/jersey-json-1.9.jar:/root/hadoop/share/hadoop/yarn/lib/commons-httpclient-3.1.jar:/root/hadoop/share/hadoop/yarn/lib/commons-collections-3.2.1.jar:/root/hadoop/share/hadoop/yarn/lib/activation-1.1.jar:/root/hadoop/share/hadoop/yarn/lib/jersey-server-1.9.jar:/root/hadoop/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/root/hadoop/share/hadoop/yarn/lib/commons-cli-1.2.jar:/root/hadoop/share/hadoop/yarn/lib/servlet-api-2.5.jar:/root/hadoop/share/hadoop/yarn/lib/commons-lang-2.6.jar:/root/hadoop/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/root/hadoop/share/hadoop/yarn/lib/jersey-client-1.9.jar:/root/hadoop/share/hadoop/yarn/lib/jettison-1.1.jar:/root/hadoop/share/hadoop/yarn/lib/zookeeper-3.4.5.jar:/root/hadoop/share/hadoop/yarn/lib/jackson-jaxrs-1.8.8.jar:/root/hadoop/share/hadoop/yarn/lib/guice-3.0.jar:/root/hadoop/share/hadoop/yarn/hadoop-yarn-server-common-2.4.1.jar:/root/hadoop/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.4.1.jar:/root/hadoop/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.4.1.jar:/root/hadoop/share/hadoop/yarn/hadoop-yarn-server-tests-2.4.1.jar:/root/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.4.1.jar:/root/hadoop/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.4.1.jar:/root/hadoop/share/hadoop/yarn/hadoop-yarn-api-2.4.1.jar:/root/hadoop/share/hadoop/yarn/hadoop-yarn-client-2.4.1.jar:/root/hadoop/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.4.1.jar:/root/hadoop/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.4.1.jar:/root/hadoop/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.4.1.jar:/root/hadoop/share/hadoop/mapreduce/lib/asm-3.2.jar:/root/hadoop/share/hadoop/mapreduce/lib/xz-1.0.jar:/root/hadoop/share/hadoop/mapreduce/lib/hamcrest-core-1.1.jar:/root/hadoop/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/root/hadoop/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/root/hadoop/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/root/hadoop/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/root/hadoop/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/root/hadoop/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.8.8.jar:/root/hadoop/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/root/hadoop/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/root/hadoop/share/hadoop/mapreduce/lib/javax.inject-1.jar:/root/hadoop/share/hadoop/mapreduce/lib/hadoop-annotations-2.4.1.jar:/root/hadoop/share/hadoop/mapreduce/lib/jackson-core-asl-1.8.8.jar:/root/hadoop/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/root/hadoop/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/root/hadoop/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/root/hadoop/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/root/hadoop/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/root/hadoop/share/hadoop/mapreduce/lib/junit-4.10.jar:/root/hadoop/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/root/hadoop/share/hadoop/mapreduce/lib/guice-3.0.jar:/root/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.4.1.jar:/root/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.4.1.jar:/root/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.4.1.jar:/root/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.4.1.jar:/root/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.4.1.jar:/root/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.4.1.jar:/root/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.4.1-tests.jar:/root/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.4.1.jar:/root/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.4.1.jar:/root/hadoop/contrib/capacity-scheduler/*.jar
STARTUP_MSG: build = http://svn.apache.org/repos/asf/hadoop/common -r 1604318; compiled by 'jenkins' on 2014-06-21T05:43Z
STARTUP_MSG: java = 1.7.0_15
************************************************************/
15/06/19 19:28:45 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
15/06/19 19:28:45 INFO namenode.NameNode: createNameNode [-bootstrapStandby]
15/06/19 19:28:47 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/06/19 19:28:49 INFO ipc.Client: Retrying connect to server: v1/192.168.32.91:9000. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
15/06/19 19:28:50 INFO ipc.Client: Retrying connect to server: v1/192.168.32.91:9000. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
15/06/19 19:28:51 INFO ipc.Client: Retrying connect to server: v1/192.168.32.91:9000. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
15/06/19 19:28:52 INFO ipc.Client: Retrying connect to server: v1/192.168.32.91:9000. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
15/06/19 19:28:53 INFO ipc.Client: Retrying connect to server: v1/192.168.32.91:9000. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
15/06/19 19:28:54 INFO ipc.Client: Retrying connect to server: v1/192.168.32.91:9000. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
15/06/19 19:28:55 INFO ipc.Client: Retrying connect to server: v1/192.168.32.91:9000. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
15/06/19 19:28:56 INFO ipc.Client: Retrying connect to server: v1/192.168.32.91:9000. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
15/06/19 19:28:57 INFO ipc.Client: Retrying connect to server: v1/192.168.32.91:9000. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
15/06/19 19:28:58 INFO ipc.Client: Retrying connect to server: v1/192.168.32.91:9000. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
15/06/19 19:28:58 FATAL ha.BootstrapStandby: Unable to fetch namespace information from active NN at v1/192.168.32.91:9000: Call From v2/192.168.32.92 to v1:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
15/06/19 19:28:58 INFO util.ExitUtil: Exiting with status 2
15/06/19 19:28:58 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at v2/192.168.32.92
************************************************************/[[email protected] hadoop]# hdfs namenode -bootstrapStandby
15/06/19 19:31:39 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = v2/192.168.32.92
STARTUP_MSG: args = [-bootstrapStandby]
STARTUP_MSG: version = 2.4.1
STARTUP_MSG: classpath = /root/hadoop/etc/hadoop:/root/hadoop/share/hadoop/common/lib/asm-3.2.jar:/root/hadoop/share/hadoop/common/lib/xz-1.0.jar:/root/hadoop/share/hadoop/common/lib/junit-4.8.2.jar:/root/hadoop/share/hadoop/common/lib/httpcore-4.2.5.jar:/root/hadoop/share/hadoop/common/lib/jetty-6.1.26.jar:/root/hadoop/share/hadoop/common/lib/commons-net-3.1.jar:/root/hadoop/share/hadoop/common/lib/jetty-util-6.1.26.jar:/root/hadoop/share/hadoop/common/lib/commons-compress-1.4.1.jar:/root/hadoop/share/hadoop/common/lib/log4j-1.2.17.jar:/root/hadoop/share/hadoop/common/lib/commons-io-2.4.jar:/root/hadoop/share/hadoop/common/lib/hadoop-auth-2.4.1.jar:/root/hadoop/share/hadoop/common/lib/stax-api-1.0-2.jar:/root/hadoop/share/hadoop/common/lib/jackson-xc-1.8.8.jar:/root/hadoop/share/hadoop/common/lib/netty-3.6.2.Final.jar:/root/hadoop/share/hadoop/common/lib/httpclient-4.2.5.jar:/root/hadoop/share/hadoop/common/lib/jsr305-1.3.9.jar:/root/hadoop/share/hadoop/common/lib/avro-1.7.4.jar:/root/hadoop/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/root/hadoop/share/hadoop/common/lib/jackson-mapper-asl-1.8.8.jar:/root/hadoop/share/hadoop/common/lib/commons-codec-1.4.jar:/root/hadoop/share/hadoop/common/lib/commons-el-1.0.jar:/root/hadoop/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/root/hadoop/share/hadoop/common/lib/guava-11.0.2.jar:/root/hadoop/share/hadoop/common/lib/xmlenc-0.52.jar:/root/hadoop/share/hadoop/common/lib/paranamer-2.3.jar:/root/hadoop/share/hadoop/common/lib/commons-math3-3.1.1.jar:/root/hadoop/share/hadoop/common/lib/mockito-all-1.8.5.jar:/root/hadoop/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/root/hadoop/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/root/hadoop/share/hadoop/common/lib/jsch-0.1.42.jar:/root/hadoop/share/hadoop/common/lib/hadoop-annotations-2.4.1.jar:/root/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/root/hadoop/share/hadoop/common/lib/jackson-core-asl-1.8.8.jar:/root/hadoop/share/hadoop/common/lib/jersey-core-1.9.jar:/root/hadoop/share/hadoop/common/lib/commons-logging-1.1.3.jar:/root/hadoop/share/hadoop/common/lib/jets3t-0.9.0.jar:/root/hadoop/share/hadoop/common/lib/jsp-api-2.1.jar:/root/hadoop/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/root/hadoop/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/root/hadoop/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/root/hadoop/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/root/hadoop/share/hadoop/common/lib/jersey-json-1.9.jar:/root/hadoop/share/hadoop/common/lib/commons-httpclient-3.1.jar:/root/hadoop/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/root/hadoop/share/hadoop/common/lib/commons-collections-3.2.1.jar:/root/hadoop/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/root/hadoop/share/hadoop/common/lib/activation-1.1.jar:/root/hadoop/share/hadoop/common/lib/jersey-server-1.9.jar:/root/hadoop/share/hadoop/common/lib/commons-cli-1.2.jar:/root/hadoop/share/hadoop/common/lib/commons-configuration-1.6.jar:/root/hadoop/share/hadoop/common/lib/commons-digester-1.8.jar:/root/hadoop/share/hadoop/common/lib/servlet-api-2.5.jar:/root/hadoop/share/hadoop/common/lib/commons-lang-2.6.jar:/root/hadoop/share/hadoop/common/lib/jettison-1.1.jar:/root/hadoop/share/hadoop/common/lib/zookeeper-3.4.5.jar:/root/hadoop/share/hadoop/common/lib/jackson-jaxrs-1.8.8.jar:/root/hadoop/share/hadoop/common/hadoop-nfs-2.4.1.jar:/root/hadoop/share/hadoop/common/hadoop-common-2.4.1-tests.jar:/root/hadoop/share/hadoop/common/hadoop-common-2.4.1.jar:/root/hadoop/share/hadoop/hdfs:/root/hadoop/share/hadoop/hdfs/lib/asm-3.2.jar:/root/hadoop/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/root/hadoop/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/root/hadoop/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/root/hadoop/share/hadoop/hdfs/lib/commons-io-2.4.jar:/root/hadoop/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/root/hadoop/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/root/hadoop/share/hadoop/hdfs/lib/jackson-mapper-asl-1.8.8.jar:/root/hadoop/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/root/hadoop/share/hadoop/hdfs/lib/commons-el-1.0.jar:/root/hadoop/share/hadoop/hdfs/lib/guava-11.0.2.jar:/root/hadoop/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/root/hadoop/share/hadoop/hdfs/lib/jackson-core-asl-1.8.8.jar:/root/hadoop/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/root/hadoop/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/root/hadoop/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/root/hadoop/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/root/hadoop/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/root/hadoop/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/root/hadoop/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/root/hadoop/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/root/hadoop/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/root/hadoop/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/root/hadoop/share/hadoop/hdfs/hadoop-hdfs-nfs-2.4.1.jar:/root/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.4.1.jar:/root/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.4.1-tests.jar:/root/hadoop/share/hadoop/yarn/lib/asm-3.2.jar:/root/hadoop/share/hadoop/yarn/lib/xz-1.0.jar:/root/hadoop/share/hadoop/yarn/lib/jetty-6.1.26.jar:/root/hadoop/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/root/hadoop/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/root/hadoop/share/hadoop/yarn/lib/log4j-1.2.17.jar:/root/hadoop/share/hadoop/yarn/lib/commons-io-2.4.jar:/root/hadoop/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/root/hadoop/share/hadoop/yarn/lib/jackson-xc-1.8.8.jar:/root/hadoop/share/hadoop/yarn/lib/jsr305-1.3.9.jar:/root/hadoop/share/hadoop/yarn/lib/jackson-mapper-asl-1.8.8.jar:/root/hadoop/share/hadoop/yarn/lib/commons-codec-1.4.jar:/root/hadoop/share/hadoop/yarn/lib/aopalliance-1.0.jar:/root/hadoop/share/hadoop/yarn/lib/jline-0.9.94.jar:/root/hadoop/share/hadoop/yarn/lib/guava-11.0.2.jar:/root/hadoop/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/root/hadoop/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/root/hadoop/share/hadoop/yarn/lib/javax.inject-1.jar:/root/hadoop/share/hadoop/yarn/lib/jackson-core-asl-1.8.8.jar:/root/hadoop/share/hadoop/yarn/lib/jersey-core-1.9.jar:/root/hadoop/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/root/hadoop/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/root/hadoop/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/root/hadoop/share/hadoop/yarn/lib/jersey-json-1.9.jar:/root/hadoop/share/hadoop/yarn/lib/commons-httpclient-3.1.jar:/root/hadoop/share/hadoop/yarn/lib/commons-collections-3.2.1.jar:/root/hadoop/share/hadoop/yarn/lib/activation-1.1.jar:/root/hadoop/share/hadoop/yarn/lib/jersey-server-1.9.jar:/root/hadoop/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/root/hadoop/share/hadoop/yarn/lib/commons-cli-1.2.jar:/root/hadoop/share/hadoop/yarn/lib/servlet-api-2.5.jar:/root/hadoop/share/hadoop/yarn/lib/commons-lang-2.6.jar:/root/hadoop/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/root/hadoop/share/hadoop/yarn/lib/jersey-client-1.9.jar:/root/hadoop/share/hadoop/yarn/lib/jettison-1.1.jar:/root/hadoop/share/hadoop/yarn/lib/zookeeper-3.4.5.jar:/root/hadoop/share/hadoop/yarn/lib/jackson-jaxrs-1.8.8.jar:/root/hadoop/share/hadoop/yarn/lib/guice-3.0.jar:/root/hadoop/share/hadoop/yarn/hadoop-yarn-server-common-2.4.1.jar:/root/hadoop/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.4.1.jar:/root/hadoop/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.4.1.jar:/root/hadoop/share/hadoop/yarn/hadoop-yarn-server-tests-2.4.1.jar:/root/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.4.1.jar:/root/hadoop/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.4.1.jar:/root/hadoop/share/hadoop/yarn/hadoop-yarn-api-2.4.1.jar:/root/hadoop/share/hadoop/yarn/hadoop-yarn-client-2.4.1.jar:/root/hadoop/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.4.1.jar:/root/hadoop/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.4.1.jar:/root/hadoop/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.4.1.jar:/root/hadoop/share/hadoop/mapreduce/lib/asm-3.2.jar:/root/hadoop/share/hadoop/mapreduce/lib/xz-1.0.jar:/root/hadoop/share/hadoop/mapreduce/lib/hamcrest-core-1.1.jar:/root/hadoop/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/root/hadoop/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/root/hadoop/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/root/hadoop/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/root/hadoop/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/root/hadoop/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.8.8.jar:/root/hadoop/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/root/hadoop/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/root/hadoop/share/hadoop/mapreduce/lib/javax.inject-1.jar:/root/hadoop/share/hadoop/mapreduce/lib/hadoop-annotations-2.4.1.jar:/root/hadoop/share/hadoop/mapreduce/lib/jackson-core-asl-1.8.8.jar:/root/hadoop/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/root/hadoop/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/root/hadoop/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/root/hadoop/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/root/hadoop/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/root/hadoop/share/hadoop/mapreduce/lib/junit-4.10.jar:/root/hadoop/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/root/hadoop/share/hadoop/mapreduce/lib/guice-3.0.jar:/root/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.4.1.jar:/root/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.4.1.jar:/root/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.4.1.jar:/root/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.4.1.jar:/root/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.4.1.jar:/root/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.4.1.jar:/root/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.4.1-tests.jar:/root/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.4.1.jar:/root/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.4.1.jar:/root/hadoop/contrib/capacity-scheduler/*.jar
STARTUP_MSG: build = http://svn.apache.org/repos/asf/hadoop/common -r 1604318; compiled by 'jenkins' on 2014-06-21T05:43Z
STARTUP_MSG: java = 1.7.0_15
************************************************************/
15/06/19 19:31:39 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
15/06/19 19:31:39 INFO namenode.NameNode: createNameNode [-bootstrapStandby]
15/06/19 19:31:41 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
=====================================================
About to bootstrap Standby ID bnn2 from:
Nameservice ID: bns1
Other Namenode ID: bnn1
Other NN's HTTP address: http://v1:50070
Other NN's IPC address: v1/192.168.32.91:9000
Namespace ID: 32897138
Block pool ID: BP-134496651-192.168.32.91-1434751114027
Cluster ID: CID-c5890502-45a9-425e-9ae3-aa70c3eff8a6
Layout version: -56
=====================================================
15/06/19 19:31:44 INFO common.Storage: Storage directory /root/hadoop/tmp/dfs/name has been successfully formatted.
15/06/19 19:31:45 INFO namenode.TransferFsImage: Opening connection to http://v1:50070/imagetransfer?getimage=1&txid=0&storageInfo=-56:32897138:0:CID-c5890502-45a9-425e-9ae3-aa70c3eff8a6
15/06/19 19:31:46 INFO namenode.TransferFsImage: Image Transfer timeout configured to 60000 milliseconds
15/06/19 19:31:46 INFO namenode.TransferFsImage: Transfer took 0.01s at 0.00 KB/s
15/06/19 19:31:46 INFO namenode.TransferFsImage: Downloaded file fsimage.ckpt_0000000000000000000 size 350 bytes.
15/06/19 19:31:46 INFO util.ExitUtil: Exiting with status 0
15/06/19 19:31:46 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at v2/192.168.32.92
************************************************************/4.hbase啟動時，如果不同節點時間不同的話，啟動只有HMaster會起來，別的HRegionServer都會掛掉。同步時間即可，HBase需要用到hadoop的{core-site.xml/hdfs-site.xml}，否則會將配置好的ha hdfsname（hdfs://bns1/hbase）當成hostname去解析，把hdfs相關檔案拷貝到hbase/conf下就能正常運行了。 5.hive配置元資料快記法： 4個配置項字首：javax.jdo.option.Connection[URL/UserName/Password/DriverName]
createDatabaseIfNotExist 配置mysql允許遠端連線： #(執行下面的語句 *.*:所有庫下的所有表 %：任何IP地址或主機都可以連線) GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '123' WITH GRANT OPTION; FLUSH PRIVILEGES; 如果這麼配置完了之後，可以遠端登入但是本機登入不了的時候。可以用mysqladmin給本地重新修改下密碼
/usr/bin/mysqladmin -uroot -h currentHostName password 'new-password'
相關推薦
Hadoop 2.4.1 搭建Ha遇到問題記錄
15/06/19 17:12:44 ERROR namenode.FSNamesystem: FSNamesystem initialization failed. java.io.IOException: Invalid configuration: <span s
Hadoop-2.4.1叢集搭建步驟
首先說明，博主以前也沒學過Hadoop，更別提Linux。只不過買了點資料，覺得不看也是浪費，索性找了時間學習了一下，別說，還真是吸引人。 言歸正傳，博主機器：筆記本 Y50-70 美版的 搭建環境：VM11（VMware Workstation 11），使用的是CenOS
Hadoop-2.4.1學習之edits和fsimage查看器
文件的 順序 rup oev 兼容 require aps block mean 在hadoop中edits和fsimage是兩個至關關鍵的文件。當中edits負責保存自最新檢查點後命名空間的變化。起著日誌的作用，而fsimage則保存了最新的檢查點信息
VirtualBox+Centos7+(jdk1.7.0_71+Hadoop-2.6.0)/(jdk1.10+Hadoop-2.9.1)搭建完全分散式叢集平臺
VirtualBox+Centos7+(jdk1.7.0_71+Hadoop-2.6.0)/(jdk1.10+Hadoop-2.9.1)搭建完全分散式叢集平臺 本文有很多是自定義的，可以根據自己的實際情況和需求修改，儘量會用紅色標註出來，當然按照步驟，一步一步應該也能成功，不
Hadoop-2.4.1學習之FileSystem及實戰
一提到hadoop檔案系統，通常想到的就是HDFS，即Hadoop Distributed File System，但除了HDFS外，hadoop還支援其它型別的檔案系統，比如Amazon S3、Swift檔案系統等，而這些檔案系統都擴充套件自抽象基礎類FileS
Hadoop-2.4.1學習之HDFS檔案許可權和ACL
之前在論壇看到一個關於HDFS許可權的問題，當時無法回答該問題。無法回答並不意味著對HDFS許可權一無所知，而是不能準確完整的闡述HDFS許可權，因此決定系統地學習HDFS檔案許可權。HDFS的檔案和目錄許可權模型共享了POSIX（Portable Opera
Hadoop-2.8.5的HA集群搭建
適配 bin tom proto format 切換 enabled 完成 數字 一、Hadoop HA 機制的學習 1.1、Hadoop 2.X 的架構圖 2.x版本中，HDFS架構解決了單點故障問題，即引入雙NameNode架構，同時借助共享存儲
Hive-0.13.1本地獨立模式安裝 元資料儲存到MariaDB-10.1.0 Hadoop-2.4.0
tar -zxvf apache-hive-0.13.1-bin.tar.gz 解壓後,編輯java.sh(java.sh為自己建立的指令碼): vim /etc/profile.d/java.sh export HIVE_HOME=/opt/modules/hive/apache-hive-0.13.1
Centos7 實現Hadoop-2.9.1分散式叢集搭建和部署（三臺機器）
一、準備三臺虛擬機器hadoop 192.168.131.128 localhost131 192.168.131.131 localhost134 192.168.131.134(以上是我的三臺虛擬機器的hostname 和 ip)hadoop 是 master 的 hos
CentOS 7 搭建 Hadoop 2.9.1 遇到的坑之 storage directory does not exist or is not accessible.
2018-06-14 11:42:28,500 WARN org.apache.hadoop.hdfs.server.common.Storage: Storage directory /home/hadoopCluster/hadoop-2.9.1/tmp/dfs/name
Hadoop-2.7.1叢集環境搭建
摘自：http://blog.csdn.net/u014039577/article/details/49813531 由於日誌資料量越來越大，資料處理的邏輯越來越複雜，同時還涉及到大量日誌需要批處理，當前的flume-kafka-storm-Hbase-web這一套流程已經不能滿足當前的需求了，所以只
基於SpringMVC4.3.2+Spring4.3.2+MyBatis3.4.1搭建SSM框架
終於到了框架搭建的最後一步，實現我們的終極目標SpringMVC+Spring+MyBatis的SSM框架，這篇文章也是基於之前搭建的SpringMVC+Spring+Hibernate框架演變過來的，所以沒看過之前幾篇文章的同學請乘傳送帶。 在之前搭
Ubuntu 14.04 (32位)上搭建Hadoop 2.5.1單機和偽分散式環境
引言 一直用的Ubuntu 32位系統（準備下次用Fedora，Ubuntu越來越不適合學習了），今天準備學習一下Hadoop，結果下載Apache官網上釋出的最新的封裝好的2.5.1版，配置完了根本啟動不起來，檢視錯誤日誌發現是native庫的版本和系統不一致，使用fil
Canalv1.1.4版本搭建HA叢集
## 前提 `Canal`上一個正式版是於`2019-9-2`釋出的`v1.1.4`，筆者幾個月前把這個版本的`Canal`推上了生產環境，部署了`HA`叢集。過程中雖然遇到不少的坑，但是在不出問題的前提下，`Canal`的作用還是非常明顯的。上週的一次改造上線之後，去掉了原來對業務系統訂單資料通過`Rab
2.4.1 集合基礎知識
這一 也不能 time sed OS pan 逗號 運算 提示 集合是無序可變序列，使用一對大括號（作者溫馨提示：這一點和字典很類似，千萬不要搞混啊）作為界定符，元素之間使用逗號分隔，同一個集合內的每個元素都是唯一的，元素之間不允許重復。 在Python中，直接將集
elasticsearch 6.2.4運行問題處理記錄
elasticsearch問題一：max number of threads [2048] for user [work] likely too low, increase to at least [4096]解決方法：進入limits.d下的配置文件：vi /etc/security/limits.d/90
Hadoop 2.9.1 on Ubuntu 16.04
serve 用戶 datanode syn cat bashrc 格式 pat mixed Hadoop 2.9.1 on Ubuntu 16.04 環境配置 3臺機器虛擬機 ubuntu16.04 10.64.104.177 hadoop-master 10.64.10
CentOS 7 快速架設hadoop 2.5.1叢集
經過幾天的折騰，總算成功了~賬號註冊到現狀近N年了，這是第一篇部落格~ 1、軟體準備a.OS:CentOS-7.0-1406-x86_64-DVD.iso --去CentOS官網下 b.JDK:jdk-7u71-linux-x64.gz --oracle官網下，至於1
Hadoop-2.5.1安裝步驟
目標 在vmware14.1.1中的三個虛擬機器上安裝hadoop 2.5.1 穩定版本。 由於hadoop 2.x.x 都是同一個系列，所以其他hadoop 2.x.x版本的安裝可以參照這篇步驟來做。 環境介紹 三臺vmware-14.1.1中的虛擬機器 作業系統：
2 -4 -1 list列表練習解答
black new cart choose pen attr ever 枚舉 -- 答案 1 >>> names = [] >>> names.append(‘old_driver‘) >>> names