mac 下hadoop安裝並執行例子
阿新 • • 發佈:2019-02-10
1 安裝
#brew install hadoop
安裝的是2.6.0,目錄為/usr/local/Cellar/hadoop,如果想安裝其他版本,則下載tar包解壓即可。地址:http://mirrors.cnnic.cn/apache/hadoop/common/
2 配置
將hadoop可執行路徑bin和sbin都配置到環境變數中
export HADOOP_HOME=/usr/local/Cellar/hadoop/2.6.0/libexec
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.7.0_67.jdk/Contents/home
export PATH =.:$JAVA_HOME/bin:$HADOOP_HOME/sbin/:$HADOOP_HOME/bin/:$PATH
接下來配置偽分散式:
(1)
配置core-site.xml
➜ hadoop cat core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>fs.default.name</name >
<value>hdfs://localhost:9000</value>
</property>
</configuration>
(2)hdfs的節點備份設定為1
配置hdfs-site.xml
➜ hadoop cat hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property >
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
(3)配置mapred-site.xml
➜ hadoop cat mapred-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>hdfs://localhost:9000/</value>
</property>
</configuration>
(4)將hadoop-env.sh裡jdk路徑配置好即可。
然後執行:格式化namenode
➜ hadoop namenode -format
➜ hadoop start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
15/03/29 22:35:41 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
15/03/29 22:35:43 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [localhost]
localhost: namenode running as process 7082. Stop it first.
localhost: datanode running as process 7163. Stop it first.
Starting secondary namenodes [0.0.0.0]
0.0.0.0: secondarynamenode running as process 7266. Stop it first.
15/03/29 22:35:47 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
15/03/29 22:35:49 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
starting yarn daemons
resourcemanager running as process 7377. Stop it first.
localhost: nodemanager running as process 7466. Stop it first.
➜ hadoop jps
7163 DataNode
9619 Jps
7466 NodeManager
7377 ResourceManager
7082 NameNode
7266 SecondaryNameNode
➜ hadoop
可以看到五個節點都啟動了。
3 執行例子wordcount
先建立目錄input 以及檔案並上傳到hdfs
➜ libexec ls input
file1.txt file2.txt
➜libexec hadoop fs -put input/file*.txt /input
➜ libexec hadoop fs -ls /input
15/03/29 22:46:37 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
15/03/29 22:46:39 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 2 items
-rw-r--r-- 1 shenyb supergroup 35 2015-03-29 11:02 /input/file1.txt
-rw-r--r-- 1 shenyb supergroup 38 2015-03-29 11:02 /input/file2.txt
➜ libexec hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar wordcount /input /output
執行如果不出錯,結果輸出到output目錄如下:
➜ libexec hadoop fs -ls /output
15/03/29 22:56:29 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
15/03/29 22:56:30 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 2 items
-rw-r--r-- 1 shenyb supergroup 0 2015-03-29 12:14 /output/_SUCCESS
-rw-r--r-- 1 shenyb supergroup 67 2015-03-29 12:14 /output/part-r-00000
➜ libexec hadoop fs -cat /output/part-r-00000
15/03/29 22:56:50 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
15/03/29 22:56:51 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
ai 2
am 3
am2 1
are 2
i 2
man 1
man2 1
shenya 2
who 2
you 1
you2 1
➜ libexec