下載hadoop

配置hadoop

hadoop有兩種型別的配置檔案：
1.只讀配置檔案 - core-default.xml, hdfs-default.xml, yarn-default.xml 和 mapred-default.xml
2.可配置的配置檔案 etc/hadoop/core-site.xml, etc/hadoop/hdfs-site.xml, etc/hadoop/yarn-site.xml and etc/hadoop/mapred-site.xml

也可以通過etc/hadoop/hadoop-env.sh and etc/hadoop/yarn-env.sh來修改hadoop和yarn的執行時環境

修改hadoop的配置之前需要先配置ssh免密登入。如果對ssh免密登入不瞭解的可以先行百度。
首先貼上core-site.xml

<configuration>
        <property>
                <name>fs.defaultFS</name>
                <value>hdfs://hadoop:9000</value>
        </property>
        <property>
                <name
 
>hadoop.tmp.dir</name>
                <value>/home/hadoop/tmp</value>
        </property>
        <property>
                <name>io.file.buffer.size</name>
                <value>131072</value>
        </property>
</configuration>

hdfs-site.xml

<configuration>
        <property>
                <name>dfs.namenode.name.dir</name>
                <value>/home/hadoop/data/namenode</value>
        </property>
        <property>
                <name>dfs.datanode.data.dir</name>
                <value>/home/hadoop/data/datanode</value>
        </property>
</configuration>

yarn-site.xml

<configuration>

<!-- Site specific YARN configuration properties -->
        <property>
                <name>yarn.resourcemanager.hostname</name>
                <value>hadoop</value>
        </property>
        <property>
                <name>yarn.log-appregation-enable</name>
                <value>true</value>
        </property>
        <property>
                <name>yarn.nodemanager.aux-services</name>
                <value>mapreduce_shuffle</value>
        </property>
        <property>
                <name>yarn.nodemanager.env-whitelist</name>
                <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PR
EPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
        </property>
</configuration>

mapred-site.xml,需要修改mr的計算框架為yarn，修改的時候一定要修改yarn.nodemanager.env-whitelist配置項

<configuration>
    <!-- 預設的是用mr框架來計算 -->
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
</configuration>