vs2017編譯haddoop3.1.0
阿新 • • 發佈：2019-02-15
我是在windows10 64位系統使用Visual Studio 2017編譯hadoop 3.1.0原始碼
按照官方文件（BUILDING.txt），編譯hadoop需要以下環境
1、 JDK 1.8
jdk-8u171-windows-x64.exe 下載地址
2、 Maven 3.0 or later
apache-maven-3.5.3-bin.zip下載地址
3、ProtocolBuffer 2.5.0
protoc-2.5.0-win32.zip下載地址
4、 CMake 3.1 or newer
cmake-3.11.3-win64-x64.zip 下載地址
6、 Windows SDK 8.1 (if building CPU rate control for the container executor)
7、zlib headers (if building native code bindings for zlib)
zlib-1.2.11.tar.gz 下載地址(可以不用)
8、 Internet connection for first build (to fetch all Maven and Hadoop dependencies)
聯網是必須的
10、 Python ( for generation of docs using 'mvn site')
Python 3.6.5 下載地址（可以不用）
下載所需的工具，將他們路徑加到你的環境變數裡。
在命令列執行適用於 VS 2017 的 x64 本機工具命令提示，設定平臺為set Platform=x64
一個簡單的批處理，設定環境變數
@echo hadoop 編譯環境設定
@echo 設定編譯平臺
@set Platform=x64
@echo 設定C編譯環境
@call d:\tools\vs\2017\Community\VC\Auxiliary\Build\vcvars64.bat
@echo 加入maven路徑
@set path=D:\tools\maven\bin;%path%
@echo 加入protc路徑
@set path=D:\tools\protoc;%path%
@echo 加入bash路徑
@set path=D:\tools\Git\bin;%path%
@echo 加入cmake路徑
@set path=D:\tools\cmake\bin;%path% 由於預設是按照Visual Studio 2010環境配置的，我的是Visual Studio 2017，有幾個檔案需要修改一下首先修改的是兩個vs的工程檔案
用vs2017開啟hadoop-common-project\hadoop-common\src\main\native目錄下native.sln檔案，將工具和SDK定向到VS2017的版本如圖
同理還有hadoop-common-project\hadoop-common\src\main\winutils目錄下的winutils.sln，也要定向到vs2017的工具集
修改hadoop-hdfs-project\hadoop-hdfs-native-client目錄下的pom.xml檔案，將141行左右的
<condition property="generator" value="Visual Studio 10" else="Visual Studio 10 Win64">改為
<condition property="generator" value="Visual Studio 10" else="Visual Studio 15 Win64">cd進入原始碼目錄，然後根據執行mvn package -Pdist,native-win -DskipTests -Dtar。等待編譯完成，hadoop-dist\target\hadoop-3.1.0就有對應的hadoop可用目錄
編譯結果
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Hadoop HDFS Native Client 3.1.0 ............. SUCCESS [ 46.213 s]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 16.726 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [ 5.358 s]
[INFO] Apache Hadoop HDFS-RBF ............................. SUCCESS [ 24.532 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [ 0.113 s]
[INFO] Apache Hadoop YARN ................................. SUCCESS [ 0.094 s]
[INFO] Apache Hadoop YARN API ............................. SUCCESS [ 37.818 s]
[INFO] Apache Hadoop YARN Common .......................... SUCCESS [01:27 min]
[INFO] Apache Hadoop YARN Registry ........................ SUCCESS [ 9.039 s]
[INFO] Apache Hadoop YARN Server .......................... SUCCESS [ 0.080 s]
[INFO] Apache Hadoop YARN Server Common ................... SUCCESS [ 25.877 s]
[INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [ 34.372 s]
[INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [ 3.654 s]
[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [ 10.063 s]
[INFO] Apache Hadoop YARN Timeline Service ................ SUCCESS [ 7.387 s]
[INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [ 58.543 s]
[INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [ 1.698 s]
[INFO] Apache Hadoop YARN Client .......................... SUCCESS [ 9.020 s]
[INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [ 3.970 s]
[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [ 3.358 s]
[INFO] Apache Hadoop YARN TimelineService HBase Backend ... SUCCESS [ 0.079 s]
[INFO] Apache Hadoop YARN TimelineService HBase Common .... SUCCESS [ 11.089 s]
[INFO] Apache Hadoop YARN TimelineService HBase Client .... SUCCESS [ 7.618 s]
[INFO] Apache Hadoop YARN TimelineService HBase Servers ... SUCCESS [ 0.083 s]
[INFO] Apache Hadoop YARN TimelineService HBase Server 1.2 SUCCESS [ 3.598 s]
[INFO] Apache Hadoop YARN TimelineService HBase tests ..... SUCCESS [ 4.049 s]
[INFO] Apache Hadoop YARN Router .......................... SUCCESS [ 5.197 s]
[INFO] Apache Hadoop YARN Applications .................... SUCCESS [ 0.089 s]
[INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [ 3.055 s]
[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [ 2.027 s]
[INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [ 0.173 s]
[INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [ 53.781 s]
[INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [ 24.747 s]
[INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [ 3.953 s]
[INFO] Apache Hadoop MapReduce App ........................ SUCCESS [ 15.146 s]
[INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [ 7.131 s]
[INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [ 9.522 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [ 0.985 s]
[INFO] Apache Hadoop YARN Services ........................ SUCCESS [ 0.075 s]
[INFO] Apache Hadoop YARN Services Core ................... SUCCESS [ 3.959 s]
[INFO] Apache Hadoop YARN Services API .................... SUCCESS [ 1.262 s]
[INFO] Apache Hadoop YARN Site ............................ SUCCESS [ 0.092 s]
[INFO] Apache Hadoop YARN UI .............................. SUCCESS [ 0.073 s]
[INFO] Apache Hadoop YARN Project ......................... SUCCESS [ 18.167 s]
[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [ 2.072 s]
[INFO] Apache Hadoop MapReduce NativeTask ................. SUCCESS [ 6.425 s]
[INFO] Apache Hadoop MapReduce Uploader ................... SUCCESS [ 1.995 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [ 8.095 s]
[INFO] Apache Hadoop MapReduce ............................ SUCCESS [ 5.299 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [ 12.191 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [ 9.260 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [ 1.863 s]
[INFO] Apache Hadoop Archive Logs ......................... SUCCESS [ 1.992 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [ 8.485 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [ 4.835 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [ 2.299 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [ 2.983 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [ 0.067 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [ 5.312 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [ 23.122 s]
[INFO] Apache Hadoop Kafka Library support ................ SUCCESS [ 2.702 s]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [ 7.845 s]
[INFO] Apache Hadoop Aliyun OSS support ................... SUCCESS [ 3.313 s]
[INFO] Apache Hadoop Client Aggregator .................... SUCCESS [ 4.009 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 6.494 s]
[INFO] Apache Hadoop Resource Estimator Service ........... SUCCESS [ 6.009 s]
[INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [ 3.185 s]
[INFO] Apache Hadoop Image Generation Tool ................ SUCCESS [ 3.122 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 11.333 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [ 0.115 s]
[INFO] Apache Hadoop Client API ........................... SUCCESS [ 39.096 s]
[INFO] Apache Hadoop Client Runtime ....................... SUCCESS [ 29.791 s]
[INFO] Apache Hadoop Client Packaging Invariants .......... SUCCESS [ 0.378 s]
[INFO] Apache Hadoop Client Test Minicluster .............. SUCCESS [ 44.059 s]
[INFO] Apache Hadoop Client Packaging Invariants for Test . SUCCESS [ 0.160 s]
[INFO] Apache Hadoop Client Packaging Integration Tests ... SUCCESS [ 0.287 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [04:56 min]
[INFO] Apache Hadoop Client Modules ....................... SUCCESS [ 0.077 s]
[INFO] Apache Hadoop Cloud Storage ........................ SUCCESS [ 0.789 s]
[INFO] Apache Hadoop Cloud Storage Project 3.1.0 .......... SUCCESS [ 0.081 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 18:37 min
[INFO] Finished at: 2018-06-14T09:04:32+08:00