Hadoop家族環境搭建各種版本選擇
System Requirements
Supported Platforms
• GNU/Linuxis supported as a development and production platform for both server andclient.
• SunSolaris is supported as a development and production platform for both serverand client.
• FreeBSDis supported as a development and production platform for both server andclient.
• Win32is supported as a development platformonly for both server and client.
• Win64is supported as a development platformonly for both server and client.
• MacOSXis supported as a development platformonly for both server and client.
Hive™ Requirements
• Java1.7
Note: Hive versions
• Hadoop2.x (preferred), 1.x (not supported by Hive 2.0.0 onward).
Hive versions up to 0.13 also supported Hadoop 0.20.x, 0.23.x.
• Hiveis commonly used in production Linux and Windows environment. Mac is a commonlyused development environment. The instructions in this document are applicableto Linux and Mac. Using it on Windows would require slightly differentsteps.
HBase™ Requirements
This section lists required servicesand some required system configuration.
Table2. Java
|
HBase Version |
JDK 6 |
JDK 7 |
JDK 8 |
|
1.1 |
yes |
Running with JDK 8 will work but is not well tested. |
|
|
1.0 |
yes |
Running with JDK 8 will work but is not well tested. |
|
|
0.98 |
yes |
yes |
Running with JDK 8 works but is not well tested. Building with JDK 8 would require removal of the deprecated remove() method of the PoolMap class and is under consideration. See HBASE-7608 for more information about JDK 8 support. |
|
0.94 |
yes |
yes |
N/A |
Hadoop Requirements
Hadoop version support matrix
• "S"= supported
• "X"= not supported
• "NT"= Not tested
|
HBase-0.94.x |
HBase-0.98.x (Support for Hadoop 1.1+ is deprecated.) |
HBase-1.0.x (Hadoop 1.x is NOT supported) |
HBase-1.1.x |
HBase-1.2.x |
|
|
Hadoop-1.0.x |
X |
X |
X |
X |
X |
|
Hadoop-1.1.x |
S |
NT |
X |
X |
X |
|
Hadoop-0.23.x |
S |
X |
X |
X |
X |
|
Hadoop-2.0.x-alpha |
NT |
X |
X |
X |
X |
|
Hadoop-2.1.0-beta |
NT |
X |
X |
X |
X |
|
Hadoop-2.2.0 |
NT |
S |
NT |
NT |
X |
|
Hadoop-2.3.x |
NT |
S |
NT |
NT |
X |
|
Hadoop-2.4.x |
NT |
S |
S |
S |
S |
|
Hadoop-2.5.x |
NT |
S |
S |
S |
S |
|
Hadoop-2.6.0 |
X |
X |
X |
X |
X |
|
Hadoop-2.6.1+ |
NT |
NT |
NT |
NT |
S |
|
Hadoop-2.7.0 |
X |
X |
X |
X |
X |
|
Hadoop-2.7.1+ |
NT |
NT |
NT |
NT |
S |
Pig™ Requirements
Unix and Windows users need thefollowing:
• Hadoop 0.23.X, 1.X or 2.X- http://hadoop.apache.org/common/releases.html(You can run Pig with different versions of Hadoop by setting HADOOP_HOME topoint to the directory where you have installed Hadoop. If you do not setHADOOP_HOME, by default Pig will run with the embedded version, currentlyHadoop 1.0.4.)
Optional
• Python 2.7- https://www.python.org(when using Streaming Python UDFs)
Nutch 2.3.1 Requirements
The recommended Gora backends for thisNutch release are
• ApacheAvro 1.7.6
• ApacheHadoop 1.2.1 and 2.5.2
• ApacheHBase 0.98.8-hadoop2 (although also tested with 1.X)
• ApacheCassandra 2.0.2
• ApacheSolr 4.10.3
• MongoDB2.6.X
• ApacheAccumlo 1.5.1
• ApacheSpark 1.4.1
Nutch 2.3 Requirements
The supported Apache Gorav0.5 backends are;
ZooKeeper runs in Java, release 1.6 orgreater (JDK 6 or greater, FreeBSD support requires openjdk7). It runs as an ensemble of ZooKeeper servers. ThreeZooKeeper servers is the minimum recommended size for an ensemble, and we alsorecommend that they run on separate machines. At Yahoo!, ZooKeeper is usuallydeployed on dedicated RHEL boxes, with dual-core processors, 2GB of RAM, and80GB IDE hard drives.
Chukwa™ Requirements
Chukwa should work on any POSIXplatform, but GNU/Linux is the only production platform that has been testedextensively. Chukwa has also been used successfully on Mac OS X, which severalmembers of the Chukwa team use for development.
The only absolute software requirementsare Java 1.6 or better and Hadoop 0.20.205.0+. HICC, the Chukwa visualizationinterface, requires HBase 0.90.4.
The Chukwa cluster management scriptsrely on ssh; these scripts, however, are not required if you have somealternate mechanism for starting and stopping daemons.
Spark™ Requirements
Mahout™ Requirements
Avro™ Requirements
Tez™ Requirements