本地環境資訊 spark2.1.0 hadoop:2.6.0 hive:2.1.1 scala:2.11.8 java:1.8.0_141 mysql

1 保證hive可用，配置hive-site.xml如下，下面的配置使用的是mysql作為元資料儲存工具，所以還需要將mysql驅動包複製到hive安裝路徑的lib目錄下，需要配置hive使用mysql座位底層儲存工具schematool -initSchema -dbType mysql。

<configuration>  
   <property>  
        <name>javax.jdo.option.ConnectionURL</name>  
        <value>jdbc:mysql://10.118.58.27:3306/hive?createDatabaseIfNotExist=true</value>  
    </property>  
    <property>  
        <name>javax.jdo.option.ConnectionDriverName</name>  
        <value>com.mysql.jdbc.Driver</value>  
    </property>  
    <property>  
        <name>javax.jdo.option.ConnectionUserName</name>  
        <value>××××</value>  
    </property>  
    <property>  
        <name>javax.jdo.option.ConnectionPassword</name>  
        <value>××××</value>  
    </property>  
    <property>    
   <name>hive.metastore.schema.verification</name>    
   <value>false</value>    
    <description>    
    Enforce metastore schema version consistency.    
    True: Verify that version information stored in metastore matches with one from Hive jars.  Also disable automatic    
          schema migration attempt. Users are required to manully migrate schema after Hive upgrade which ensures    
          proper metastore schema migration. (Default)    
    False: Warn if the version information stored in metastore doesn't match with one from in Hive jars.    
    </description>    
 </property> 
 <property>
	<name>hive.server2.thrift.bind.host</name>
	<value>bigData</value>
 </property>
 <property>
        <name>hive.server2.thrift.port</name>
        <value>10000</value>
 </property>
</configuration>  
 

3 spark程式設計，我使用的是idea，通過maven建立scala工程，如果要通過本地除錯，需要將hive-site.xml複製到工程的resources目錄下，同時將mysql/spark_hive的jar包也新增到專案路徑下，由於spark2.1.0使用sparkSession，支援hive，所以在初始化sparkSession時只需啟用hive即可，如下：

objectTestWork {
  def 
 
main(args: Array[String]): Unit = {
    val sparkConf = new SparkConf()
    args.map(_.split("=")).filter(_.size==2).map(x=>sparkConf.set(x(0),x(1)))
    val spark = SparkSession.builder()
      .config(sparkConf)
      .appName("test").enableHiveSupport()
//      .master("spark://10.118.58.27:7077")
.getOrCreate()
    spark.sql("select * from selfwork.srczkw"
 
).show()
  }
}

web預設埠：

namenode的webUI埠：50070
yarn的web埠：8088
spark叢集的web埠：8080
spark-job監控埠：4040