Spark啟動報 java.lang.ClassNotFoundException: parquet.hadoop.ParquetOutputCommitter
我安裝的是hadoop-2.6.0-cdh5.12.1和spark-1.6.0-cdh5.12.1
解決的版本是
將下面的jar包下載下來放到Spark的啟動ClassPath下,然後重啟Spark

<dependency>
    <groupId>com.twitter</groupId>
    <artifactId>parquet-hadoop</artifactId>
 

    <version>1.4.3</version>
</dependency>

執行

bin/spark-shell

可以正常啟動

[email protected]:~/install-soft/bigdata/spark-1.6.0-cdh5.12.1/bin$ ./spark-shell 
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel).
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
 

   /___/ .__/\_,_/_/ /_/\_\   version 1.6.0
      /_/

Using Scala version 2.10.5 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_144)
Type in expressions to have them evaluated.
Type :help for more information.
17/09/19 16:40:28 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using
 
 builtin-java classes where applicable
17/09/19 16:40:29 WARN util.Utils: Your hostname, Levin-PC resolves to a loopback address: 127.0.1.1; using 10.236.103.148 instead (on interface enp0s3)
17/09/19 16:40:29 WARN util.Utils: Set SPARK_LOCAL_IP if you need to bind to another address
Spark context available as sc (master = local[*], app id = local-1505810430958).
SQL context available as sqlContext.
scala>