spark之 Idea操作
阿新 • • 發佈:2020-09-21
一、pom.xml
<!-- spark --> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-core_2.11</artifactId> <version>2.3.4</version> </dependency> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-sql_2.11</artifactId> <version>2.3.4</version> </dependency> <!-- log4j 1.2.17 --> <dependency> <groupId>log4j</groupId> <artifactId>log4j</artifactId> <version>1.2.17</version> </dependency> <!-- https://mvnrepository.com/artifact/com.google.guava/guava --> <dependency> <groupId>com.google.guava</groupId> <artifactId>guava</artifactId> <version>14.0.1</version> </dependency>
因為需要打包scala的程式碼和maven胖包:
<build> <!-- 縮短jar包名字 --> <finalName>myspark</finalName> <plugins> <!-- scala --> <plugin> <groupId>org.scala-tools</groupId> <artifactId>maven-scala-plugin</artifactId> <version>2.15.2</version> <executions> <execution> <goals> <goal>compile</goal> <goal>testCompile</goal> </goals> </execution> </executions> </plugin> <!-- 胖包 --> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-assembly-plugin</artifactId> <version>2.5.5</version> <configuration> <archive> <manifest> <mainClass>com.njbdqn.MySpark</mainClass> </manifest> </archive> <descriptorRefs> <descriptorRef>jar-with-dependencies</descriptorRef> </descriptorRefs> </configuration> <executions> <execution> <id>make-assembly</id> <phase>package</phase> <goals> <goal>single</goal> </goals> </execution> </executions> </plugin> </plugins> </build>
二、關於scala的配置,setting中java compiler-> 1.8 ;project structure -> 8; libraries 中新增scala-sdk
三、目錄
四、spark程式碼
// 1, sparkConf val conf = new SparkConf().setMaster("local[2]").setAppName("myjob")// 本地2核 ;*表示所有的核 // 2, sparkcontext val sc = new SparkContext(conf) //建立SparkContext,該物件是提交spark App的入口 // 3, 使用sc建立RDD並執行相應的transformation和action sc.textFile("file:////root/sparkdata/wordcount.txt") // sc.textFile("file:////D:\\idea\\ideaProjects\\spark_projects\\myspark8\\src\\main\\scala\\com\\njbdqn\\wordcount.txt") .flatMap(_.split(" ")).map((_,1)).reduceByKey(_+_).foreach(println(_)) // 4, 停止sc,結束該任務 sc.stop()
五、maven打包package後,把target下胖包(是with-dependences那個jar包)放到linux下java -jar 路徑 執行
./spark-submit --class com.njbdqn.MySpark /root/myspark-jar-with-dependencies.jar