如何使程式在eclipse上Run As,在叢集中Run
一、First Step
將配置檔案拿出來包括(core-site.xml、hdfs-site.xml、mapred-site.xml、yarn-site.xml)
注意:修改配置檔案!把主機名改成對應的埠號(或者在window下設定hosts配置主機名和ip的對映)
拿出來之後放到你所要執行類的src下!如圖所示:
因為我要執行WordCount類,所以我將這四個配置檔案放入到了此專案的src下。
二、Second Step
在你所寫的(WordCount.java)類中的主方法裡新增一下內容:
如圖:
其中:
Path pathInput = new Path("/betty");
Path pathOutput = new Path("/betty/output");
可以改為:
Path pathInput = new Path(args[0]);
Path pathOutput = new Path(args[1]);
但是要將路徑寫到 Run As ----Run Configurations---Arguments中,將不同路徑空格分開。如圖:
注意:如果沒有Run Configurations就先將程式Run As以下,就會出現。還要檢查一下Main中包和專案的資訊是否為你所要執行的專案資訊,如果不是改過來。如圖:
注意先檢查左邊框框中的Java Application中是否有你所要執行的類的名字,沒有的話,可以嘗試點選,該框框上方的第一個圖示。
其實是添加了以下兩條內容:
1、conf.set("mapreduce.app-submission.cross-platform", "true");
注意:!!此條內容要放在(Job job = Job.getInstance(conf, "word count");)前邊(如圖所示),否則會報以下錯誤 Exception message: /bin/bash: line 0: fg: no job control Stack trace: ExitCodeException exitCode=1: /bin/bash: line 0: fg: no job control
因為Job.getInstance(conf,"word count");是從Configuration中獲取的資訊,所以在獲取資訊之前,一定要將此conf資訊更改好,否則你獲取的conf資訊還是和沒有conf.set("mapreduce.app-submission.cross-platform", "true")時的資訊一樣。
或者是你可以直接去mapred-site.xml中更改mapreduce.app-submission.cross-platform此屬性為true,如下:<property> <description>If enabled, user can submit an application cross-platform i.e. submit an application from a Windows client to a Linux/Unix server or vice versa. </description> <name>mapreduce.app-submission.cross-platform</name> <value>true</value> </property>
2、job.setJar("mc.jar");
注意:!!!此項括號內的字串名稱和Third Step有關
三、Third Step
將mrmission此專案打成jar包,放在專案下(請注意是專案下,也就是此圖中的mrmission下)!
打包步驟:選中你所要打包的專案--右鍵---Export---Java--JAR File---next---JAR file(設定生成jar file的位置)--next---next---main class(選擇你所要執行的類--我的選擇為WordCount)--finish
四、點選Run As --->選擇--->Run On Hadoop--->執行成功。
五、完整程式碼
package com.hyxy.mrmission;
import java.io.IOException;
import java.util.StringTokenizer;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
public class WordCount {
public static class MyMapper extends Mapper<Object, Text, Text, IntWritable> {
Text word = new Text();
final IntWritable one = new IntWritable(1);
@Override
protected void map(Object key, Text value, Context context) throws IOException, InterruptedException {
StringTokenizer itr = new StringTokenizer(value.toString());
while (itr.hasMoreTokens()) {
word.set(itr.nextToken());
context.write(word, one);
}
}
}
public static class MyReducer extends Reducer<Text, IntWritable, Text, IntWritable> {
final IntWritable result = new IntWritable();
@Override
protected void reduce(Text key, Iterable<IntWritable> values, Context context)
throws IOException, InterruptedException {
int sum = 0;
for (IntWritable val : values) {
sum += val.get();
}
result.set(sum);
context.write(key, result);
}
}
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
conf.set("mapreduce.app-submission.cross-platform", "true");
Job job = Job.getInstance(conf, "word count");
job.setJarByClass(WordCount.class);
job.setJar("mc.jar");
job.setMapperClass(MyMapper.class);
job.setReducerClass(MyReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
FileInputFormat.addInputPath(job, new Path("/betty"));
FileOutputFormat.setOutputPath(job, new Path("/betty/output"));
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
}