1. 程式人生 > >使用mapreduce複製hbase表

使用mapreduce複製hbase表

準備工作如下:

一.兩張hbase表——fruit(有資料)和fruitresult(無資料),我們將使用mapreduce將hbase中的frult以可自定義的方式複製到fruitresult中

二.建立一個hbase工程,匯入對應jar包 

程式碼如下:

map端:

public class FruitMapper extends TableMapper<ImmutableBytesWritable, Put> {
	@Override
	protected void map(
			ImmutableBytesWritable key,
			Result value,
			Mapper<ImmutableBytesWritable, Result, ImmutableBytesWritable, Put>.Context context)
			throws IOException, InterruptedException {
		Put put = new Put(key.get());
		Cell[] cells = value.rawCells();
		for (Cell cell : cells) {
			if ("name".equals(Bytes.toString(CellUtil.cloneQualifier(cell)))) {
				put.add(cell);
			} else if ("color".equals(Bytes.toString(CellUtil
					.cloneQualifier(cell)))) {
				put.add(cell);
			}
		}
		context.write(key, put);
	}
}

reduce端如下:

public class FruitReducer extends
		TableReducer<ImmutableBytesWritable, Put, NullWritable> {
	@Override
	protected void reduce(
			ImmutableBytesWritable key,
			Iterable<Put> values,
			Reducer<ImmutableBytesWritable, Put, NullWritable, Mutation>.Context context)
			throws IOException, InterruptedException {
		// TODO Auto-generated method stub
		for (Put value : values) {
			context.write(NullWritable.get(), value);
		}
	}
}

 drive端:

public class FruitDriver extends Configuration implements Tool {
	Configuration configuration = null;

	public static void main(String[] args) throws Exception {
		Configuration conf = HBaseConfiguration.create();
		int tool = ToolRunner.run(conf, new FruitDriver(), args);
	}

	@Override
	public void setConf(Configuration conf) {
		// TODO Auto-generated method stub
		this.configuration = conf;
	}

	@Override
	public Configuration getConf() {
		// TODO Auto-generated method stub
		return null;
	}

	@Override
	public int run(String[] args) throws Exception {
		Job job = Job.getInstance(configuration);
		// job.setMapperClass(FruitMapper.class);
		// job.setReducerClass(FruitReducer.class);
		job.setJarByClass(FruitDriver.class);
		Scan scan = new Scan();
		TableMapReduceUtil.initTableMapperJob("fruit", scan, FruitMapper.class,
				ImmutableBytesWritable.class, Put.class, job);
		TableMapReduceUtil.initTableReducerJob("fruitresult",
				FruitReducer.class, job);
		System.out.println();
		return job.waitForCompletion(true) ? 0 : 1;
	}
}

執行結果如下: 

除了hbase表之間的複製,還可以使用mapreduce從hdfs本地複製內容到hbase表中,參考:使用MapReduce將Hadoop HDFS中的檔案匯入HBase