使用mapreduce複製hbase表
阿新 • • 發佈:2018-12-16
準備工作如下:
一.兩張hbase表——fruit(有資料)和fruitresult(無資料),我們將使用mapreduce將hbase中的frult以可自定義的方式複製到fruitresult中
二.建立一個hbase工程,匯入對應jar包
程式碼如下:
map端:
public class FruitMapper extends TableMapper<ImmutableBytesWritable, Put> { @Override protected void map( ImmutableBytesWritable key, Result value, Mapper<ImmutableBytesWritable, Result, ImmutableBytesWritable, Put>.Context context) throws IOException, InterruptedException { Put put = new Put(key.get()); Cell[] cells = value.rawCells(); for (Cell cell : cells) { if ("name".equals(Bytes.toString(CellUtil.cloneQualifier(cell)))) { put.add(cell); } else if ("color".equals(Bytes.toString(CellUtil .cloneQualifier(cell)))) { put.add(cell); } } context.write(key, put); } }
reduce端如下:
public class FruitReducer extends TableReducer<ImmutableBytesWritable, Put, NullWritable> { @Override protected void reduce( ImmutableBytesWritable key, Iterable<Put> values, Reducer<ImmutableBytesWritable, Put, NullWritable, Mutation>.Context context) throws IOException, InterruptedException { // TODO Auto-generated method stub for (Put value : values) { context.write(NullWritable.get(), value); } } }
drive端:
public class FruitDriver extends Configuration implements Tool { Configuration configuration = null; public static void main(String[] args) throws Exception { Configuration conf = HBaseConfiguration.create(); int tool = ToolRunner.run(conf, new FruitDriver(), args); } @Override public void setConf(Configuration conf) { // TODO Auto-generated method stub this.configuration = conf; } @Override public Configuration getConf() { // TODO Auto-generated method stub return null; } @Override public int run(String[] args) throws Exception { Job job = Job.getInstance(configuration); // job.setMapperClass(FruitMapper.class); // job.setReducerClass(FruitReducer.class); job.setJarByClass(FruitDriver.class); Scan scan = new Scan(); TableMapReduceUtil.initTableMapperJob("fruit", scan, FruitMapper.class, ImmutableBytesWritable.class, Put.class, job); TableMapReduceUtil.initTableReducerJob("fruitresult", FruitReducer.class, job); System.out.println(); return job.waitForCompletion(true) ? 0 : 1; } }
執行結果如下:
除了hbase表之間的複製,還可以使用mapreduce從hdfs本地複製內容到hbase表中,參考:使用MapReduce將Hadoop HDFS中的檔案匯入HBase