mapreduce读取配置单元表并使用上下文写入hdfs位置

tct7dpnv  于 2021-06-01  发布在  Hadoop
关注(0)|答案(0)|浏览(130)

我期待mapreduce程序从一个配置单元表中读取并将每个记录的第一列值写入hdfs位置。它应该只包含Map相位,而不是还原相位。
下面是Map

public class Map extends Mapper<WritableComparable, HCatRecord, NullWritable, IntWritable> {

      protected void map( WritableComparable key,
                        HCatRecord value,
                        org.apache.hadoop.mapreduce.Mapper<WritableComparable, HCatRecord,
                        NullWritable, IntWritable>.Context context)
          throws IOException, InterruptedException {
          // The group table from /etc/group has name, 'x', id
        //  groupname = (String) value.get(0);
          int id = (Integer) value.get(1);
          // Just select and emit the name and ID
          context.write(null, new IntWritable(id));
      }
  }

主要类别

public class mapper1   {

    public static void main(String[] args) throws Exception {
    mapper1 m=new mapper1();
        m.run(args);
    }

    public void run(String[] args) throws IOException, Exception, InterruptedException {
    Configuration conf =new  Configuration();

    // Get the input and output table names as arguments
    String inputTableName = args[0];

    // Assume the default database
    String dbName = "xademo";

    Job job = new Job(conf, "UseHCat");
    job.setJarByClass(mapper1.class);
    HCatInputFormat.setInput(job, dbName, inputTableName);

    job.setMapperClass(Map.class);

    // An HCatalog record as input
    job.setInputFormatClass(HCatInputFormat.class);

    // Mapper emits a string as key and an integer as value
    job.setMapOutputKeyClass(NullWritable.class);
    job.setMapOutputValueClass(IntWritable.class);

    FileOutputFormat.setOutputPath((JobConf) conf, new Path(args[1]));

    job.waitForCompletion(true);
    }
}

这个代码有什么问题吗?
这是由于字符串5s中的numberformat异常而产生的错误。我不知道它是从哪里来的。在hcatinputformat.setinput()下一行显示错误

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题