我想在mapreduce中将序列文件转换为orc文件。键/值的输入类型为text/text。
我的程序看起来像
public class ANR extends Configured implements Tool{
public static void main(String[] args) throws Exception {
// TODO Auto-generated method stub
int res = ToolRunner.run(new Configuration(),new ANR(), args);
System.exit(res);
}
public int run(String[] args) throws Exception {
Logger log = Logger.getLogger(ANRmap.class.getName());
Configuration conf = getConf();
Job job;
String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();
conf.set("orc.create.index", "true");
job = Job.getInstance(conf);
/////
job.setJobName("ORC Output");
job.setJarByClass(ANR.class);
job.setInputFormatClass(SequenceFileInputFormat.class);
SequenceFileInputFormat.addInputPath(job, new Path(args[0]));
job.setMapperClass(ANRmap.class);
job.setNumReduceTasks(0);
job.setOutputFormatClass(OrcNewOutputFormat.class);
OrcNewOutputFormat.setCompressOutput(job,true);
OrcNewOutputFormat.setOutputPath(job,new Path(args[1]));
return job.waitForCompletion(true) ? 0: 1;
}
制图器
public class ANRmap extends Mapper<Text,Text,NullWritable,Writable> {
private final OrcSerde serde = new OrcSerde();
public void map(Text key, Text value,
OutputCollector<NullWritable, Writable> output)
throws IOException {
output.collect(NullWritable.get(),serde.serialize(value, null));
}
}
这是个例外
Error: java.lang.ClassCastException: org.apache.hadoop.io.Text cannot be cast to org.apache.hadoop.io.NullWritable
at org.apache.hadoop.hive.ql.io.orc.OrcNewOutputFormat$OrcRecordWriter.write(OrcNewOutputFormat.java:37)
at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:635)
at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
at org.apache.hadoop.mapreduce.Mapper.map(Mapper.java:124)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
orcnewoutputformat中的输出键为空可写。如何将文本转换为nullwritable或以其他方式修复此异常?
1条答案
按热度按时间h22fl7wq1#
尝试使用
Context
而不是OutputCollector
.