我一直在努力寻找所有Map者使用map\u input\u records变量解析的总记录数。以下是我使用的代码:
Counters counters = job.getCounters();
for (CounterGroup group : counters) {
System.out.println("* Counter Group: " + group.getDisplayName() + " (" + group.getName() + ")");
System.out.println(" number of counters in this group: " + group.size());
for (Counter counter : group) {
System.out.println(" - " + counter.getDisplayName() + ": " + counter.getName() + ": "+counter.getValue());
}
}
此外,我还尝试使用以下代码片段:{
Counters counters = job.getCounters();
int recordCountData = (int)
//counters.getGroup("org.apache.hadoop.mapred.Task$Counter").findCounter("MAP_INPUT_RECORDS").getValue();
int recordCountData = (int) counters.findCounter(
"org.apache.hadoop.mapred.Task$Counter","MAP_INPUT_RECORDS")
.getValue();
}
但每次它都会抛出以下错误:
Exception in thread "main" java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.CounterGroup, but class was expected
at com.ssga.common.riskmeasures.validation.mr.RiskMeasuresValidationDriver.run(RiskMeasuresValidationDriver.java:169)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at com.ssga.common.riskmeasures.validation.mr.RiskMeasuresValidationDriver.main(RiskMeasuresValidationDriver.java:189)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
附言:我正在试着使用上述方法后,job.waitforcompletion(真)在司机类。有什么办法可以解决这个问题吗?
提前谢谢。阿基列什
1条答案
按热度按时间yyyllmsg1#
新的api支持抽象类而不是接口,因为抽象类更容易进化。例如,您可以向抽象类添加一个方法(带有一个默认实现),而无需中断class2的旧实现。例如,旧api中的mapper和reducer接口是新api中的抽象类。”
“新的api在org.apache.hadoop.mapreduce包(和子包)中。旧的api仍然可以在org.apache.hadoop.mapred中找到。”
-hadoop由tom white编写的权威指南,第3版,第28页
检查Map器和还原器,将它们用作类。