我将导出的jar作为mapreduce作业hadoop运行,并且0字节被写入输出文件。
日志
2022-10-22 21:38:19,004 INFO mapreduce.Job: map 100% reduce 100%
2022-10-22 21:38:19,012 INFO mapreduce.Job: Job job_1666492742770_0009 completed successfully
2022-10-22 21:38:19,159 INFO mapreduce.Job: Counters: 54
File System Counters
FILE: Number of bytes read=6
FILE: Number of bytes written=1134025
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=446009085
HDFS: Number of bytes written=0
HDFS: Number of read operations=17
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
HDFS: Number of bytes read erasure-coded=0
Job Counters
Launched map tasks=4
Launched reduce tasks=1
Rack-local map tasks=4
Total time spent by all maps in occupied slots (ms)=38622
Total time spent by all reduces in occupied slots (ms)=6317
Total time spent by all map tasks (ms)=38622
Total time spent by all reduce tasks (ms)=6317
Total vcore-milliseconds taken by all map tasks=38622
Total vcore-milliseconds taken by all reduce tasks=6317
Total megabyte-milliseconds taken by all map tasks=39548928
Total megabyte-milliseconds taken by all reduce tasks=6468608
Map-Reduce Framework
Map input records=3208607
Map output records=0
Map output bytes=0
Map output materialized bytes=24
Input split bytes=424
Combine input records=0
Combine output records=0
Reduce input groups=0
Reduce shuffle bytes=24
Reduce input records=0
Reduce output records=0
Spilled Records=0
Shuffled Maps =4
Failed Shuffles=0
Merged Map outputs=4
GC time elapsed (ms)=505
CPU time spent (ms)=9339
Physical memory (bytes) snapshot=2058481664
Virtual memory (bytes) snapshot=2935365632
Total committed heap usage (bytes)=1875378176
Peak Map Physical memory (bytes)=501469184
Peak Map Virtual memory (bytes)=643743744
Peak Reduce Physical memory (bytes)=206155776
Peak Reduce Virtual memory (bytes)=384512000
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=446008661
File Output Format Counters
Bytes Written=0
任何帮助,感激不尽!Map功能:
public void map(LongWritable arg0, Text Value, Context context) throws IOException, InterruptedException {
String line = Value.toString();
if(line.length() == 0 && !line.contains("MAX")) {
String date = line.substring(14,21);
float temp_Max;
float temp_Min;
try {
temp_Max = Float.parseFloat(line.substring(104,108).trim());
}catch(NumberFormatException e) {
temp_Max = Float.parseFloat(line.substring(104,107).trim());
}
try {
temp_Min = Float.parseFloat(line.substring(112,117).trim());
}catch(NumberFormatException e) {
temp_Min = Float.parseFloat(line.substring(112,116).trim());
}
if(temp_Max > 35.0) {
context.write(new Text("Hot Day" + date), new FloatWritable(temp_Max));
}
if(temp_Min < 10) {
context.write(new Text("Cold Day" + date), new FloatWritable(temp_Min));
}
}
}
减速器功能:
public static class MaxMinTemperatureReducer extends Reducer<Text, Text, Text, FloatWritable> {
FloatWritable res = new FloatWritable();
public void reduce(Text key, Iterable<IntWritable> values,
Context context
) throws IOException, InterruptedException {
float sum = 0;
for (IntWritable val : values) {
sum += val.get();
}
res.set(sum);
LogManager lgmngr = LogManager.getLogManager();
// lgmngr now contains a reference to the log manager.
Logger log = lgmngr.getLogger(Logger.GLOBAL_LOGGER_NAME);
// Getting the global application level logger
// from the Java Log Manager
log.log(Level.INFO, "LOL_PLS_WORK",res.toString());
context.write(key,res);
}
}
主要:
Configuration conf = new Configuration();
Job job = Job.getInstance(conf,"weather example");
job.setJarByClass(MyMaxMin.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(Text.class);
job.setMapperClass(MaxMinTemperatureMapper.class);
job.setReducerClass(MaxMinTemperatureReducer.class);
job.setInputFormatClass(TextInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);
Path OutputPath = new Path(args[1]);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
OutputPath.getFileSystem(conf).delete(OutputPath, true);
System.exit(job.waitForCompletion(true) ? 0 : 1);
1条答案
按热度按时间q8l4jmvw1#
根据您的Map器代码:
line.length() == 0
您正在放弃任何非空白的输入。您需要line.length() != 0
。