Hadoop Reducer尽管写入上下文但不写入任何内容

i2byvkas  于 2022-11-01  发布在  Hadoop
关注(0)|答案(1)|浏览(164)

我将导出的jar作为mapreduce作业hadoop运行,并且0字节被写入输出文件。
日志

2022-10-22 21:38:19,004 INFO mapreduce.Job:  map 100% reduce 100%
2022-10-22 21:38:19,012 INFO mapreduce.Job: Job job_1666492742770_0009 completed successfully
2022-10-22 21:38:19,159 INFO mapreduce.Job: Counters: 54
        File System Counters
                FILE: Number of bytes read=6
                FILE: Number of bytes written=1134025
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=446009085
                HDFS: Number of bytes written=0
                HDFS: Number of read operations=17
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=2
                HDFS: Number of bytes read erasure-coded=0
        Job Counters
                Launched map tasks=4
                Launched reduce tasks=1
                Rack-local map tasks=4
                Total time spent by all maps in occupied slots (ms)=38622
                Total time spent by all reduces in occupied slots (ms)=6317
                Total time spent by all map tasks (ms)=38622
                Total time spent by all reduce tasks (ms)=6317
                Total vcore-milliseconds taken by all map tasks=38622
                Total vcore-milliseconds taken by all reduce tasks=6317
                Total megabyte-milliseconds taken by all map tasks=39548928
                Total megabyte-milliseconds taken by all reduce tasks=6468608
        Map-Reduce Framework
                Map input records=3208607
                Map output records=0
                Map output bytes=0
                Map output materialized bytes=24
                Input split bytes=424
                Combine input records=0
                Combine output records=0
                Reduce input groups=0
                Reduce shuffle bytes=24
                Reduce input records=0
                Reduce output records=0
                Spilled Records=0
                Shuffled Maps =4
                Failed Shuffles=0
                Merged Map outputs=4
                GC time elapsed (ms)=505
                CPU time spent (ms)=9339
                Physical memory (bytes) snapshot=2058481664
                Virtual memory (bytes) snapshot=2935365632
                Total committed heap usage (bytes)=1875378176
                Peak Map Physical memory (bytes)=501469184
                Peak Map Virtual memory (bytes)=643743744
                Peak Reduce Physical memory (bytes)=206155776
                Peak Reduce Virtual memory (bytes)=384512000
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=0
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        File Input Format Counters
                Bytes Read=446008661
        File Output Format Counters
                Bytes Written=0

任何帮助,感激不尽!Map功能:

public void map(LongWritable arg0, Text Value, Context context) throws IOException, InterruptedException {
            String line = Value.toString();
            if(line.length() == 0 && !line.contains("MAX")) {
                String date = line.substring(14,21);
                float temp_Max;
                float temp_Min;

                try {
                    temp_Max = Float.parseFloat(line.substring(104,108).trim());
                }catch(NumberFormatException e) {
                    temp_Max = Float.parseFloat(line.substring(104,107).trim());

                }

                try {
                    temp_Min = Float.parseFloat(line.substring(112,117).trim());
                }catch(NumberFormatException e) {
                    temp_Min = Float.parseFloat(line.substring(112,116).trim());
                }

                if(temp_Max > 35.0) {
                    context.write(new Text("Hot Day" + date), new FloatWritable(temp_Max));
                }
                if(temp_Min < 10) {
                    context.write(new Text("Cold Day" + date), new FloatWritable(temp_Min));
                }

            }

        }

减速器功能:

public static class MaxMinTemperatureReducer extends Reducer<Text, Text, Text, FloatWritable>  {

        FloatWritable res = new FloatWritable();

            public void reduce(Text key, Iterable<IntWritable> values,
                    Context context
                       ) throws IOException, InterruptedException {
                float sum = 0;
                for (IntWritable val : values) {
                    sum += val.get();
                }
                res.set(sum);
                LogManager lgmngr = LogManager.getLogManager();

                // lgmngr now contains a reference to the log manager.
                Logger log = lgmngr.getLogger(Logger.GLOBAL_LOGGER_NAME);

                // Getting the global application level logger
                // from the Java Log Manager
                log.log(Level.INFO, "LOL_PLS_WORK",res.toString());
                context.write(key,res);
            }
    }

主要:

Configuration conf = new Configuration();
        Job job = Job.getInstance(conf,"weather example");

        job.setJarByClass(MyMaxMin.class);
        job.setMapOutputKeyClass(Text.class);
        job.setMapOutputValueClass(Text.class);
        job.setMapperClass(MaxMinTemperatureMapper.class);
        job.setReducerClass(MaxMinTemperatureReducer.class);
        job.setInputFormatClass(TextInputFormat.class);
        job.setOutputFormatClass(TextOutputFormat.class);
        Path OutputPath = new Path(args[1]);
        FileInputFormat.addInputPath(job, new Path(args[0]));
        FileOutputFormat.setOutputPath(job, new Path(args[1]));
        OutputPath.getFileSystem(conf).delete(OutputPath, true);
        System.exit(job.waitForCompletion(true) ? 0 : 1);
q8l4jmvw

q8l4jmvw1#

根据您的Map器代码:

public void map(LongWritable arg0, Text Value, Context context) throws IOException, InterruptedException {
    String line = Value.toString();
    if(line.length() == 0 && !line.contains("MAX")) {

line.length() == 0您正在放弃任何非空白的输入。您需要line.length() != 0

相关问题