Map器执行的确定

vngu2lb8  于 2021-06-03  发布在  Hadoop
关注(0)|答案(1)|浏览(317)

是否执行了定义Map器的帮助，如果没有执行，原因是什么。我编写了一个从数据库到本地文件系统的文本文件的读取方式的输出，在该文件系统上执行mapper。我给你一个密码

package org.myorg;

import java.io.*;
import java.util.*;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.logging.Level;
import org.apache.hadoop.fs.*;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.*;
import org.apache.hadoop.util.*;

public class ParallelIndexation {

    public static class Map extends MapReduceBase implements
            Mapper<LongWritable, Text, Text, LongWritable> {
        private final static LongWritable zero = new LongWritable(0);
        private Text word = new Text();

        public void map(LongWritable key, Text value,
                OutputCollector<Text, LongWritable> output, Reporter reporter)
                throws IOException {

            Configuration conf = new Configuration();
            int CountComputers;
            FileInputStream fstream = new FileInputStream(
                    "/export/hadoop-1.0.1/bin/countcomputers.txt");
            BufferedReader br = new BufferedReader(new InputStreamReader(fstream));
            String result=br.readLine();
            CountComputers=Integer.parseInt(result);
            input.close();
            fstream.close();
            Connection con = null;
            Statement st = null;
                ResultSet rs = null;    
                String url = "jdbc:postgresql://192.168.1.8:5432/NexentaSearch";
                String user = "postgres";
                String password = "valter89";
            ArrayList<String> paths = new ArrayList<String>();
            try
            {
                con = DriverManager.getConnection(url, user, password);
                        st = con.createStatement();
                        rs = st.executeQuery("select path from tasks order by id");
                while (rs.next()) { paths.add(rs.getString(1)); };
                PrintWriter zzz = null;
                    try
                    {
                            zzz = new PrintWriter(new FileOutputStream("/export/hadoop-1.0.1/bin/readwaysfromdatabase.txt"));
                    }
                    catch(FileNotFoundException e)
                    {
                            System.out.println("Error");
                            System.exit(0);
                    }
                    for (int i=0; i<paths.size(); i++)
                {
                    zzz.println("paths[i]=" + paths.get(i) + "\n");
                    }
                    zzz.close();
            }
            catch (SQLException e) 
            {
                System.out.println("Connection Failed! Check output console");
                e.printStackTrace();
            }

但是，尽管它位于一个从属节点上，/export/hadoop-1.0.1/bin/readwaysfromdatabase.txt文件并没有被创建。是否从这里开始，什么Map器根本没有被执行？我还将一个输出带到程序的执行文件中

args[0]=/export/hadoop-1.0.1/bin/input
13/04/22 14:00:53 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
13/04/22 14:00:53 INFO mapred.FileInputFormat: Total input paths to process : 0
13/04/22 14:00:54 INFO mapred.JobClient: Running job: job_201304221331_0003
13/04/22 14:00:55 INFO mapred.JobClient:  map 0% reduce 0%
13/04/22 14:01:12 INFO mapred.JobClient:  map 0% reduce 100%
13/04/22 14:01:17 INFO mapred.JobClient: Job complete: job_201304221331_0003
13/04/22 14:01:17 INFO mapred.JobClient: Counters: 15
13/04/22 14:01:17 INFO mapred.JobClient:   Job Counters 
13/04/22 14:01:17 INFO mapred.JobClient:     Launched reduce tasks=1
13/04/22 14:01:17 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=9079
13/04/22 14:01:17 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
13/04/22 14:01:17 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
13/04/22 14:01:17 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=7983
13/04/22 14:01:17 INFO mapred.JobClient:   File Output Format Counters 
13/04/22 14:01:17 INFO mapred.JobClient:     Bytes Written=0
13/04/22 14:01:17 INFO mapred.JobClient:   FileSystemCounters
13/04/22 14:01:17 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=21536
13/04/22 14:01:17 INFO mapred.JobClient:   Map-Reduce Framework
13/04/22 14:01:17 INFO mapred.JobClient:     Reduce input groups=0
13/04/22 14:01:17 INFO mapred.JobClient:     Combine output records=0
13/04/22 14:01:17 INFO mapred.JobClient:     Reduce shuffle bytes=0
13/04/22 14:01:17 INFO mapred.JobClient:     Reduce output records=0
13/04/22 14:01:17 INFO mapred.JobClient:     Spilled Records=0
13/04/22 14:01:17 INFO mapred.JobClient:     Total committed heap usage (bytes)=16252928
13/04/22 14:01:17 INFO mapred.JobClient:     Combine input records=0
13/04/22 14:01:17 INFO mapred.JobClient:     Reduce input records=0

我还将在一个虚拟机上成功执行程序的输出带到文件中

12/10/28 10:41:14 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
12/10/28 10:41:14 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
12/10/28 10:41:14 INFO mapred.FileInputFormat: Total input paths to process : 1
12/10/28 10:41:15 INFO mapred.JobClient: Running job: job_local_0001
12/10/28 10:41:15 INFO mapred.Task:  Using ResourceCalculatorPlugin : null
12/10/28 10:41:15 INFO mapred.MapTask: numReduceTasks: 1
12/10/28 10:41:15 INFO mapred.MapTask: io.sort.mb = 100
12/10/28 10:41:15 INFO mapred.MapTask: data buffer = 79691776/99614720
12/10/28 10:41:15 INFO mapred.MapTask: record buffer = 262144/327680
12/10/28 10:41:15 INFO mapred.MapTask: Starting flush of map output
12/10/28 10:41:16 INFO mapred.JobClient:  map 0% reduce 0%
12/10/28 10:41:17 INFO mapred.MapTask: Finished spill 0
12/10/28 10:41:17 INFO mapred.Task: Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting
12/10/28 10:41:18 INFO mapred.LocalJobRunner: file:/export/hadoop-1.0.1/bin/input/paths.txt:0+156
12/10/28 10:41:18 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0' done.
12/10/28 10:41:18 INFO mapred.Task:  Using ResourceCalculatorPlugin : null
12/10/28 10:41:18 INFO mapred.LocalJobRunner: 
12/10/28 10:41:18 INFO mapred.Merger: Merging 1 sorted segments
12/10/28 10:41:18 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 199 bytes
12/10/28 10:41:18 INFO mapred.LocalJobRunner: 
12/10/28 10:41:19 INFO mapred.JobClient:  map 100% reduce 0%
12/10/28 10:41:19 INFO mapred.Task: Task:attempt_local_0001_r_000000_0 is done. And is in the process of commiting
12/10/28 10:41:19 INFO mapred.LocalJobRunner: 
12/10/28 10:41:19 INFO mapred.Task: Task attempt_local_0001_r_000000_0 is allowed to commit now
12/10/28 10:41:19 INFO mapred.FileOutputCommitter: Saved output of task 'attempt_local_0001_r_000000_0' to file:/export/hadoop-1.0.1/bin/output
12/10/28 10:41:21 INFO mapred.LocalJobRunner: reduce > reduce
12/10/28 10:41:21 INFO mapred.Task: Task 'attempt_local_0001_r_000000_0' done.
12/10/28 10:41:22 INFO mapred.JobClient:  map 100% reduce 100%
12/10/28 10:41:22 INFO mapred.JobClient: Job complete: job_local_0001
12/10/28 10:41:22 INFO mapred.JobClient: Counters: 18
12/10/28 10:41:22 INFO mapred.JobClient:   File Input Format Counters 
12/10/28 10:41:22 INFO mapred.JobClient:     Bytes Read=156
12/10/28 10:41:22 INFO mapred.JobClient:   File Output Format Counters 
12/10/28 10:41:22 INFO mapred.JobClient:     Bytes Written=177
12/10/28 10:41:22 INFO mapred.JobClient:   FileSystemCounters
12/10/28 10:41:22 INFO mapred.JobClient:     FILE_BYTES_READ=9573
12/10/28 10:41:22 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=73931
12/10/28 10:41:22 INFO mapred.JobClient:   Map-Reduce Framework
12/10/28 10:41:22 INFO mapred.JobClient:     Reduce input groups=4
12/10/28 10:41:22 INFO mapred.JobClient:     Map output materialized bytes=203
12/10/28 10:41:22 INFO mapred.JobClient:     Combine output records=4
12/10/28 10:41:22 INFO mapred.JobClient:     Map input records=1
12/10/28 10:41:22 INFO mapred.JobClient:     Reduce shuffle bytes=0
12/10/28 10:41:22 INFO mapred.JobClient:     Reduce output records=4
12/10/28 10:41:22 INFO mapred.JobClient:     Spilled Records=8
12/10/28 10:41:22 INFO mapred.JobClient:     Map output bytes=189
12/10/28 10:41:22 INFO mapred.JobClient:     Total committed heap usage (bytes)=321527808
12/10/28 10:41:22 INFO mapred.JobClient:     Map input bytes=156
12/10/28 10:41:22 INFO mapred.JobClient:     Combine input records=0
12/10/28 10:41:22 INFO mapred.JobClient:     Map output records=4
12/10/28 10:41:22 INFO mapred.JobClient:     SPLIT_RAW_BYTES=98
12/10/28 10:41:22 INFO mapred.JobClient:     Reduce input records=0

@我在命令的帮助下运行程序

./hadoop jar /export/hadoop-1.0.1/bin/ParallelIndexation.jar org.myorg.ParallelIndexation /export/hadoop-1.0.1/bin/input /export/hadoop-1.0.1/bin/output -D mapred.map.tasks=1 1> resultofexecute.txt 2&>1

我在一个集群中有4个节点，其中一个主体，一个为secondarynamenode，还有2个下属。

JavahadooplinuxmapreduceMapper

1条答案

按热度按时间
8zzbczxx

8zzbczxx1#

为您的作业计划了多少Map任务，您的集群有多大？如果假设您的作业只运行了4个Map任务和一个包含32个节点的集群，那么28/32节点可能没有任何输出（因为这些节点上没有运行任何Map任务）。
您可以通过job tracker web ui查看有关组成作业的Map任务数以及这些作业计划在何处运行的信息。
奇怪的是，您的首次运行转储没有显示任何已启动的Map作业，只是减少了任务：

13/04/22 14:01:17 INFO mapred.JobClient:     Launched reduce tasks=1

而且Map输入/输出记录也没有计数器，因此运行此作业的方式有点奇怪-能否共享用于启动作业的完整命令行以及配置和运行作业的驱动程序代码？

赞(0）分享  回复(0）举报  2021-06-03
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备2022004421号-2 NULL123 https://www.null123.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com