hadoophprof评测没有编写cpu示例

sirbozc5  于 2021-06-02  发布在  Hadoop
关注(0)|答案(2)|浏览(275)

我想用hprof来分析我的hadoop工作。问题是我 TRACES 但是没有 CPU SAMPLESprofile.out 文件。我在run方法中使用的代码是:

/**Get configuration */
    Configuration conf = getConf();
    conf.set("textinputformat.record.delimiter","\n\n");
    conf.setStrings("args", args);

    /**JVM PROFILING */
    conf.setBoolean("mapreduce.task.profile", true);
    conf.set("mapreduce.task.profile.params", "-agentlib:hprof=cpu=samples," +
       "heap=sites,depth=6,force=n,thread=y,verbose=n,file=%s");
    conf.set("mapreduce.task.profile.maps", "0-2");
    conf.set("mapreduce.task.profile.reduces", "");

    /**Job configuration */
    Job job = new Job(conf, "HadoopSearch");
    job.setJarByClass(Search.class);
    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(NullWritable.class);

    /**Set Mapper and Reducer, use identity reducer*/
    job.setMapperClass(Map.class);
    job.setReducerClass(Reducer.class);

    /**Set input and output formats */
    job.setInputFormatClass(TextInputFormat.class);
    job.setOutputFormatClass(TextOutputFormat.class);

    /**Set input and output path */
    FileInputFormat.addInputPath(job, new Path("/user/niko/16M"));  
    FileOutputFormat.setOutputPath(job, new Path(cmd.getOptionValue("output")));

    job.waitForCompletion(true);

    return 0;

我怎么才能拿到钱 CPU SAMPLES 要写入输出吗?
我也有一个trange错误消息在 stderr 但我认为这是不相关的,因为当分析设置为false或启用分析的代码被注解掉时,它也存在。错误是

log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.impl.MetricsSystemImpl).
 log4j:WARN Please initialize the log4j system properly.
 log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
64jmpszr

64jmpszr1#

Yarn(或mrv1)是杀死容器后,你的工作刚刚结束。cpu示例无法写入分析文件。事实上,您的跟踪也应该被截断。
必须添加以下选项(或hadoop版本上的等效选项):

yarn.nodemanager.sleep-delay-before-sigkill.ms = 30000

# No. of ms to wait between sending a SIGTERM and SIGKILL to a container

yarn.nodemanager.process-kill-wait.ms = 30000

# Max time to wait for a process to come up when trying to cleanup a container

mapreduce.tasktracker.tasks.sleeptimebeforesigkill = 30000

# Same en MRv1 ?

(30秒似乎足够了)

oipij1gg

oipij1gg2#

这可能是由于https://issues.apache.org/jira/browse/mapreduce-5465,在较新的hadoop版本中已修复。
因此,解决方案似乎是:
使用alsimon回答中提到的设置,或
升级至hadoop>=2.8.0

相关问题