无法在hadoop群集上启动h2o-classnotfound异常

h5qlskok  于 2021-06-02  发布在  Hadoop
关注(0)|答案(2)|浏览(353)

我想在hadoop集群上启动h2o。遗憾的是，它不工作，并给我一个错误，类water.hadoop.h2omapper找不到。
hadoop环境是版本2.6中的hdp，包括5个节点，其中1个运行yarn资源管理器，3个节点是yarn客户机的数据节点。每个数据节点都有32gbram和4个cpu核的资源。没有其他应用程序在其上运行。我在ambari的每个节点上配置了最多16gb和3个芯线应用程序。
我从终端启动h2o集群（在所有节点上都尝试过，到处都是相同的错误），输出如下：

[root@host3 h2o-3.14.0.6-hdp2.6]# sudo -u hdfs hadoop jar h2odriver.jar -nodes 3 -mapperXmx 6g -output h2o-test
Determining driver host interface for mapper->driver callback...
[Possible callback IP address: 192.168.20.35]
[Possible callback IP address: 127.0.0.1]
Using mapper->driver callback IP address and port: 192.168.20.35:46619
(You can override these with -driverif and -driverport/-driverportrange.)
Memory Settings:
mapreduce.map.java.opts:     -Xms6g -Xmx6g -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Dlog4j.defaultInitOverride=true
Extra memory percent:        10
mapreduce.map.memory.mb:     6758
17/10/13 07:49:14 INFO client.RMProxy: Connecting to ResourceManager at host2/192.168.20.34:8050
17/10/13 07:49:14 INFO client.AHSProxy: Connecting to Application History server at host2/192.168.20.34:10200
17/10/13 07:49:15 WARN mapreduce.JobResourceUploader: No job jar file set.  User classes may not be found. See Job or Job#setJar(String).

17/10/13 07:49:15 INFO mapreduce.JobSubmitter: number of splits:3
17/10/13 07:49:15 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1507793796947_0002
17/10/13 07:49:15 INFO mapred.YARNRunner: Job jar is not present. Not adding any jar to the list of resources.
17/10/13 07:49:15 INFO impl.YarnClientImpl: Submitted application application_1507793796947_0002
17/10/13 07:49:15 INFO mapreduce.Job: The url to track the job: http://host2:8088/proxy/application_1507793796947_0002/
Job name 'H2O_86929' submitted
JobTracker job ID is 'job_1507793796947_0002'
For YARN users, logs command is 'yarn logs -applicationId application_1507793796947_0002'
Waiting for H2O cluster to come up...
17/10/13 07:49:29 INFO client.RMProxy: Connecting to ResourceManager at host2/192.168.20.34:8050
17/10/13 07:49:29 INFO client.AHSProxy: Connecting to Application History server at host2/192.168.20.34:10200

----- YARN cluster metrics -----
Number of YARN worker nodes: 3

----- Nodes -----
Node: http://host5:8042 Rack: /default-rack, RUNNING, 1 containers used, 4,0 / 16,0 GB used, 1 / 3 vcores used
Node: http://host4:8042 Rack: /default-rack, RUNNING, 0 containers used, 0,0 / 16,0 GB used, 0 / 3 vcores used
Node: http://host3:8042 Rack: /default-rack, RUNNING, 0 containers used, 0,0 / 16,0 GB used, 0 / 3 vcores used

----- Queues -----
Queue name:            default
Queue state:       RUNNING
Current capacity:  0,11
Capacity:          1,00
Maximum capacity:  1,00
Application count: 1
----- Applications in this queue -----
Application ID:                  application_1507793796947_0002 (H2O_86929)
    Started:                     hdfs (Fri Oct 13 07:49:15 CEST 2017)
    Application state:           FINISHED
    Tracking URL:                http://host2:8088/proxy/application_1507793796947_0002/
    Queue name:                  default
    Used/Reserved containers:    1 / 0
    Needed/Used/Reserved memory: 4,0 GB / 4,0 GB / 0,0 GB
    Needed/Used/Reserved vcores: 1 / 1 / 0

Queue 'default' approximate utilization: 4,0 / 48,0 GB used, 1 / 9 vcores used

----------------------------------------------------------------------

ERROR: Unable to start any H2O nodes; please contact your YARN administrator.

   A common cause for this is the requested container size (6,6 GB)
   exceeds the following YARN settings:

       yarn.nodemanager.resource.memory-mb
       yarn.scheduler.maximum-allocation-mb

Yarn应用程序的系统日志中的相应错误条目：

2017-10-13 07:49:24,505 FATAL [IPC Server handler 1 on 40503] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1507793796947_0002_m_000002_0 - exited : java.lang.RuntimeException: java.lang.ClassNotFoundException: Class water.hadoop.h2omapper not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2241)
at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:186)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:745)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
Caused by: java.lang.ClassNotFoundException: Class water.hadoop.h2omapper not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2147)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2239)
... 8 more

2017-10-13 07:49:24,506 INFO [IPC Server handler 1 on 40503] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report from attempt_1507793796947_0002_m_000002_0: Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class water.hadoop.h2omapper not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2241)
at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:186)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:745)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
Caused by: java.lang.ClassNotFoundException: Class water.hadoop.h2omapper not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2147)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2239)
... 8 more

2017-10-13 07:49:24,507 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1507793796947_0002_m_000002_0: Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class water.hadoop.h2omapper not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2241)
at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:186)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:745)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
Caused by: java.lang.ClassNotFoundException: Class water.hadoop.h2omapper not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2147)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2239)
... 8 more

这里有完整的日志。
任何帮助都将不胜感激。
顺祝你，马库斯

hadoopyarnH2O

2条答案

按热度按时间
q35jwt9p

q35jwt9p1#

你能做其他的工作吗(就像pi的例子。）
应用程序id显示您的hadoop群集于2017年10月12日星期四07:36:36 utc（即昨天）启动，这是群集尝试运行的第一个（也是第二个）作业。
而且，集群中节点的大小非常小。
在我看来，所有这些都像是你试图成为自己的hadoop管理员，但还没有成功
继续尝试，当集群配置正确时，h2o将运行。

赞(0）分享  回复(0）举报  2021-06-02
gojuced7

gojuced72#

根据您提供的部分日志，下面一行有助于理解一些内容：

2017-10-12 07:45:02,172 FATAL [IPC Server handler 1 on 39365] 
 org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: 
  attempt_1507726330188_0001_m_000002_0 - exited : 
    java.lang.RuntimeException: java.lang.ClassNotFoundException: Class water.hadoop.h2omapper not found

很明显，任务id#2示例#0（或第一个示例）由于缺少类而失败。因此，如果任务2显示错误，则表示另一个任务1已经在运行。这也意味着作业已启动或处于运行状态。所以问题发生在作业执行阶段。这意味着执行此特定任务的datanode无法找到h2o驱动程序库。因此，可能由于某种原因节点或节点上的文件系统不可用。如果你研究你的详细日志，你将能够看到为什么会发生这种情况。
[其他详细信息]
由于权限或文件系统问题，任何Map程序都无法访问h2odriver.jar，这就是未启动作业的原因。您应该确保正确地修复hadoop权限和可访问性，这样您就可以在没有root的情况下启动任何“hadoop”命令，并且需要超级用户别名。

赞(0）分享  回复(0）举报  2021-06-02
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备2022004421号-2 NULL123 https://www.null123.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com