在emr上运行hadoop作业时出现错误“starting mrappmaster”

ioekq8ef  于 2021-06-02  发布在  Hadoop
关注(0)|答案(0)|浏览(264)

当我试图在emr集群上运行hadoop作业时,我得到了下面提到的错误,它说由于“am container”而无法启动,在查看容器日志时,我可以看到它在启动“mrappmaster”时失败。在本地开发人员桌面上,我能够成功地运行该作业。
我在网上搜索了一下,发现了这个问题(hadoop2.6.0:安装后的基本错误“starting mrappmaster”),它说这个问题可能是因为hadoop的多个版本“hadoop.so”。根据堆栈中提到的建议,over flow可以获取“java.library.path”和我的结果::/usr/lib/hadoop lzo/lib/native:/usr/lib/hadoop/lib/native path:/usr/lib/hadoop lzo/lib/native,不包含任何hadoop.so文件。路径:/usr/lib/hadoop/lib/native,不包含hadoop.so和hadoop.so.1.0。
我试图删除一个从稍后运行我的工作,但它继续失败,并与相同的错误。
错误消息:
我的emr工作的标准日志:
2015-11-26 11:18:17074 info org.apache.hadoop.mapreduce.job(main):作业作业\u 1448532525181 \u 0004失败,状态失败,原因是:应用程序应用程序\u 1448532525181 \u 0004由于appattempt的am容器失败2次\u 1448532525181 \u 0004 \u000002退出,退出代码:1诊断:容器启动异常。集装箱id:集装箱\U 1448532525181 \U 0004001出口代码:1

Stack trace: ExitCodeException exitCode=1: 
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

容器退出,退出代码为非零,尝试失败。应用程序失败。2015-11-26 11:18:17095 info org.apache.hadoop.mapreduce.job(主):计数器:0
容器日志:

2015-11-26 10:11:40,020 INFO [main] com.amazonaws.latency: ClientExecuteTime=[44.047], 2015-11-26 10:11:40,021 FATAL [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster java.lang.NoSuchMethodError: com.amazonaws.AmazonWebServiceRequest.getDelegationToken()Ljava/lang/String;
at com.amazonaws.handlers.internal.S3SecurityTokenRequestHandler.beforeRequest(S3SecurityTokenRequestHandler.java:33)
at com.amazonaws.handlers.RequestHandler2Adaptor.beforeRequest(RequestHandler2Adaptor.java:37)
at com.amazonaws.http.AmazonHttpClient.requestHandler2s(AmazonHttpClient.java:434)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:292)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3595)
at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1041)
at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:1013)
at com.amazon.ws.emr.hadoop.fs.s3n.Jets3tNativeFileSystemStore.ensureBucketExists(Jets3tNativeFileSystemStore.java:105)
at com.amazon.ws.emr.hadoop.fs.s3n.Jets3tNativeFileSystemStore.initialize(Jets3tNativeFileSystemStore.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy27.initialize(Unknown Source)
at com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem.initialize(S3NativeFileSystem.java:466)
at com.amazon.ws.emr.hadoop.fs.EmrFileSystem.initialize(EmrFileSystem.java:116)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2644)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:90)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2678)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2660)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:374)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.<init>(FileOutputCommitter.java:91)
at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.<init>(FileOutputCommitter.java:74)
at org.apache.hadoop.mapreduce.lib.output.DirectFileOutputCommitter.<init>(DirectFileOutputCommitter.java:31)
at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.getOutputCommitter(FileOutputFormat.java:310)
at org.apache.hadoop.mapreduce.lib.output.FilterOutputFormat.getOutputCommitter(FilterOutputFormat.java:67)
at org.apache.hadoop.mapreduce.lib.output.LazyOutputFormat.getOutputCommitter(LazyOutputFormat.java:92)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.call(MRAppMaster.java:471)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.call(MRAppMaster.java:453)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.callWithJobClassLoader(MRAppMaster.java:1542)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:453)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:371)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1500)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1497)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1430) 2015-11-26 10:11:40,026 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting with status 1

在这方面的任何帮助都将是很大的帮助,请让我知道如果你需要任何更多的信息从我这边。
谢谢,阿努沙。好的

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题