执行oozie作业-Yarn问题

fhg3lkii  于 2021-06-02  发布在  Hadoop
关注(0)|答案(1)|浏览(328)

我正在尝试执行bash脚本,它包含使用oozie的多个配置单元命令。我遇到了几个例外:
权限问题-模拟oozie作业-权限问题-通过禁用安全性或更改权限来解决
文件未找到异常(org.apache.hadoop.yarn.exceptions.yarnruntimeexception:java.io.filenotfoundexception:文件不存在:hdfs://:8020/user/hdfs/.staging//job.splitmetainfo)完整堆栈跟踪显示如下。
两个事实:
我们使用的是HortwnWorks 2.0堆栈2.2.6
我很确定问题出在ozzie作业/配置的某个地方(脚本在命令行中运行良好)。
我错过了什么?

org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.FileNotFoundException: File does not exist: hdfs://<my_hadoop_server>:8020/user/hdfs/.staging/<job_id>/job.splitmetainfo
    at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.createSplits(JobImpl.java:1568)
    at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:1432)
    at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:1390)
    at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
    at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
    at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
    at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
    at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:996)
    at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:138)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1289)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1057)
    at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1500)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1496)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1429)
Caused by: java.io.FileNotFoundException: File does not exist: hdfs://<my_hadoop_server>:8020/user/hdfs/.staging/<job_id>/job.splitmetainfo
    at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1140)
    at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1132)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1132)
    at org.apache.hadoop.mapreduce.split.SplitMetaInfoReader.readSplitMetaInfo(SplitMetaInfoReader.java:51)
    at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.createSplits(JobImpl.java:1563)
    ... 17 more

请告知。
参考文献:
通过oozie hdp 2.1的作业未创建job.splitmetainfo
https://community.cloudera.com/t5/batch-processing-and-workflow/arnruntimeexception-java-io-filenotfoundexception-job/td-p/27063 等等。。。。

7kqas0il

7kqas0il1#

在提交wfs、coordinators、Bundler之前,您可以尝试切换到yarn user而不是hue(从web浏览器注销,使用用户名yarn重新登录)

相关问题