在hadoop2.6.5ha和oozie(使用oozie-4.1.0-cdh5.12.1)中运行oozie示例。
[oozie@master shell]$ cat job.properties
nameNode=hdfs://cluster1:8020
jobTracker=master:8032
queueName=default
examplesRoot=examples
oozie.wf.application.path=${nameNode}/user/oozie/${examplesRoot}/apps/shell
[hadoop@master sbin]$
[hadoop@master sbin]$ oozie job -oozie http://master.bigdata.com:11000/oozie -config /home/hadoop/app/oozie/examples/apps/map-reduce/job.properties -run
Error: HTTP error code: 500 : Internal Server Error
[hadoop@master sbin]$
[hadoop@master shell]$ oozie job -oozie http://master.bigdata.com:11000/oozie -config /home/hadoop/app/oozie/examples/apps/shell/job.properties -run
Error: HTTP error code: 500 : Internal Server Error
[hadoop@master shell]$
错误是:
[oozie@master logs]$ pwd
/home/hadoop/app/oozie/logs
[oozie@master logs]$ vi oozie.log
2017-09-06 00:33:19,850 WARN AuthenticationFilter:532 - SERVER[master.bigdata.com] AuthenticationToken ignored: org.apache.hadoop.security.authentication.util.SignerException: Invalid signature
2017-09-06 00:33:19,924 ERROR SubmitXCommand:517 - SERVER[master.bigdata.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Error,
java.lang.NoSuchMethodError: org.apache.hadoop.fs.FsServerDefaults.<init>(JIISIZJLorg/apache/hadoop/util/DataChecksum$Type;)V
at org.apache.hadoop.hdfs.protocolPB.PBHelper.convert(PBHelper.java:1327)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getServerDefaults(ClientNamenodeProtocolTranslatorPB.java:267)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:260)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
at com.sun.proxy.$Proxy28.getServerDefaults(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getServerDefaults(DFSClient.java:996)
at org.apache.hadoop.hdfs.DFSClient.shouldEncryptData(DFSClient.java:2032)
at org.apache.hadoop.hdfs.DFSClient.newDataEncryptionKey(DFSClient.java:2038)
at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.checkTrustAndSend(SaslDataTransferClient.java:208)
at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.peerSend(SaslDataTransferClient.java:159)
at org.apache.hadoop.hdfs.net.TcpPeerServer.peerFromSocketAndKey(TcpPeerServer.java:90)
at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:3093)
at org.apache.hadoop.hdfs.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:778)
at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:693)
at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:354)
at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:617)
at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:841)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:889)
at java.io.DataInputStream.read(DataInputStream.java:149)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.Reader.read(Reader.java:140)
at org.apache.oozie.util.IOUtils.copyCharStream(IOUtils.java:171)
at org.apache.oozie.service.WorkflowAppService.readDefinition(WorkflowAppService.java:135)
at org.apache.oozie.service.LiteWorkflowAppService.parseDef(LiteWorkflowAppService.java:46)
at org.apache.oozie.command.wf.SubmitXCommand.execute(SubmitXCommand.java:165)
at org.apache.oozie.command.wf.SubmitXCommand.execute(SubmitXCommand.java:76)
at org.apache.oozie.command.XCommand.call(XCommand.java:286)
at org.apache.oozie.DagEngine.submitJob(DagEngine.java:114)
at org.apache.oozie.servlet.V1JobsServlet.submitWorkflowJob(V1JobsServlet.java:192)
at org.apache.oozie.servlet.V1JobsServlet.submitJob(V1JobsServlet.java:92)
at org.apache.oozie.servlet.BaseJobsServlet.doPost(BaseJobsServlet.java:102)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:643)
at org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:289)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
at org.apache.oozie.command.wf.SubmitXCommand.execute(SubmitXCommand.java:76)
at org.apache.oozie.command.XCommand.call(XCommand.java:286)
at org.apache.oozie.DagEngine.submitJob(DagEngine.java:114)
at org.apache.oozie.servlet.V1JobsServlet.submitWorkflowJob(V1JobsServlet.java:192)
at org.apache.oozie.servlet.V1JobsServlet.submitJob(V1JobsServlet.java:92)
at org.apache.oozie.servlet.BaseJobsServlet.doPost(BaseJobsServlet.java:102)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:643)
at org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:289)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:171)
at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:631)
at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:579)
at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:176)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:86)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:610)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:503)
at java.lang.Thread.run(Thread.java:745)
2017-09-06 00:33:58499信息状态传输服务$statustransitrunnable:520 - 服务器[master.bigdata.com]用户[-]组[-]令牌[-]应用程序[-]作业[-]操作[-]获取了[org.apache.oozie.service.statustransitservice]的锁
1条答案
按热度按时间vqlkdk9b1#
在我的hadoop2.6.5ha和oozie中(使用
oozie-4.1.0-cdh5.12.1
)oozie-4.1.0+cdh5.12.1
主要目标是hadoop-2.6.0+cdh5.12.1
尝试混合版本或自己编译任何更高版本只会引起问题。具体来说,您有一个类路径问题
java.lang.NoSuchMethodError: org.apache.hadoop.fs.FsServerDefaults.<init>
如果您坚持使用cloudera打包,您可以在这里找到必要的下载。https://www.cloudera.com/documentation/enterprise/release-notes/topics/cm_vd_cdh_package_tarball_512.html#cm_vd_cdh_package_tarball_512
我的建议是安装clouderamanager,让它为您安装和配置cdh组件