我需要安排一个ooziejava操作,该操作与安全的hbase交互,因此我需要向java操作提供hbase凭据。我使用的是一个安全的hortonworks2.2环境,我的工作流程xml如下
<workflow-app xmlns="uri:oozie:workflow:0.4" name="solr-wf">
<credentials>
<credential name="hbase" type="hbase">
</credential>
</credentials>
<start to="java-node"/>
<action name="java-node" cred="hbase">
<java>
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<main-class>com.test.hbase.TestHBaseSecure</main-class>
<arg>${arg1}</arg>
</java>
<ok to="end"/>
<error to="fail"/>
</action>
<kill name="fail">
<message>Java failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<end name="end"/>
</workflow-app>
我还修改了oozie属性以包含hbasecredentials类
oozie.credentials.credentialclasses=hcat=org.apache.oozie.action.hadoop.HCatCredentials,hbase=org.apache.oozie.action.hadoop.HbaseCredentials
但我无法运行的作业,它抛出了一个错误,下面是stacktrace
java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/HBaseConfiguration
at org.apache.oozie.action.hadoop.HbaseCredentials.copyHbaseConfToJobConf(HbaseCredentials.java:60)
at org.apache.oozie.action.hadoop.HbaseCredentials.addtoJobConf(HbaseCredentials.java:49)
at org.apache.oozie.action.hadoop.JavaActionExecutor.setCredentialTokens(JavaActionExecutor.java:1054)
at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:913)
at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1135)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:228)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:63)
at org.apache.oozie.command.XCommand.call(XCommand.java:281)
at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:323)
at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:252)
at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:174)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
其他作业运行良好,只有hbase交互的作业失败。我已经在lib目录中包含了所有的hbase jar,我无法找出问题所在。
更新的workflow.xml:
<workflow-app xmlns="uri:oozie:workflow:0.4" name="${appName}">
<credentials>
<credential name="hbase-cred" type="hbase">
<property>
<name>hbase.master.kerberos.principal</name>
<value>hbase/_HOST@ABC.COM</value>
</property>
<property>
<name>hbase.master.keytab.file</name>
<value>/etc/security/keytabs/hbase.service.keytab</value>
</property>
<property>
<name>hbase.regionserver.kerberos.principal</name>
<value>hbase/_HOST@ABC.COM</value>
</property>
<property>
<name>hbase.regionserver.keytab.file</name>
<value>/etc/security/keytabs/hbase.service.keytab</value>
</property>
<property>
<name>hbase.security.authentication</name>
<value>kerberos</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>dev1-dn2,dev1-dn3,dev1-dn1</value>
</property>
<property>
<name>zookeeper.znode.parent</name>
<value>/hbase-secure</value>
</property>
</credential>
</credentials>
<start to="java-node" />
<action name="java-node" cred='hbase-cred'>
<java>
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<main-class>com.test.hbase.TestHBaseSecure</main-class>
</java>
<ok to="end" />
<error to="fail" />
</action>
<kill name="fail">
<message>Java failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<end name="end" />
</workflow-app>
2条答案
按热度按时间tjrkku2a1#
该溶液在hdp2.2.8上测试:
复制到
/usr/hdp/current/oozie-server/oozie-server/webapps/oozie/WEB-INF/lib
以下jar:hbase客户端--hadoop2.jar
hbase公共--hadoop2.jar
hbase协议--hadoop2.jar
hbase服务器--hadoop2.jar
htrace-core-2.04.jar
重新启动oozie服务器。
sg24os4d2#
这些“凭证”由oozie服务管理,而不是由您的作业管理。
所以,如果hortonworks在 Package 发行版方面做得不错的话。。。
hbase-common-*-hadoop2.jar
会被部署在/usr/hdp/current/oozie-client/libserver/
安装时在org.apache.hadoop.conf.configuration类的定义上,该jar不会与其他jar冲突
最后,您将能够在oozie中管理hbase凭据
对于安装在prod集群上的hdp2.2.4,情况并非如此。啊。那该死的东西在那该死的释放器里坏了。您必须自己管理kerberos票证,下载一个keytab
<file>
在实际连接到hbase之前创建tgt。我们去过那里。看看那篇文章,了解一下如何做到这一点。