使用kerberos安全性连接到mapper中的配置单元

4xy9mtcn 于 2021-05-29 发布在 Hadoop

关注(0)|答案(2)|浏览(385)

我的目标是在安全集群（kerberos）hdp2.3上运行mapreduce并连接到oozie工作流调度器中的配置单元。
我可以使用以下连接字符串以直线方式连接到hive，或者以java应用程序（yarn jar）的形式运行它：

DriverManager.getConnection("jdbc:hive2://host:10000/;principal=hive/_HOST@REALM", "", "");

但当我在mapper中运行它时失败了。

ERROR [main] org.apache.thrift.transport.TSaslTransport: SASL negotiation failure
    javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
        at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
        at org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
        at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
        at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
        at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
        at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
        at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
        at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:190)
        ...
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
        at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
        at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
        at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
        at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
        at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
        at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
        at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)

如何使它在mapper中工作？

hadoop Hive mapreduce oozie kerberos

来源：https://stackoverflow.com/questions/33255350/connecting-to-hive-in-mapper-with-kerberos-security

2条答案

按热度按时间

cvxl0en21#

错误“failed to find any kerberos tgt”非常简单。在连接到正在执行Map程序的节点上的配置单元之前，需要获取kerberos票证。

赞(0）回复(0）举报 2021-05-30

bmp9r5qi2#

它使用配置单元委派令牌：
奥齐
添加属性：

hive2.server.principal=hive/_HOST@REALM
hive2.jdbc.url=jdbc:hive2://{host}:10000/default

将凭据设置为配置单元2
Map器示例：

public class HiveMapperExample extends Mapper<LongWritable, Text, Text, Text> {

    @Override
    protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
        try {
            Class.forName("org.apache.hive.jdbc.HiveDriver");
            Connection connect = DriverManager.getConnection("jdbc:hive2://{host}:10000/;auth=delegationToken", "", "");
            Statement state = connect.createStatement();
            ResultSet resultSet = state.executeQuery("select * from some_table");
            while (resultSet.next()) {
                ...
            }
        } catch (Exception e) {
            ...
        }
    }
 }

刀槽

public class HiveTestApplication extends Configured implements Tool {

    public static void main(String[] args) throws Exception {
        System.exit(ToolRunner.run(new HiveTestApplication(), args));
    }

    @Override
    public int run(String[] args) throws Exception {
        Configuration conf = new Configuration();
        //set your conf
        Job job = Job.getInstance(conf);
        job.setMapperClass(HiveMapperExample.class);

        addHiveDelegationToken(job.getCredentials(), "jdbc:hive2://{host}:10000/", "hive/_HOST@REALM");

        job.waitForCompletion(true);

        return 0;
    }

    public void addHiveDelegationToken(Credentials creds, String url, String principal) throws Exception {
        Class.forName("org.apache.hive.jdbc.HiveDriver");

        Connection con = DriverManager.getConnection(url + ";principal=" + principal);
        // get delegation token for the given proxy user
        String tokenStr = ((HiveConnection) con).getDelegationToken(UserGroupInformation.getCurrentUser().getShortUserName(), principal);
        con.close();

        Token<DelegationTokenIdentifier> hive2Token = new Token<>();
        hive2Token.decodeFromUrlString(tokenStr);
        creds.addToken(new Text("hive.server2.delegation.token"), hive2Token);
        creds.addToken(new Text(HiveAuthFactory.HS2_CLIENT_TOKEN), hive2Token);
    }
}

赞(0）回复(0）举报 2021-05-29

我来回答

使用kerberos安全性连接到mapper中的配置单元

2条答案

相关问题

热门标签

最新问答