Spark流Kafka

h43kikqp  于 2021-06-07  发布在  Kafka
关注(0)|答案(1)|浏览(235)

我在Kafka做poc spark流媒体工作。我可以对不安全的kafka 0.10群集使用相同的代码,但是当我切换到针对ssl/kerberos(hdp 2.5)设置运行时,我遇到了一个异常:

Caused by: javax.security.auth.login.LoginException: Could not login: the client is being asked for a password, but the Kafka client code does not currently support obtaining a password from the user. not available to garner  authentication information from the user
at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:940)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:760)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.kafka.common.security.authenticator.AbstractLogin.login(AbstractLogin.java:69)
at org.apache.kafka.common.security.kerberos.KerberosLogin.login(KerberosLogin.java:110)
at org.apache.kafka.common.security.authenticator.LoginManager.<init>(LoginManager.java:46)
at org.apache.kafka.common.security.authenticator.LoginManager.acquireLoginManager(LoginManager.java:68)
at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:78)
... 29 more

spark会话创建得很好,但是当执行器启动以使用来自主题的新内容时,我得到了上面的异常。
提交代码相当简单:

spark-submit \
--master yarn \
--keytab ./bilsch.keytab \
--principal bilsch@HDP.SOME.ORG \
--files kafka_client_jaas.conf,bilsch.keytab \
--packages org.apache.spark:spark-streaming-kafka-0-10_2.11:2.0.2 \
--repositories http://repo.hortonworks.com/content/repositories/releases \
--num-executors 1 \
--class producer \
--driver-java-options "-Djava.security.auth.login.config=./kafka_client_jaas.conf -Dhdp.version=2.5.3.0-37" \
--conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=./kafka_client_jaas.conf" \
streaming_0331_2.11-1.0.jar 2>&1 | tee out

如果有帮助的话,我会跟随迈克尔·诺尔斯的《Kafka》教程
不确定哪些内容需要传递给执行器,而我还没有传递,或者可能只是jaas配置问题?

htzpubme

htzpubme1#

尝试遵循提供的步骤below:-
删除 --keytab 以及 --principal 来自spark的submit命令,因为此信息已经存在于jaas配置中。
kinit -kt bilsch.keytab bilsch@HDP.SOME.ORG 然后尝试运行spark submit命令。

相关问题