hive—hiveserver2在hadoop群集中频繁关闭

zfciruhq  于 2021-06-01  发布在  Hadoop
关注(0)|答案(2)|浏览(573)

面对这个问题,从相当长的一段时间以来,无法跟踪它发生的原因。每当我们使用命令->

./hiveserver2 &

它会启动并保持一段时间,然后关闭。在配置单元日志中,当配置单元服务器启动并运行时,它确实显示以下错误。

2018-03-12 04:44:57,029 ERROR [HiveServer2-Handler-Pool: Thread-33]: server.TThreadPoolServer (TThreadPoolServer.java:run(296)) - Erro
r occurred during processing of message.
java.lang.RuntimeException: org.apache.thrift.transport.TSaslTransportException: No data or no sasl data in the stream
        at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:268)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.thrift.transport.TSaslTransportException: No data or no sasl data in the stream
        at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:328)
        at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
        at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
        ... 4 more
2018-03-12 04:45:55,361 INFO  [main]: SessionState (SessionState.java:printInfo(951)) -
Logging initialized using configuration in file:/usr/local/hive/conf/hive-log4j.properties

但是我不确定hiveserver的关闭是否是由于上面的错误,因为它在关闭之前持续运行了几个小时。
以下是hiveserver关闭时出现的配置单元日志

2018-03-12 04:46:25,285 INFO  [main]: ql.Driver (SessionState.java:printInfo(951)) - Stage-Stage-1: Map: 4  Reduce: 1   Cumulative CPU
: 18.09 sec   HDFS Read: 763046 HDFS Write: 2217 SUCCESS
2018-03-12 04:46:25,286 INFO  [main]: ql.Driver (SessionState.java:printInfo(951)) - Total MapReduce CPU Time Spent: 18 seconds 90 mse
c
2018-03-12 04:46:25,286 INFO  [main]: ql.Driver (SessionState.java:printInfo(951)) - OK
2018-03-12 04:46:25,286 INFO  [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG method=releaseLocks from=org.apach
e.hadoop.hive.ql.Driver>
2018-03-12 04:46:25,295 INFO  [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(148)) - </PERFLOG method=releaseLocks start=152082998
5286 end=1520829985295 duration=9 from=org.apache.hadoop.hive.ql.Driver>
2018-03-12 04:46:25,295 INFO  [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(148)) - </PERFLOG method=Driver.run start=15208299614
77 end=1520829985295 duration=23818 from=org.apache.hadoop.hive.ql.Driver>
2018-03-12 04:46:25,304 INFO  [main]: CliDriver (SessionState.java:printInfo(951)) - Time taken: 23.818 seconds
2018-03-12 04:46:25,304 INFO  [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG method=releaseLocks from=org.apach
e.hadoop.hive.ql.Driver>
2018-03-12 04:46:25,305 INFO  [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(148)) - </PERFLOG method=releaseLocks start=152082998
5304 end=1520829985305 duration=1 from=org.apache.hadoop.hive.ql.Driver>
2018-03-12 04:46:36,351 INFO  [Thread-9]: server.HiveServer2 (HiveServer2.java:stop(305)) - Shutting down HiveServer2
2018-03-12 04:46:36,351 INFO  [Thread-9]: thrift.ThriftCLIService (ThriftCLIService.java:stop(201)) - Thrift server has stopped
2018-03-12 04:46:36,351 INFO  [Thread-9]: service.AbstractService (AbstractService.java:stop(125)) - Service:ThriftBinaryCLIService is
 stopped.
2018-03-12 04:46:36,351 INFO  [Thread-9]: service.AbstractService (AbstractService.java:stop(125)) - Service:OperationManager is stopp
ed.
2018-03-12 04:46:36,351 INFO  [Thread-9]: service.AbstractService (AbstractService.java:stop(125)) - Service:SessionManager is stopped
.
2018-03-12 04:46:36,351 INFO  [Thread-3]: server.HiveServer2 (HiveStringUtils.java:run(709)) - SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down HiveServer2 at SERVER-HOSTNAME/192.168.***.**

************************************************************/

2018-03-12 04:46:46,352 WARN  [Thread-9]: service.CompositeService (SessionManager.java:cleanupLoggingRootDir(213)) - Failed to cleanu
p root dir of HS2 logging: /usr/local/hive/log
java.io.FileNotFoundException: File does not exist: /usr/local/hive/log
        at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2275)
        at org.apache.hive.service.cli.session.SessionManager.cleanupLoggingRootDir(SessionManager.java:211)
        at org.apache.hive.service.cli.session.SessionManager.stop(SessionManager.java:205)
        at org.apache.hive.service.CompositeService.stop(CompositeService.java:102)
        at org.apache.hive.service.CompositeService.stop(CompositeService.java:92)
        at org.apache.hive.service.cli.CLIService.stop(CLIService.java:165)
        at org.apache.hive.service.CompositeService.stop(CompositeService.java:102)
        at org.apache.hive.service.CompositeService.stop(CompositeService.java:92)
        at org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:307)
        at org.apache.hive.service.server.HiveServer2$1.run(HiveServer2.java:107)
2018-03-12 04:46:46,353 INFO  [Thread-9]: service.AbstractService (AbstractService.java:stop(125)) - Service:CLIService is stopped.
2018-03-12 04:46:46,353 INFO  [Thread-9]: service.AbstractService (AbstractService.java:stop(125)) - Service:HiveServer2 is stopped.
2018-03-12 04:51:07,336 INFO  [main]: SessionState (SessionState.java:printInfo(951)) -
Logging initialized using configuration in file:/usr/local/hive/conf/hive-log4j.properties

如果问题真的是因为。。。

ERROR [HiveServer2-Handler-Pool: Thread-33]: server.TThreadPoolServer (TThreadPoolServer.java:run(296)) - Erro
    r occurred during processing of message.
    java.lang.RuntimeException: org.apache.thrift.transport.TSaslTransportException: No data or no sasl data in the stream

…然后这里是我的hive-site.xml设置,它与许多其他相关文章中提到的内容相关。

<name>hive.server2.authentication</name>
<value>PAM</value>

<name>hive.server2.authentication.pam.services</name>
<value>sshd,sudo</value>

<name>hive.server2.thrift.sasl.qop</name>
<value>auth</value>

<name>hive.metastore.sasl.enabled</name>
<value>false</value>

编辑

尝试在将hive.server2.authentication从pam更改为none后启动hiveserver,但hiveserver再次启动,出现以下错误

ERROR [HiveServer2-Handler-Pool: Thread-31]: server.TThreadPoolServer (TThreadPoolServer.java:run(296)) - Error occurred during processing of message.
java.lang.RuntimeException: org.apache.thrift.transport.TSaslTransportException: No data or no sasl data in the stream

另外,当尝试连接到beeline时,它会按预期引发连接异常,

bin$ ./beeline
Beeline version 1.2.2 by Apache Hive
beeline> !connect jdbc:hive2://192.168.XXX.XX:XXX7 myuser myp@sw0rd
Connecting to jdbc:hive2://192.168.XXX.XX:XXX7
Error: Could not open client transport with JDBC Uri: jdbc:hive2://192.168.203.XXX.XX:XXX7: java.net.ConnectException: Connection timed out (Connection timed out) (state=08S01,code=0)
0: jdbc:hive2://192.168.XXX.XX:XXX7 (closed)>
0: jdbc:hive2://192.168.XXX.XX:XXX7 (closed)>

而ps-ef | grep hive显示hiveserver已启动

ps -ef | grep hive
hduser   30902 30165  1 05:39 pts/1    00:00:15 /data/apps/jdk/bin/java -Xmx4000m -Djava.library.path=/usr/local/hadoop/lib -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/usr/local/hadoop/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/local/hadoop -Dhadoop.id.str=hduser -Dhadoop.root.logger=INFO,console -Djava.library.path=/usr/local/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/local/hive/lib/hive-service-1.2.2.jar org.apache.hive.service.server.HiveServer2
jm2pwxwz

jm2pwxwz1#

hiveserver2文档提到 PAM 身份验证模式,如果用户的密码已过期,则会导致服务器宕机。请检查是否是这种情况,您也可以尝试设置 hive.server2.authenticationNONE 并检查是否允许您连接到服务器。

2o7dmzc5

2o7dmzc52#

连接超时可能只是因为它根本没有在端口上侦听,或者没有被授权连接。 netstat -na 检查端口监听 /etc/security/access.conf 或者 iptable -L ?

相关问题