最近,我的团队正在cdh上使用hbase indexer将hbase表列索引到solr。当我们部署hbase indexer服务器(称为键值存储索引器)并开始测试时。我们发现一种情况,当我们频繁地将数据放入hbase(我们使用的是apache phoenix,hbase之上的一个sql层)时,hbase indexer进程会自动退出,我们检查了日志,发现zookeeper session的错误日志过期如下:
2016-04-18 12:17:50,340 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 31481ms for sessionid 0x2541e69d8a2001a, closing socket connection and attempting reconnect
2016-04-18 12:17:50,446 WARN com.ngdata.hbaseindexer.util.zookeeper.StateWatchingZooKeeper: Disconnected from ZooKeeper
2016-04-18 12:17:51,202 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server slave1/192.168.27.166:2181. Will not attempt to authenticate using SASL (unknown error)
2016-04-18 12:17:51,204 INFO org.apache.zookeeper.ClientCnxn: Socket connection established, initiating session, client: /192.168.27.167:59529, server: slave1/192.168.27.166:2181
2016-04-18 12:17:51,211 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x2541e69d8a2001a has expired, closing socket connection
2016-04-18 12:17:51,211 ERROR com.ngdata.hbaseindexer.util.zookeeper.StateWatchingZooKeeper: ZooKeeper session expired, shutting down.
2016-04-18 12:17:51,228 INFO org.mortbay.log: Stopped SelectChannelConnector@0.0.0.0:11060
2016-04-18 12:17:51,336 INFO com.ngdata.hbaseindexer.supervisor.IndexerSupervisor: IndexerWorker.EventWorker interrupted.
2016-04-18 12:17:51,448 INFO org.apache.zookeeper.ZooKeeper: Session: 0x2541e69d8a20020 closed
2016-04-18 12:17:51,448 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2016-04-18 12:17:51,462 INFO org.apache.hadoop.hbase.ipc.RpcServer: Stopping server on 44594
2016-04-18 12:17:51,463 INFO org.apache.hadoop.hbase.ipc.RpcServer: RpcServer.listener,port=44594: stopping
2016-04-18 12:17:51,473 INFO org.apache.hadoop.hbase.ipc.RpcServer: RpcServer.responder: stopped
2016-04-18 12:17:51,473 INFO org.apache.hadoop.hbase.ipc.RpcServer: RpcServer.responder: stopping
2016-04-18 12:17:51,488 ERROR com.ngdata.sep.util.io.Closer: Do not know how to close object of type com.ngdata.hbaseindexer.morphline.MorphlineResultToSolrMapper
2016-04-18 12:17:51,488 ERROR com.ngdata.sep.util.io.Closer: Do not know how to close object of type com.ngdata.hbaseindexer.uniquekey.StringUniqueKeyFormatter
2016-04-18 12:17:51,514 INFO org.apache.zookeeper.ZooKeeper: Session: 0x2541e69d8a2001f closed
2016-04-18 12:17:51,515 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2016-04-18 12:17:51,515 INFO org.apache.hadoop.hbase.ipc.RpcServer: Stopping server on 47364
2016-04-18 12:17:51,516 INFO org.apache.hadoop.hbase.ipc.RpcServer: RpcServer.listener,port=47364: stopping
2016-04-18 12:17:51,518 ERROR com.ngdata.sep.util.io.Closer: Do not know how to close object of type com.ngdata.hbaseindexer.morphline.MorphlineResultToSolrMapper
2016-04-18 12:17:51,518 INFO org.apache.hadoop.hbase.ipc.RpcServer: RpcServer.responder: stopped
2016-04-18 12:17:51,519 INFO org.apache.hadoop.hbase.ipc.RpcServer: RpcServer.responder: stopping
2016-04-18 12:17:51,519 ERROR com.ngdata.sep.util.io.Closer: Do not know how to close object of type com.ngdata.hbaseindexer.uniquekey.StringUniqueKeyFormatter
2016-04-18 12:17:51,527 INFO org.apache.zookeeper.ZooKeeper: Session: 0x2541e69d8a2001e closed
2016-04-18 12:17:51,527 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2016-04-18 12:17:51,528 INFO org.apache.hadoop.hbase.ipc.RpcServer: Stopping server on 49605
2016-04-18 12:17:51,528 INFO org.apache.hadoop.hbase.ipc.RpcServer: RpcServer.listener,port=49605: stopping
2016-04-18 12:17:51,530 INFO org.apache.hadoop.hbase.ipc.RpcServer: RpcServer.responder: stopped
2016-04-18 12:17:51,530 INFO org.apache.hadoop.hbase.ipc.RpcServer: RpcServer.responder: stopping
2016-04-18 12:17:51,531 ERROR com.ngdata.sep.util.io.Closer: Do not know how to close object of type com.ngdata.hbaseindexer.morphline.MorphlineResultToSolrMapper
2016-04-18 12:17:51,531 ERROR com.ngdata.sep.util.io.Closer: Do not know how to close object of type com.ngdata.hbaseindexer.uniquekey.StringUniqueKeyFormatter
2016-04-18 12:17:51,539 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2016-04-18 12:17:51,539 INFO org.apache.zookeeper.ZooKeeper: Session: 0x2541e69d8a2001c closed
2016-04-18 12:17:51,540 INFO org.apache.hadoop.hbase.ipc.RpcServer: Stopping server on 39464
2016-04-18 12:17:51,540 INFO org.apache.hadoop.hbase.ipc.RpcServer: RpcServer.listener,port=39464: stopping
2016-04-18 12:17:51,546 INFO org.apache.hadoop.hbase.ipc.RpcServer: RpcServer.responder: stopped
2016-04-18 12:17:51,547 INFO org.apache.hadoop.hbase.ipc.RpcServer: RpcServer.responder: stopping
2016-04-18 12:17:51,547 ERROR com.ngdata.sep.util.io.Closer: Do not know how to close object of type com.ngdata.hbaseindexer.morphline.MorphlineResultToSolrMapper
2016-04-18 12:17:51,547 ERROR com.ngdata.sep.util.io.Closer: Do not know how to close object of type com.ngdata.hbaseindexer.uniquekey.StringUniqueKeyFormatter
软件环境是:
cdh5.4型
hbase1.0版
Phoenix4.6
hbase索引器(hbase-solr-1.5-cdh5.4.2)
hbase indexer的java堆大小配置为1gb。
有人遇到这种情况吗?
1条答案
按热度按时间pdsfdshx1#
好吧,这是由于我们集群的网络不好导致hbase索引器zookeeper会话超时。最后导致hbase indexer进程自动关闭。