lily hbase索引器无缘无故退出

zz2j4svz  于 2021-05-30  发布在  Hadoop
关注(0)|答案(1)|浏览(381)

我正在运行一个cloudera/solr集群,并尝试使用hbase solr(lily)索引器将hbase的nrt索引到solr中。批处理模式索引工作正常。
然而,在我开始在一个恒定流中加载数据之后,lily索引器开始一个接一个地消亡。他们不会打印出一个特定的错误消息,但都以相同的方式结束:

2014-09-10 16:04:56,770 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=ip-172-31-1-204.ap-southeast-2.compute.internal,44013,1410329096767 connecting to ZooKeeper ensemble=ip-172-31-1-205.ap-southeast-2.compute.internal:2181,ip-172-31-1-206.ap-southeast-2.compute.internal:2181,ip-172-31-1-204.ap-southeast-2.compute.internal:2181
2014-09-10 16:04:56,771 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server ip-172-31-1-206.ap-southeast-2.compute.internal/172.31.1.206:2181. Will not attempt to authenticate using SASL (unknown error)
2014-09-10 16:04:56,772 INFO org.apache.hadoop.ipc.RpcServer: RpcServer.listener,port=44013: starting
2014-09-10 16:04:56,771 INFO org.apache.hadoop.ipc.RpcServer: RpcServer.responder: starting
2014-09-10 16:04:56,773 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to ip-172-31-1-206.ap-southeast-2.compute.internal/172.31.1.206:2181, initiating session
2014-09-10 16:04:56,775 INFO com.ngdata.hbaseindexer.supervisor.IndexerSupervisor: Started indexer for indexFeature
2014-09-10 16:04:56,776 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server ip-172-31-1-206.ap-southeast-2.compute.internal/172.31.1.206:2181, sessionid = 0x1485c7ff13602fd, negotiated timeout = 60000
2014-09-10 16:04:56,813 INFO org.kitesdk.morphline.api.MorphlineContext: Importing commands
2014-09-10 16:04:57,287 INFO org.kitesdk.morphline.api.MorphlineContext: Done importing commands
2014-09-10 16:04:57,289 INFO org.apache.solr.client.solrj.impl.HttpClientUtil: Creating new http client, config:
2014-09-10 16:04:57,297 INFO org.apache.hadoop.ipc.RpcServer: regionserver/ip-172-31-1-204.ap-southeast-2.compute.internal/172.31.1.204:0: started 10 reader(s).
2014-09-10 16:04:57,299 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=ip-172-31-1-205.ap-southeast-2.compute.internal:2181,ip-172-31-1-206.ap-southeast-2.compute.internal:2181,ip-172-31-1-204.ap-southeast-2.compute.internal:2181 sessionTimeout=60000 watcher=ip-172-31-1-204.ap-southeast-2.compute.internal,44713,1410329097297, quorum=ip-172-31-1-205.ap-southeast-2.compute.internal:2181,ip-172-31-1-206.ap-southeast-2.compute.internal:2181,ip-172-31-1-204.ap-southeast-2.compute.internal:2181, baseZNode=/hbase
2014-09-10 16:04:57,301 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=ip-172-31-1-204.ap-southeast-2.compute.internal,44713,1410329097297 connecting to ZooKeeper ensemble=ip-172-31-1-205.ap-southeast-2.compute.internal:2181,ip-172-31-1-206.ap-southeast-2.compute.internal:2181,ip-172-31-1-204.ap-southeast-2.compute.internal:2181
2014-09-10 16:04:57,302 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server ip-172-31-1-204.ap-southeast-2.compute.internal/172.31.1.204:2181. Will not attempt to authenticate using SASL (unknown error)
2014-09-10 16:04:57,303 INFO org.apache.hadoop.ipc.RpcServer: RpcServer.responder: starting
2014-09-10 16:04:57,303 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to ip-172-31-1-204.ap-southeast-2.compute.internal/172.31.1.204:2181, initiating session
2014-09-10 16:04:57,304 INFO org.apache.hadoop.ipc.RpcServer: RpcServer.listener,port=44713: starting
2014-09-10 16:04:57,306 INFO com.ngdata.hbaseindexer.supervisor.IndexerSupervisor: Started indexer for indexSeenBlock
2014-09-10 16:04:57,307 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server ip-172-31-1-204.ap-southeast-2.compute.internal/172.31.1.204:2181, sessionid = 0x3485c7fee8f0374, negotiated timeout = 60000
2014-09-10 16:04:57,349 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2014-09-10 16:04:57,536 INFO org.mortbay.log: jetty-6.1.26.cloudera.2
2014-09-10 16:04:58,663 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:11060
2014-09-10 16:05:01,591 INFO org.kitesdk.morphline.api.MorphlineContext: Importing commands
2014-09-10 16:05:01,597 INFO org.kitesdk.morphline.api.MorphlineContext: Importing commands
2014-09-10 16:05:01,641 INFO org.kitesdk.morphline.api.MorphlineContext: Importing commands
2014-09-10 16:05:01,650 INFO org.kitesdk.morphline.api.MorphlineContext: Importing commands
2014-09-10 16:05:01,688 INFO org.kitesdk.morphline.api.MorphlineContext: Importing commands
2014-09-10 16:05:01,726 INFO org.kitesdk.morphline.api.MorphlineContext: Importing commands
2014-09-10 16:05:01,732 INFO org.kitesdk.morphline.api.MorphlineContext: Importing commands
2014-09-10 16:05:01,740 INFO org.kitesdk.morphline.api.MorphlineContext: Importing commands
2014-09-10 16:05:01,752 INFO org.kitesdk.morphline.api.MorphlineContext: Importing commands

cloudera经理除了说进程已经退出之外,也没有给出任何有用的信息。solr索引中的某些记录确实会得到更新,这表明索引器至少在一段时间内工作正常。
我正在rhel6.5和jdk7上运行最新的CDH5.1。

jc3wubiy

jc3wubiy1#

如果您使用的是wal,即在您的hbase客户端程序(如mapreduce)中,write a head log选项为false(出于性能原因)。它不会写入hlog。
nrt在wal/hlog上工作。
批处理索引可以正常工作,因为它在wal上不起作用

相关问题