如何连接到特定的zookeeper集群?

s5a0g9ez  于 2021-05-29  发布在  Hadoop
关注(0)|答案(0)|浏览(277)

我正在实现一个scala函数,它解析json文件并将数据加载到hbase中。它在zookeeper服务器为localhost的本地机器上运行良好。
现在为了缩放的目的,我尝试在一个hadoop集群上实现这个功能,这个集群有三个特定的zookeeper集群node00,node01,solr。
我想知道如何通过我的代码访问特定的集群。我是否需要事先连接它们,因为总是有以下错误。

17/12/27 18:20:17 WARN zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting to reconnect

java.net.ConnectException: Connection refused

请尽快回复我
编辑1:
非常感谢您的回复。我无法按您建议的方法进行检查,因为我没有足够的权限。下面是接收输入json并解析为hbase的代码。

val input_base =""

val files = List("ffff")

for (collection_name <- files){
    val input_filename = input_base + collection_name + ".json"
    var col_name = collection_name
    var collection_Id = "1007"

    @transient val hadoopConf = new Configuration()
    @transient val conf = HBaseConfiguration.create(hadoopConf)
    conf.set(TableOutputFormat.OUTPUT_TABLE, tableName)
    @transient val jobConfig: JobConf = new JobConf(conf, this.getClass)
    jobConfig.setOutputFormat(classOf[TableOutputFormat])
    jobConfig.set(TableOutputFormat.OUTPUT_TABLE, tableName)

    sc.textFile(input_filename).map(l => parse_json(l,col_name,collection_Id)).saveAsHadoopDataset(jobConfig)

}

我需要帮助在我的代码中提供这些节点作为参数。zookeeper仲裁将solr2.dlrl、node02.dlrl、node00.dlrl作为节点,将2181作为端口。请帮我解决这个问题
如果我运行rowcounter函数,它就能够正确地连接到集群

hbase org.apache.hadoop.hbase.mapreduce.RowCounter ram

具体如下:

18/01/02 14:52:41 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
18/01/02 14:52:41 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
18/01/02 14:52:41 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
18/01/02 14:52:41 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
18/01/02 14:52:41 INFO zookeeper.ZooKeeper: Client environment:os.version=2.6.32-696.6.3.el6.x86_64
18/01/02 14:52:41 INFO zookeeper.ZooKeeper: Client environment:user.name=cs5604f17_cta
18/01/02 14:52:41 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/cs5604f17_cta
18/01/02 14:52:41 INFO zookeeper.ZooKeeper: Client environment:user.dir=/home/cs5604f17_cta/ram/cmt/CMT/cs5604f17_cmt
18/01/02 14:52:41 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=solr2.dlrl:2181,node02.dlrl:2181,node00.dlrl:2181 sessionTimeout=60000 watcher=hconnection-0x524af2a60x0, quorum=solr2.dlrl:2181,node02.dlrl:2181,node00.dlrl:2181, baseZNode=/hbase
18/01/02 14:52:41 INFO zookeeper.ClientCnxn: Opening socket connection to server solr2.dlrl/10.0.0.125:2181. Will not attempt to authenticate using SASL (unknown error)
18/01/02 14:52:41 INFO zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.0.0.127:54862, server: solr2.dlrl/10.0.0.125:2181
18/01/02 14:52:41 INFO zookeeper.ClientCnxn: Session establishment complete on server solr2.dlrl/10.0.0.125:2181, sessionid = 0x160aa7b7c311ea2, negotiated timeout = 60000
18/01/02 14:52:41 INFO util.RegionSizeCalculator: Calculating region sizes for table "ram-irma".
18/01/02 14:52:42 INFO client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
18/01/02 14:52:42 INFO client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x160aa7b7c311ea2
18/01/02 14:52:42 INFO zookeeper.ZooKeeper: Session: 0x160aa7b7c311ea2 closed
18/01/02 14:52:42 INFO zookeeper.ClientCnxn: EventThread shut down
18/01/02 14:52:42 INFO mapreduce.JobSubmitter: number of splits:12
18/01/02 14:52:42 INFO Configuration.deprecation: io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
18/01/02 14:52:43 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1514688768112_0007
18/01/02 14:52:43 INFO impl.YarnClientImpl: Submitted application application_1514688768112_0007
18/01/02 14:52:43 INFO mapreduce.Job: The url to track the job: http://node02.dlrl:8088/proxy/application_1514688768112_0007/
18/01/02 14:52:43 INFO mapreduce.Job: Running job: job_1514688768112_0007
18/01/02 14:52:48 INFO mapreduce.Job: Job job_1514688768112_0007 running in uber mode : false

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题