无法拒绝ssh权限hbase hadoop

3bygqnnd  于 2021-06-03  发布在  Hadoop
关注(0)|答案(1)|浏览(445)

我有一个amazonec2示例，我正在运行一个名为seqware的工具。它基本上是一个使用hbase后端的基因组数据查询引擎。我在一个ami上运行，这个ami在伪分布式模式下安装了hbase。我想在一个完全分布式的模式下使用它。所以我建立了一个2节点的hadoop集群。一个节点是主节点，另一个是从节点。我可以在完全分布式模式下运行hadoop示例。要使seqware使用我的完全编辑设置，它需要6个东西，zookeeper quorum、zookeeper client port、hbase master、mapred job tracker、fs default fs和fs name。在设置文件中指定。我在文件中有这样的设置：

HBASE.ZOOKEEPER.QUORUM=ip-10-x.x.x
HBASE.ZOOKEEPER.PROPERTY.CLIENTPORT=2181
HBASE.MASTER=ip-10-x.x.x:60010
MAPRED.JOB.TRACKER=ip-10-x.x.x:9001
FS.DEFAULT.NAME=hdfs://ip-10-x.x.x:9000
FS.DEFAULTFS=hdfs://ip-10-x.x.x:9000

然而，当我开始使用查询引擎时，我遇到了一个zookeeper连接丢失异常。我在seqware的authorized\ u keys中有主密钥，反之亦然，但是我不能ssh

ssh ip-10.x.x.x

甚至使用公共dns：

ssh {public DNS of instance}

其中ip-10.x.x.x是示例的ip地址，我必须使用用户名：

ssh {username}@ip-10-x.x.x

或

ssh username@{public DNS of instance}

我可以不使用用户名从主hadoop示例ssh到从hadoop示例ssh，反之亦然，我在配置文件中有没有用户名的ip地址
我试着在设置中把用户名加在ip地址之前，认为99%的可能性是不起作用的，我没有失望，我还是得到了同样的例外
我需要做什么才能从seqware示例ssh到hadoop和hbase主节点，而不必像在主节点和从节点之间那样指定用户名。
以下是在hadoop主机上配置zookeeper的方式：

<configuration>
<property>
    <name>hbase.rootdir</name>
    <value>hdfs://ip-10-x.x.x:9000/hbase</value>
  </property>

  <property>
    <name>hbase.cluster.distributed</name>
    <value>true</value>
  </property>

  <property>
    <name>hbase.zookeeper.quorum</name>
    <value>ip-10-x.x.x</value>
  </property>

  <property>
    <name>dfs.replication</name>
    <value>2</value>
  </property>

  <property>
    <name>hbase.zookeeper.property.dataDir</name>
    <value>/home/{username}/hbase/zookeeper</value>
  </property>

  <property>
    <name>hbase.zookeeper.property.clientPort</name>
    <value>2181</value>
  </property>
</configuration>

我不能自信地谈论seqware的内部实现，但我知道它使用设置文件来设置zookeepr和hbase master的位置。在默认的伪分布式工作设置中，这些是我前面提到的变量的值：

HBASE.ZOOKEEPER.QUORUM=localhost
HBASE.ZOOKEEPER.PROPERTY.CLIENTPORT=2181
HBASE.MASTER=localhost:60000
MAPRED.JOB.TRACKER=localhost:8021
FS.DEFAULT.NAME=hdfs://localhost:8020
FS.DEFAULTFS=hdfs://localhost:8020

zoo.cfg文件如下所示：


# Licensed to the Apache Software Foundation (ASF) under one or more

# contributor license agreements.  See the NOTICE file distributed with

# this work for additional information regarding copyright ownership.

# The ASF licenses this file to You under the Apache License, Version 2.0

# (the "License"); you may not use this file except in compliance with

# the License.  You may obtain a copy of the License at

# 

# http://www.apache.org/licenses/LICENSE-2.0

# 

# Unless required by applicable law or agreed to in writing, software

# distributed under the License is distributed on an "AS IS" BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and

# limitations under the License.

maxClientCnxns=50

# The number of milliseconds of each tick

tickTime=2000

# The number of ticks that the initial

# synchronization phase can take

initLimit=10

# The number of ticks that can pass between

# sending a request and getting an acknowledgement

syncLimit=5

# the directory where the snapshot is stored.

dataDir=/var/lib/zookeeper

# the port at which the clients will connect

clientPort=2181

zookeeper堆栈跟踪： [seqware@master target]$ java -classpath seqware-distribution-0.13.6.8-qe-full.jar ```
com.github.seqware.queryengine.system.ReferenceCreator hg_19 keyValue_ref.out
[SeqWare Query Engine] 0 [main] ERROR org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper - ZooKeeper exists failed after 3 retries
[SeqWare Query Engine] 1 [main] ERROR org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher - hconnection Received unexpected KeeperException, re-throwing exception org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master at
org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at
org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at
org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeep‌​er.java:154) at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:226)at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracke‌​r.java:82) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setu‌​pZookeeperTrackers(HConnectionManager.java:580)

08/13/2013显然，需要为远程hbase设置的变量不是我正在编辑的变量，根据seqware constants.java文件，它们是qe变量：https://github.com/seqware/seqware/blob/develop/seqware-queryengine/src/main/java/com/github/seqware/queryengine/constants.java
我对它们进行了如下编辑：

SEQWARE QUERY ENGINE AND GENERAL HADOOP SETTINGS

HBASE.ZOOKEEPER.QUORUM=localhost
HBASE.ZOOKEEPER.PROPERTY.CLIENTPORT=2181
HBASE.MASTER=localhost:60000
MAPRED.JOB.TRACKER=localhost:8021
FS.DEFAULT.NAME=hdfs://localhost:8020
FS.DEFAULTFS=hdfs://localhost:8020
FS.HDFS.IMPL=org.apache.hadoop.hdfs.DistributedFileSystem

SEQWARE QUERY ENGINE SETTINGS

QE_NAMESPACE=SeqWareQE
QE_DEVELOPMENT_DEPENDENCY=file:/home/seqware/jars/seqware-distribution-0.13.6.5-qe-full.jar
QE_PERSIST=true
QE_HBASE_REMOTE_TESTING=true
QE_HBASE_PROPERTIES=HBOOT
QE_HBOOT_HBASE_ZOOKEEPER_QUORUM=ip-10-x.x.x.ec2.internal
QE_HBOOT_HBASE_ZOOKEEPER_PROPERTY_CLIENTPORT=2181
QE_HBOOT_HBASE_MASTER=ip-10-x.x.x.ec2.internal:60010
QE_HBOOT_MAPRED_JOB_TRACKER=ip-10-x.x.x.ec2.internal:9001
QE_HBOOT_FS_DEFAULT_NAME=hdfs://ip-10-x.x.x.ec2.internal:9000
QE_HBOOT_FS_DEFAULTFS=hdfs://ip-10-x.x.x.ec2.internal:9000
QE_HBOOT_FS_HDFS_IMPL=org.apache.hadoop.hdfs.DistributedFileSystem

我不再得到zookeeper exceptiona，但是创建工作区的命令只是挂起几分钟，然后我停止它。
我在我的Zookeper日志中发现了这个，我不确定这是否意味着Zookeper崩溃了，或者它与它所声明的客户端失去了连接。我不知道为什么它接受来自端口3699736998、37000和37034的套接字连接，我甚至还没有在ec2安全组上授予它们权限：

2013-08-13 16:44:55,560 INFO org.apache.zookeeper.server.ZooKeeperServer: Established session 0x1407890cb630000 with negotiated timeout 180000 for client /10.x.x.x:36997
2013-08-13 16:44:57,633 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /10.x.x.x:36998
2013-08-13 16:44:57,662 INFO org.apache.zookeeper.server.ZooKeeperServer: Client attempting to establish new session at /10.x.x.x:36998
2013-08-13 16:44:57,666 INFO org.apache.zookeeper.server.ZooKeeperServer: Established session 0x1407890cb630001 with negotiated timeout 180000 for client /10.x.x.x:36998
2013-08-13 16:44:57,917 INFO org.apache.zookeeper.server.PrepRequestProcessor: Got user-level KeeperException when processing sessionid:0x1407890cb630001 type:create cxid:0x8 zxid:0x219 txntype:-1 reqpath:n/a Error Path:/hbase/online-snapshot/acquired Error:KeeperErrorCode = NodeExists for /hbase/online-snapshot/acquired
2013-08-13 16:44:58,450 INFO org.apache.zookeeper.server.PrepRequestProcessor: Got user-level KeeperException when processing sessionid:0x1407890cb630000 type:create cxid:0xb zxid:0x21a txntype:-1 reqpath:n/a Error Path:/hbase/master Error:KeeperErrorCode = NodeExists for /hbase/master
2013-08-13 16:45:00,927 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /10.x.x.x:37000
2013-08-13 16:45:00,928 INFO org.apache.zookeeper.server.ZooKeeperServer: Client attempting to establish new session at /10.x.x.x:37000
2013-08-13 16:45:00,930 INFO org.apache.zookeeper.server.ZooKeeperServer: Established session 0x1407890cb630002 with negotiated timeout 180000 for client /10.x.x.x:37000
2013-08-13 16:45:02,165 INFO org.apache.zookeeper.server.PrepRequestProcessor: Got user-level KeeperException when processing sessionid:0x1407890cb630000 type:create cxid:0x24 zxid:0x221 txntype:-1 reqpath:n/a Error Path:/hbase/online-snapshot/acquired Error:KeeperErrorCode = NodeExists for /hbase/online-snapshot/acquired
2013-08-13 16:45:14,172 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /10.x.x.x:37034
2013-08-13 16:45:14,173 INFO org.apache.zookeeper.server.ZooKeeperServer: Client attempting to establish new session at /10.x.x.x:37034
2013-08-13 16:45:14,178 INFO org.apache.zookeeper.server.ZooKeeperServer: Established session 0x1407890cb630003 with negotiated timeout 180000 for client /10.x.x.x:37034
2013-08-13 16:47:51,000 INFO org.apache.zookeeper.server.ZooKeeperServer: Expiring session 0x1407800784a0003, timeout of 180000ms exceeded
2013-08-13 16:47:51,001 INFO org.apache.zookeeper.server.ZooKeeperServer: Expiring session 0x1407800784a0001, timeout of 180000ms exceeded
2013-08-13 16:47:51,001 INFO org.apache.zookeeper.server.ZooKeeperServer: Expiring session 0x1407800784a0000, timeout of 180000ms exceeded
2013-08-13 16:47:51,001 INFO org.apache.zookeeper.server.ZooKeeperServer: Expiring session 0x1407800784a0002, timeout of 180000ms exceeded
2013-08-13 16:47:51,001 INFO org.apache.zookeeper.server.PrepRequestProcessor: Processed session termination for sessionid: 0x1407800784a0003
2013-08-13 16:47:51,001 INFO org.apache.zookeeper.server.PrepRequestProcessor: Processed session termination for sessionid: 0x1407800784a0001
2013-08-13 16:47:51,001 INFO org.apache.zookeeper.server.PrepRequestProcessor: Processed session termination for sessionid: 0x1407800784a0000
2013-08-13 16:47:51,002 INFO org.apache.zookeeper.server.PrepRequestProcessor: Processed session termination for sessionid: 0x1407800784a0002

我查看了hbase的web界面，它显示表实际上正在创建，但是要创建的命令从来没有返回响应，它们只是挂起。
![](https://i.stack.imgur.com/DAFhF.jpg)
![](https://i.stack.imgur.com/6Ne67.jpg)
hadoophbaseapache-zookeepersshamazon-ec2

1条答案

按热度按时间
3phpmpom

3phpmpom1#

尝试改变，

<property>
   <name>hbase.zookeeper.quorum</name>
   <value>ip-10-x.x.x</value>
</property>

对此，

<property>
   <name>hbase.zookeeper.quorum</name>
   <value>localhost</value>
</property>

然后重新启动hbase和zookeeper
另外，还建议将此作为seqware配置


# SEQWARE QUERY ENGINE AND GENERAL HADOOP SETTINGS

# 

HBASE.ZOOKEEPER.QUORUM=localhost
HBASE.ZOOKEEPER.PROPERTY.CLIENTPORT=2181
HBASE.MASTER=localhost:60000
MAPRED.JOB.TRACKER=localhost:8021
FS.DEFAULT.NAME=hdfs://localhost:8020
FS.DEFAULTFS=hdfs://localhost:8020
FS.HDFS.IMPL=org.apache.hadoop.hdfs.DistributedFileSystem

# 

# SEQWARE QUERY ENGINE SETTINGS

# 

QE_NAMESPACE=SeqWareQE
QE_DEVELOPMENT_DEPENDENCY=file:/home/seqware/jars/seqware-distribution-0.13.6.5-qe-full.jar
QE_PERSIST=true
QE_HBASE_REMOTE_TESTING=true
QE_HBASE_PROPERTIES=HBOOT
QE_HBOOT_HBASE_ZOOKEEPER_QUORUM=ip-10-x.x.x.ec2.internal
QE_HBOOT_HBASE_ZOOKEEPER_PROPERTY_CLIENTPORT=2181
QE_HBOOT_HBASE_MASTER=ip-10-x.x.x.ec2.internal:60010
QE_HBOOT_MAPRED_JOB_TRACKER=ip-10-x.x.x.ec2.internal:9001
QE_HBOOT_FS_DEFAULT_NAME=hdfs://ip-10-x.x.x.ec2.internal:9000
QE_HBOOT_FS_DEFAULTFS=hdfs://ip-10-x.x.x.ec2.internal:9000
QE_HBOOT_FS_HDFS_IMPL=org.apache.hadoop.hdfs.DistributedFileSystem

另外，试试这里和这里的指南。
你还提到了一个很可能没有打开的zookeeper端口。为了测试pupose，我建议您禁用防火墙。因为我之前看到了几个问题，这是因为防火墙阻止了重要的端口。

赞(0）分享  回复(0）举报  2021-06-03
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备2022004421号-2 NULL123 https://www.null123.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com