cloudera quick start vm cdh5.8.0中的hbase scala连接问题

anhgbhbe  于 2021-06-09  发布在  Hbase
关注(0)|答案(1)|浏览(473)

我试图从scala代码连接hbase，但出现以下错误。

17/03/28 11:40:53 INFO client.RpcRetryingCaller: Call exception, tries=30, retries=35, started=450502 ms ago, cancelled=false, msg=  
17/03/28 11:41:13 INFO client.RpcRetryingCaller: Call exception, tries=31, retries=35, started=470659 ms ago, cancelled=false, msg=  
17/03/28 11:41:33 INFO client.RpcRetryingCaller: Call exception, tries=32, retries=35, started=490824 ms ago, cancelled=false, msg=  
17/03/28 11:41:53 INFO client.RpcRetryingCaller: Call exception, tries=33, retries=35, started=510834 ms ago, cancelled=false, msg=  
17/03/28 11:42:13 INFO client.RpcRetryingCaller: Call exception, tries=34, retries=35, started=530956 ms ago, cancelled=false, msg=  
[error] (run-main-0) org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions:  
[error] Tue Mar 28 11:33:22 PDT 2017, RpcRetryingCaller{globalStartTime=1490726002560, pause=100, retries=35}, org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: java.lang.NoClassDefFoundError: org/apache/hadoop/net/SocketInputWrapper  
[error] Tue Mar 28 11:33:23 PDT 2017, RpcRetryingCaller{globalStartTime=1490726002560, pause=100, retries=35}, org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: java.lang.NoClassDefFoundError: org/apache/hadoop/net/SocketInputWrapper  
[error] Tue Mar 28 11:33:23 PDT 2017, RpcRetryingCaller{globalStartTime=1490726002560, pause=100, retries=35}, org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: java.lang.NoClassDefFoundError: org/apache/hadoop/net/SocketInputWrapper  
[error] Tue Mar 28 11:33:24 PDT 2017, RpcRetryingCaller{globalStartTime=1490726002560, pause=100, retries=35}, org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: java.lang.NoClassDefFoundError: org/apache/hadoop/net/SocketInputWrapper  
.  
.  
.  
.  
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:147)
    at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4117)
    at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4110)
    at org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:427)
    at org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:411)
    at Hi$.main(hw.scala:12)
    at Hi.main(hw.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
Caused by: org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: java.lang.NoClassDefFoundError: org/apache/hadoop/net/SocketInputWrapper
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1560)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1580)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1737)
    at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38)
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:124)
    at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4117)
    at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4110)
    at org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:427)
    at org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:411)
    at Hi$.main(hw.scala:12)
    at Hi.main(hw.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
Caused by: com.google.protobuf.ServiceException: java.lang.NoClassDefFoundError: org/apache/hadoop/net/SocketInputWrapper
    at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:239)
    at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:331)
    at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:58383)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.isMasterRunning(ConnectionManager.java:1591)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1529)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1551)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1580)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1737)
    at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38)
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:124)
    at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4117)
    at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4110)
    at org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:427)
    at org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:411)
    at Hi$.main(hw.scala:12)
    at Hi.main(hw.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/net/SocketInputWrapper
    at org.apache.hadoop.hbase.ipc.RpcClientImpl.createConnection(RpcClientImpl.java:138)
    at org.apache.hadoop.hbase.ipc.RpcClientImpl.getConnection(RpcClientImpl.java:1316)
    at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1224)
    at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:226)
    at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:331)
    at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:58383)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.isMasterRunning(ConnectionManager.java:1591)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1529)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1551)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1580)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1737)
    at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38)
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:124)
    at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4117)
    at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4110)
    at org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:427)
    at org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:411)
    at Hi$.main(hw.scala:12)
    at Hi.main(hw.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.net.SocketInputWrapper
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    at org.apache.hadoop.hbase.ipc.RpcClientImpl.createConnection(RpcClientImpl.java:138)
    at org.apache.hadoop.hbase.ipc.RpcClientImpl.getConnection(RpcClientImpl.java:1316)
    at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1224)
    at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:226)
    at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:331)
    at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:58383)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.isMasterRunning(ConnectionManager.java:1591)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1529)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1551)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1580)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1737)
    at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38)
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:124)
    at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4117)
    at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4110)
    at org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:427)
    at org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:411)
    at Hi$.main(hw.scala:12)
    at Hi.main(hw.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
[trace] Stack trace suppressed: run last compile:run for the full output.
17/03/28 07:56:55 ERROR zookeeper.ClientCnxn: Event thread exiting due to interruption
java.lang.InterruptedException
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2052)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
17/03/28 07:56:55 INFO zookeeper.ClientCnxn: EventThread shut down
java.lang.RuntimeException: Nonzero exit code: 1
    at scala.sys.package$.error(package.scala:27)
[trace] Stack trace suppressed: run last compile:run for the full output.
[error] (compile:run) Nonzero exit code: 1
[error] Total time: 544 s, completed Mar 28, 2017 7:56:56 AM

•主机操作系统为windows 7，具有8 gb ram和64位arch。英特尔酷睿i5。
•我正在使用cloudera quick start vm cdh 5.8.0。在我的床上。
•虚拟机使用6gb ram、2个处理器和64 gb硬盘。
•在cloudera manager中运行的服务：

Hbase  
    HDFS  
    YARN  
    Zookeeper  
    Key-Value Indexer

•在cloudera manager中停止的服务：

Hive
    Hue
    Impala
    Oozie
    Solar
    Spark
    Sqoop 1 Client
    Sqoop 2

•hbase版本1.2.0-cdh5.8.0
•我的客户端代码仅在虚拟机中。
•创建sbt项目。
•我引用了此urlhttps://hbase.apache.org/book.html#scala 用于与scala的hbase连接。
•设置类路径。我没有提到链接中提到的类路径中的“/path/to/scala library.jar”。

$ export CLASSPATH=$CLASSPATH:/usr/lib/hadoop/lib/native:/usr/lib/hbase/lib/native/Linux-amd64-64

•project root directory=/home/cloudera/desktop/play sbt项目
•my/home/cloudera/desktop/play sbt project/build.sbt如下所示。我根据我的环境更改了依赖库的版本。作为错误排除的一部分，我添加了更多的依赖项，如“hbase client”、“hbase common”和“hbase server”，但仍然没有成功。

name := "play-sbt-project"
version := "1.0"
scalaVersion := "2.10.2"
resolvers += "Apache HBase" at "https://repository.apache.org/content/repositories/releases"
resolvers += "Thrift" at "http://people.apache.org/~rawson/repo/"
libraryDependencies ++= Seq(
"org.apache.hadoop" % "hadoop-core" % "1.2.1",
"org.apache.hbase" % "hbase" % "1.2.0",
"org.apache.hbase" % "hbase-client" % "1.2.0",
"org.apache.hbase" % "hbase-common" % "1.2.0",
"org.apache.hbase" % "hbase-server" % "1.2.0"
)

•我的hbase connectivity/home/cloudera/desktop/play sbt project/src/main/scala/pw.scala主代码如下

import org.apache.hadoop.hbase.HBaseConfiguration
import org.apache.hadoop.hbase.client.{ConnectionFactory,HBaseAdmin,HTable,Put,Get}
import org.apache.hadoop.hbase.util.Bytes

object Hi {
def main(args: Array[String]) = {
println("Hi!")
val conf = new HBaseConfiguration()
val connection = ConnectionFactory.createConnection(conf);
val admin = connection.getAdmin();

// list the tables
val listtables=admin.listTables()
listtables.foreach(println)
}
}

•my/etc/hbase/conf/hbase-site.xml如下所示：

<?xml version="1.0" encoding="UTF-8"?>

<!--Autogenerated by Cloudera Manager-->
<configuration>
  <property>
    <name>hbase.rootdir</name>
    <value>hdfs://quickstart.cloudera:8020/hbase</value>
  </property>
  <property>
    <name>hbase.replication</name>
    <value>true</value>
  </property>
  <property>
    <name>hbase.client.write.buffer</name>
    <value>2097152</value>
  </property>
  <property>
    <name>hbase.client.pause</name>
    <value>100</value>
  </property>
  <property>
    <name>hbase.client.retries.number</name>
    <value>35</value>
  </property>
  <property>
    <name>hbase.client.scanner.caching</name>
    <value>100</value>
  </property>
  <property>
    <name>hbase.client.keyvalue.maxsize</name>
    <value>10485760</value>
  </property>
  <property>
    <name>hbase.ipc.client.allowsInterrupt</name>
    <value>true</value>
  </property>
  <property>
    <name>hbase.client.primaryCallTimeout.get</name>
    <value>10</value>
  </property>
  <property>
    <name>hbase.client.primaryCallTimeout.multiget</name>
    <value>10</value>
  </property>
  <property>
    <name>hbase.coprocessor.region.classes</name>
    <value>org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint</value>
  </property>
  <property>
    <name>hbase.regionserver.thrift.http</name>
    <value>false</value>
  </property>
  <property>
    <name>hbase.thrift.support.proxyuser</name>
    <value>false</value>
  </property>
  <property>
    <name>hbase.rpc.timeout</name>
    <value>60000</value>
  </property>
  <property>
    <name>hbase.snapshot.enabled</name>
    <value>true</value>
  </property>
  <property>
    <name>hbase.snapshot.master.timeoutMillis</name>
    <value>60000</value>
  </property>
  <property>
    <name>hbase.snapshot.region.timeout</name>
    <value>60000</value>
  </property>
  <property>
    <name>hbase.snapshot.master.timeout.millis</name>
    <value>60000</value>
  </property>
  <property>
    <name>hbase.security.authentication</name>
    <value>simple</value>
  </property>
  <property>
    <name>hbase.rpc.protection</name>
    <value>authentication</value>
  </property>
  <property>
    <name>zookeeper.session.timeout</name>
    <value>60000</value>
  </property>
  <property>
    <name>zookeeper.znode.parent</name>
    <value>/hbase</value>
  </property>
  <property>
    <name>zookeeper.znode.rootserver</name>
    <value>root-region-server</value>
  </property>
  <property>
    <name>hbase.zookeeper.quorum</name>
    <!-- <value>quickstart.cloudera</value> -->
    <value>127.0.0.1</value>
  </property>
  <property>
    <name>hbase.zookeeper.property.clientPort</name>
    <value>2181</value>
  </property>
  <property>
    <name>hbase.rest.ssl.enabled</name>
    <value>false</value>
  </property>
</configuration>

我在谷歌上搜索了很多来解决这个问题，但没有成功。在解决这个问题的过程中，我做了以下改变：
•根据我的环境更改build.sbt文件中的依赖库版本
•添加了几个更依赖的库“hbase client”、“hbase common”和“hbase server”。
•将“hbase site.xml”文件中的“hbase.zookeeper.quorum”值从“quickstart.cloudera”更改为“127.0.0.1”。
请帮我解决这个问题。谢谢您。

hbasescalaapache-zookeepercloudera-cdhcloudera-quickstart-vm

1条答案

按热度按时间
qeeaahzv

qeeaahzv1#

解决了问题。需要进行以下更改：
将build.sbt文件中的“hadoop core”改为“hadoop common”。因为在最新的cdh版本中，只有运行MapReduce1的代码才支持“hadoop core”。
根据build.sbt中的cloudera 5.8.0兼容性更改所有依赖版本。更新的build.sbt如下所示：

name := "play-sbt-project"  
version := "1.0"  
scalaVersion := "2.10.2"  
resolvers += "Thrift" at "http://people.apache.org/~rawson/repo/"  
resolvers += "Cloudera Repository" at "https://repository.cloudera.com/artifactory/cloudera-repos/"  

libraryDependencies ++= Seq(  
 "org.apache.hadoop" % "hadoop-common" % "2.6.0-cdh5.8.0",  
 "org.apache.hbase" % "hbase" % "1.2.0-cdh5.8.0",  
 "org.apache.hbase" % "hbase-client" % "1.2.0-cdh5.8.0",  
 "org.apache.hbase" % "hbase-common" % "1.2.0-cdh5.8.0",  
 "org.apache.hbase" % "hbase-server" % "1.2.0-cdh5.8.0"  
)

hbaseconfiguration（）类已被清除。而是使用create（）方法。我还修改了主代码中的一些逻辑。早些时候，我在hbase中得到了表（因为这会带来一些问题，所以我放弃了这个，但我下次会尝试），因为我的moto是建立scala到hbase的连接，所以现在我正在尝试向已经存在的hbase表插入新行。新代码如下所示：

package main.scala

import org.apache.hadoop.conf.Configuration  
import org.apache.hadoop.hbase.HBaseConfiguration  
import org.apache.hadoop.hbase.client.{ConnectionFactory,HTable,Put}  
import org.apache.hadoop.hbase.util.Bytes  

object Hi {  

 def main(args: Array[String]) = {  
 println("Hi!")  
 val conf:Configuration = HBaseConfiguration.create()  
 val table:HTable = new HTable(conf, "emp1")  
 val put1:Put = new Put(Bytes.toBytes("row1"))  
 put1.add(Bytes.toBytes("personal_data"),Bytes.toBytes("qual1"),Bytes.toBytes("val1"))  
 table.put(put1)  
 println("Success")  
 }  
}
赞(0）分享  回复(0）举报  2021-06-10
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备2022004421号-2 NULL123 https://www.null123.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com