无法测试spark的s3支持的hbase

tv6aics1  于 2021-06-08  发布在  Hbase
关注(0)|答案(0)|浏览(263)

我编写了一个从hbase读取数据的简单程序,该程序在cloudera中由hdfs支持。
但是在用s3测试emr数据时出现了一个异常。

// Spark conf
        SparkConf sparkConf = new SparkConf().setMaster("local[4]").setAppName("My App");
        JavaSparkContext jsc = new JavaSparkContext(sparkConf);
        // Hbase conf
        Configuration conf = HBaseConfiguration.create();
        conf.set("hbase.zookeeper.quorum","localhost");
        conf.set("hbase.zookeeper.property.client.port","2181");
        // Submit scan into hbase conf
 //       conf.set(TableInputFormat.SCAN, TableMapReduceUtil.convertScanToString(scan));

        conf.set(TableInputFormat.INPUT_TABLE, "mytable");
        conf.set(TableInputFormat.SCAN_ROW_START, "startrow");
        conf.set(TableInputFormat.SCAN_ROW_STOP, "endrow");

        // Get RDD
        JavaPairRDD<ImmutableBytesWritable, Result> source = jsc
                .newAPIHadoopRDD(conf, TableInputFormat.class,
                        ImmutableBytesWritable.class, Result.class);

        // Process RDD
        System.out.println("&&&&&&&&&&&&&&&&&&&&&&& " + source.count());

18/05/04 00:22:02信息metricregistries:加载的metricregistries类org.apache.hadoop.hbase.metrics.impl.metricregistriesimpl 18/05/04 00:22:02错误tableinputformat:java.io.ioexception:java.lang.reflect.invocationtargetexception位于org.apache.hadoop.hbase.client.connectionfactory.createconnection(connectionfactory)。java:240)原因:sun.reflect.nativeconstructoraccessorimpl.newinstance0(本机方法)位于sun.reflect.nativeconstructoraccessorimpl.newinstance(nativeconstructoraccessorimpl)。java:62)在sun.reflect.delegatingconstructoraccessorimpl.newinstance(delegatingconstructoraccessorimpl。java:45)在java.lang.reflect.constructor.newinstance(构造函数。java:423)在org.apache.hadoop.hbase.client.connectionfactory.createconnection(connectionfactory。java:238)原因:java.lang.illegalaccesserror:尝试从org.apache.hadoop.metrics2.lib.metricsinfoimpl类访问org.apache.hadoop.metrics2.lib.dynamicmetricsregistryorg.apache.hadoop.metrics2.lib.dynamicmetricsregistry.newgauge(dynamicmetricsregistry)。java:139)在org.apache.hadoop.hbase.zookeeper.metricszookeepersourceimpl。java:59)在org.apache.hadoop.hbase.zookeeper.metricszookeepersourceimpl。java:51)在sun.reflect.nativeconstructoraccessorimpl.newinstance0(本机方法)位于sun.reflect.nativeconstructoraccessorimpl.newinstance(nativeconstructoraccessorimpl)。java:62)在sun.reflect.delegatingconstructoraccessorimpl.newinstance(delegatingconstructoraccessorimpl。java:45)在java.lang.reflect.constructor.newinstance(constructor。java:423)在java.lang.class.newinstance(类。java:442)在java.util.serviceloader$lazyiterator.nextservice(serviceloader。java:380) ... 42个以上

Exception in thread "main" java.io.IOException: Cannot create a record reader because of a previous error. Please look at the previous

记录任务完整日志中的行以获取更多详细信息。位于org.apache.hadoop.hbase.mapreduce.tableinputformatbase.getsplits(tableinputformatbase)。java:270)位于org.apache.hadoop.hbase.mapreduce.tableinputformat.getsplits(tableinputformat)。java:256)在org.apache.spark.rdd.newhadooprdd.getpartitions(newhadooprdd。scala:125)在org.apache.spark.rdd.rdd$$anonfun$partitions$2.apply(rdd。scala:252)在org.apache.spark.rdd.rdd$$anonfun$分区$2.apply(rdd。scala:250)在scala.option.getorelse(option。scala:121)在org.apache.spark.rdd.rdd.partitions(rdd。scala:250)在org.apache.spark.sparkcontext.runjob(sparkcontext。scala:2094)在org.apache.spark.rdd.rdd.count(rdd。scala:1158)在org.apache.spark.api.java.javarddlike$class.count(javarddlike。scala:455)在org.apache.spark.api.java.abstractjavarddlike.count(javarddlike。scala:45)在hbascan.main(hbascan。java:60)在sun.reflect.nativemethodaccessorimpl.invoke0(本机方法)在sun.reflect.nativemethodaccessorimpl.invoke(nativemethodaccessorimpl)。java:62)在sun.reflect.delegatingmethodaccessorimpl.invoke(delegatingmethodaccessorimpl。java:43)在java.lang.reflect.method.invoke(方法。java:498)在org.apache.spark.deploy.sparksubmit$.org$apache$spark$deploy$sparksubmit$$runmain(sparksubmit)。scala:775)在org.apache.spark.deploy.sparksubmit$.dorunmain$1(sparksubmit。scala:180)在org.apache.spark.deploy.sparksubmit$.submit(sparksubmit。scala:205)位于org.apache.spark.deploy.sparksubmit$.main(sparksubmit.com)。scala:119)在org.apache.spark.deploy.sparksubmit.main(sparksubmit.scala)上,由于:java.lang.illegalstateexception:输入格式示例未正确初始化。确保在构造函数中调用initializetable,或者在org.apache.hadoop.hbase.mapreduce.tableinputformatbase.gettable(tableinputformatbase)中调用initializetable方法。java:652)位于org.apache.hadoop.hbase.mapreduce.tableinputformatbase.getsplits(tableinputformatbase)。java:265) ... 所有apache hbase库还有20个:e.hadoop.hbase.metrics.impl.metricregistriesimpl 18/05/04 04 04:05:54 error tableinputformat:java.io.ioexception:java.lang.reflect.invocationtargetexception位于org.apache.hadoop.hbase.client.connectionfactory.createconnection(connectionfactory)。java:240)在org.apache.hadoop.hbase.client.connectionfactory.createconnection(connectionfactory。java:218)在org.apache.hadoop.hbase.client.connectionfactory.createconnection(connectionfactory。java:119)位于org.apache.hadoop.hbase.mapreduce.tableinputformat.initialize(tableinputformat)。java:202)在org.apache.hadoop.hbase.mapreduce.tableinputformatbase.getsplits(tableinputformatbase)。java:259)位于org.apache.hadoop.hbase.mapreduce.tableinputformat.getsplits(tableinputformat)。java:256)在org.apache.spark.rdd.newhadooprdd.getpartitions(newhadooprdd。scala:125)在org.apache.spark.rdd.rdd$$anonfun$partitions$2.apply(rdd。scala:252)在org.apache.spark.rdd.rdd$$anonfun$分区$2.apply(rdd。scala:250)在scala.option.getorelse(option。scala:121)在org.apache.spark.rdd.rdd.partitions(rdd。scala:250)在org.apache.spark.sparkcontext.runjob(sparkcontext。scala:2094)在org.apache.spark.rdd.rdd.count(rdd。scala:1158)在org.apache.spark.api.java.javarddlike$class.count(javarddlike。scala:455)在org.apache.spark.api.java.abstractjavarddlike.count(javarddlike。scala:45)在hbascan.main(hbascan。java:60)在sun.reflect.nativemethodaccessorimpl.invoke0(本机方法)在sun.reflect.nativemethodaccessorimpl.invoke(nativemethodaccessorimpl)。java:62)在sun.reflect.delegatingmethodaccessorimpl.invoke(delegatingmethodaccessorimpl。java:43)在java.lang.reflect.method.invoke(方法。java:498)在org.apache.spark.deploy.sparksubmit$.org$apache$spark$deploy$sparksubmit$$runmain(sparksubmit)。scala:775)在org.apache.spark.deploy.sparksubmit$.dorunmain$1(sparksubmit。scala:180)在org.apache.spark.deploy.sparksubmit$.submit(sparksubmit。scala:205)位于org.apache.spark.deploy.sparksubmit$.main(sparksubmit.com)。scala:119)在org.apache.spark.deploy.sparksubmit.main(sparksubmit.scala)上,由以下原因引起:java.lang.reflect.invocationtargetexception at sun.reflect.nativeconstructoraccessorimpl.newinstance0(本机方法)atsun.reflect.nativeconstructoraccessorimpl.newinstance(nativeconstructoraccessorimpl。java:62)在sun.reflect.delegatingconstructoraccessorimpl.newinstance(delegatingconstructoraccessorimpl。java:45)在java.lang.reflect.constructor.newinstance(constructor。java:423)在org.apache.hadoop.hbase.client.connectionfactory.createconnection(connectionfactory。java:238) ... 24更多原因:java.lang.runtimeexception:无法创建接口org.apache.hadoop.hbase.zookeeper.metricszookeepersource是类路径上的hadoop兼容性jar吗?位于org.apache.hadoop.hbase.compatibilitysingletonfactory.getinstance(compatibilitysingletonfactory)。java:75)在org.apache.hadoop.hbase.zookeeper.metricszookeeper。java:38)在org.apache.hadoop.hbase.zookeeper.recoverablezookeeper。java:130)在org.apache.hadoop.hbase.zookeeper.zkutil.connect(zkutil。java:143) 在org.apache.hadoop.hbase.zookeeper.zookeeperwatcher。java:181)在org.apache.hadoop.hbase.zookeeper.zookeeperwatcher。java:155)在org.apache.hadoop.hbase.client.zookeeperkeepaliveconnection.(zookeeperkeepaliveconnection。java:43)在org.apache.hadoop.hbase.client.connectionmanager$hconnectionimplementation.getkeepalivezookeeperwatcher(connectionmanager)。java:1737)

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题