通过kerberos的spark/phoenix(java)

mo49yndu  于 2021-05-29  发布在  Spark
关注(0)|答案(0)|浏览(292)

我正在用java编写有关计算任务的程序,该程序要求我通过kerberos将数据流从hive传输到phoenix,但是当我的作业成功运行时,我的程序无法工作,有关我的任务的信息如下:

20/06/15 10:15:20 WARN TaskSetManager: Lost task 0.0 in stage 5.0 (TID 66, node8.hde.h3c.com, executor 1): java.sql.SQLException: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=36, exceptions:
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO - Mon Jun 15 10:15:20 CST 2020, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=68453: row 'SYSTEM:CATALOG,,' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=node6.hde.h3c.com,16020,1591602784717, seqNum=0
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO - 
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call(ConnectionQueryServicesImpl.java:2467)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call(ConnectionQueryServicesImpl.java:2382)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.java:76)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:2382)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:255)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.createConnection(PhoenixEmbeddedDriver.java:149)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:221)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at java.sql.DriverManager.getConnection(DriverManager.java:664)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at java.sql.DriverManager.getConnection(DriverManager.java:208)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.phoenix.mapreduce.util.ConnectionUtil.getConnection(ConnectionUtil.java:98)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.phoenix.mapreduce.util.ConnectionUtil.getOutputConnection(ConnectionUtil.java:82)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.phoenix.mapreduce.util.ConnectionUtil.getOutputConnection(ConnectionUtil.java:70)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.phoenix.mapreduce.util.PhoenixConfigurationUtil.getUpsertColumnMetadataList(PhoenixConfigurationUtil.java:247)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.phoenix.spark.DataFrameFunctions$$anonfun$2.apply(DataFrameFunctions.scala:46)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.phoenix.spark.DataFrameFunctions$$anonfun$2.apply(DataFrameFunctions.scala:42)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:797)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:797)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.spark.scheduler.Task.run(Task.scala:99)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at java.lang.Thread.run(Thread.java:745)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO - Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=36, exceptions:
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO - Mon Jun 15 10:15:20 CST 2020, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=68453: row 'SYSTEM:CATALOG,,' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=node6.hde.h3c.com,16020,1591602784717, seqNum=0
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO - 
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:271)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:195)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:59)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:155)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:821)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.hadoop.hbase.MetaTableAccessor.fullScan(MetaTableAccessor.java:602)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.hadoop.hbase.MetaTableAccessor.tableExists(MetaTableAccessor.java:366)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:304)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call(ConnectionQueryServicesImpl.java:2413)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     ... 25 more
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO - Caused by: java.net.SocketTimeoutException: callTimeout=60000, callDuration=68453: row 'SYSTEM:CATALOG,,' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=node6.hde.h3c.com,16020,1591602784717, seqNum=0
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:159)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:64)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     ... 3 more
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO - Caused by: java.io.IOException: 断开的管道
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at sun.nio.ch.IOUtil.write(IOUtil.java:65)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:63)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:159)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:117)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at java.io.DataOutputStream.flush(DataOutputStream.java:123)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.hadoop.hbase.ipc.IPCUtil.write(IPCUtil.java:273)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.hadoop.hbase.ipc.IPCUtil.write(IPCUtil.java:261)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:901)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:856)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1200)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:32651)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:372)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:199)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:346)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:320)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
15-06-2020 10:15:20 CST hiveDataToPhoenixAndHive INFO -     ... 4 more

我想phoenix数据库连接应该有问题,但我不知道如何解决这个问题,希望能得到您真诚的帮助

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题