无法使用spark从scala ide连接到配置单元

5rgfhyps  于 2021-06-26  发布在  Hive
关注(0)|答案(0)|浏览(243)

这是我的代码和pom.xml，任何一个错误都能找出确切的原因。
代码：

def main(args:Array[String]){

    val objConf = new SparkConf().setAppName("Spark Connection").setMaster("spark://10.40.10.80:7077")
    var sc = new SparkContext(objConf)
val objHiveContext = new HiveContext(sc)
objHiveContext.sql("USE test")
var test= objHiveContext.sql("show tables")
    var i  =0

    var testing = test.collect()

      for(i<-0 until testing.length){

      println(testing(i))
    }

pom.xml文件：

<dependencies>
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-core_2.10</artifactId>
      <version>1.5.2</version>
    </dependency> 
    <dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-sql_2.10</artifactId>
    <version>1.5.2</version>
</dependency>
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-catalyst_2.10</artifactId>
    <version>1.5.2</version>
</dependency>

     <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-hive_2.10</artifactId>
        <version>1.5.2</version>
    </dependency>

  <dependency>
    <groupId>org.apache.hive</groupId>
    <artifactId>hive-jdbc</artifactId>
    <version>1.2.1</version>
</dependency>

    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>3.8.1</version>
      <scope>test</scope>
    </dependency>     

  </dependencies>

错误控制台：
使用spark的默认log4j配置文件：org/apache/spark/log4j-defaults.properties

16/01/22 18:05:37 INFO SparkContext: Running Spark version 1.5.2
16/01/22 18:05:38 INFO SecurityManager: Changing view acls to: psudhir
16/01/22 18:05:38 INFO SecurityManager: Changing modify acls to: psudhir
16/01/22 18:05:38 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(psudhir); users with modify permissions: Set(psudhir)
16/01/22 18:05:41 INFO Slf4jLogger: Slf4jLogger started
16/01/22 18:05:41 INFO Remoting: Starting remoting
16/01/22 18:05:41 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@172.16.101.215:64657]
16/01/22 18:05:41 INFO Utils: Successfully started service 'sparkDriver' on port 64657.
16/01/22 18:05:41 INFO SparkEnv: Registering MapOutputTracker
16/01/22 18:05:41 INFO SparkEnv: Registering BlockManagerMaster
16/01/22 18:05:42 INFO DiskBlockManager: Created local directory at C:\Users\psudhir\AppData\Local\Temp\blockmgr-2ed0d89c-f370-47bb-8e99-181212bff9c4
16/01/22 18:05:42 INFO MemoryStore: MemoryStore started with capacity 245.7 MB
16/01/22 18:05:42 INFO HttpFileServer: HTTP File server directory is C:\Users\psudhir\AppData\Local\Temp\spark-fddc211b-840f-4a5d-927e-2c7b5c96783b\httpd-44ceeedb-7e5e-430d-bc4d-d35f6e676703
16/01/22 18:05:42 INFO HttpServer: Starting HTTP Server
16/01/22 18:05:42 INFO Utils: Successfully started service 'HTTP file server' on port 64658.
16/01/22 18:05:42 INFO SparkEnv: Registering OutputCommitCoordinator
16/01/22 18:05:43 INFO Utils: Successfully started service 'SparkUI' on port 4040.
16/01/22 18:05:43 INFO SparkUI: Started SparkUI at http://172.16.101.215:4040
16/01/22 18:05:43 WARN MetricsSystem: Using default name DAGScheduler for source because spark.app.id is not set.
16/01/22 18:05:43 INFO AppClient$ClientEndpoint: Connecting to master spark://10.40.10.80:7077...
16/01/22 18:05:45 INFO SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20160122063401-0088
16/01/22 18:05:45 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 64679.
16/01/22 18:05:45 INFO NettyBlockTransferService: Server created on 64679
16/01/22 18:05:45 INFO BlockManagerMaster: Trying to register BlockManager
16/01/22 18:05:45 INFO BlockManagerMasterEndpoint: Registering block manager 172.16.101.215:64679 with 245.7 MB RAM, BlockManagerId(driver, 172.16.101.215, 64679)
16/01/22 18:05:45 INFO BlockManagerMaster: Registered BlockManager
16/01/22 18:05:46 INFO SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
16/01/22 18:05:47 INFO HiveContext: Initializing execution hive, version 1.2.1
16/01/22 18:05:47 INFO ClientWrapper: Inspected Hadoop version: 2.2.0
16/01/22 18:05:47 INFO ClientWrapper: Loaded org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.2.0
16/01/22 18:05:47 INFO deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
16/01/22 18:05:47 INFO deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
16/01/22 18:05:47 INFO deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
16/01/22 18:05:47 INFO deprecation: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node
16/01/22 18:05:47 INFO deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
16/01/22 18:05:47 INFO deprecation: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack
16/01/22 18:05:47 INFO deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
16/01/22 18:05:47 INFO deprecation: mapred.committer.job.setup.cleanup.needed is deprecated. Instead, use mapreduce.job.committer.setup.cleanup.needed
16/01/22 18:05:48 INFO HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
16/01/22 18:05:48 INFO ObjectStore: ObjectStore, initialize called
16/01/22 18:05:48 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored
16/01/22 18:05:48 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
16/01/22 18:06:00 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
16/01/22 18:06:01 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
16/01/22 18:06:01 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
16/01/22 18:06:06 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
16/01/22 18:06:06 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
16/01/22 18:06:08 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
16/01/22 18:06:08 INFO ObjectStore: Initialized ObjectStore
16/01/22 18:06:08 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
16/01/22 18:06:08 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
16/01/22 18:06:09 WARN : Your hostname, psudhir resolves to a loopback/non-reachable address: fe80:0:0:0:0:5efe:ac10:65d7%13, but we couldn't find any external IP address!
16/01/22 18:06:10 INFO HiveMetaStore: Added admin role in metastore
16/01/22 18:06:10 INFO HiveMetaStore: Added public role in metastore
16/01/22 18:06:11 INFO HiveMetaStore: No user is added in admin role, since config is empty
16/01/22 18:06:11 INFO HiveMetaStore: 0: get_all_databases
16/01/22 18:06:11 INFO audit: ugi=psudhir   ip=unknown-ip-addr  cmd=get_all_databases   
16/01/22 18:06:11 INFO HiveMetaStore: 0: get_functions: db=default pat=*
16/01/22 18:06:11 INFO audit: ugi=psudhir   ip=unknown-ip-addr  cmd=get_functions: db=default pat=* 
16/01/22 18:06:11 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as "embedded-only" so does not have its own datastore table.
16/01/22 18:06:15 INFO SessionState: Created local directory: C:/Users/psudhir/AppData/Local/Temp/eb66b3f1-a276-4dd4-8951-90c59d10e3ee_resources
16/01/22 18:06:15 INFO SessionState: Created HDFS directory: /tmp/hive/psudhir/eb66b3f1-a276-4dd4-8951-90c59d10e3ee
16/01/22 18:06:15 INFO SessionState: Created local directory: C:/Users/psudhir/AppData/Local/Temp/psudhir/eb66b3f1-a276-4dd4-8951-90c59d10e3ee
16/01/22 18:06:15 INFO SessionState: Created HDFS directory: /tmp/hive/psudhir/eb66b3f1-a276-4dd4-8951-90c59d10e3ee/_tmp_space.db
16/01/22 18:06:16 INFO HiveContext: default warehouse location is /user/hive/warehouse
16/01/22 18:06:16 INFO HiveContext: Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.
16/01/22 18:06:16 INFO ClientWrapper: Inspected Hadoop version: 2.2.0
16/01/22 18:06:16 INFO ClientWrapper: Loaded org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.2.0
16/01/22 18:06:16 INFO deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
16/01/22 18:06:16 INFO deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
16/01/22 18:06:16 INFO deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
16/01/22 18:06:16 INFO deprecation: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node
16/01/22 18:06:16 INFO deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
16/01/22 18:06:16 INFO deprecation: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack
16/01/22 18:06:16 INFO deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
16/01/22 18:06:16 INFO deprecation: mapred.committer.job.setup.cleanup.needed is deprecated. Instead, use mapreduce.job.committer.setup.cleanup.needed
16/01/22 18:06:16 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/01/22 18:06:17 INFO HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
16/01/22 18:06:17 INFO ObjectStore: ObjectStore, initialize called
16/01/22 18:06:17 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored
16/01/22 18:06:17 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
Exception in thread "main"
Hiveapache-sparkapache-spark-sqlHiveContext

暂无答案！

目前还没有任何答案，快来回答吧！

我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备2022004421号-2 NULL123 https://www.null123.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com