无法使用es hadoop在elasticsearch中创建外部表

mklgxw1f  于 2021-06-26  发布在  Hive
关注(0)|答案(0)|浏览(362)

我正在运行一个简单的spark提交作业,例如:

enter code here spark-submit --class com.x.y.z.logan 
 /home/test/spark/sample.jar

  table in jar file

   hiveContext.sql("CREATE TABLE IF NOT EXISTS 
   databasename.tablename(es_column_name STRING) STORED BY 
   'org.elasticsearch.hadoop.hive.EsStorageHandler' 
    TBLPROPERTIES('es.resource' = 'index_name/log','es.mapping.names' 
    ='tablecoulmname :es_column_name ', 'es.nodes' = 
     '192.168.x.1y:9200','es.input.json' = 'false', 
      'es.index.read.missing.as.empty' = 'yes' ,'es.index.auto.create' = 
     'yes') ")

      hiveContext.sql("INSERT INTO TABLE test.incompleterun SELECT 
      s.streamname FROM incomplete s");

     **ERROR**
      client token: N/A
       diagnostics: User class threw exception: java.lang.RuntimeException: 
      java.lang.RuntimeException: class 
      org.elasticsearch.hadoop.mr.EsOutputFormat$EsOutputCommitter not 
      org.apache.hadoop.mapred.OutputCommitter
      ApplicationMaster host: 192.168.x.y
      ApplicationMaster RPC port: 0
      queue: root.users.test
      start time: 1485286033939
      final status: FAILED
      tracking URL: some URL
      user: test
       Exception in thread "main" org.apache.spark.SparkException: 
      Application application_1485258812942_0008 finished with failed status
      at org.apache.spark.deploy.yarn.Client.run(Client.scala:1035)
      at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1082)
      at org.apache.spark.deploy.yarn.Client.main(Client.scala)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

当我们在配置单元接口中创建外部表并将数据从配置单元表加载到外部表时,es hadoop工作正常。当我们在jar中包含相同的查询时,它不起作用。当我们创建普通的配置单元表时,相同的jar文件可以正常工作。这里的问题是当我们在jar文件中包含外部表时出现了下面的错误,有人能帮我解决这个问题吗?

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题