我正在运行一个简单的spark提交作业,例如:
enter code here spark-submit --class com.x.y.z.logan
/home/test/spark/sample.jar
table in jar file
hiveContext.sql("CREATE TABLE IF NOT EXISTS
databasename.tablename(es_column_name STRING) STORED BY
'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES('es.resource' = 'index_name/log','es.mapping.names'
='tablecoulmname :es_column_name ', 'es.nodes' =
'192.168.x.1y:9200','es.input.json' = 'false',
'es.index.read.missing.as.empty' = 'yes' ,'es.index.auto.create' =
'yes') ")
hiveContext.sql("INSERT INTO TABLE test.incompleterun SELECT
s.streamname FROM incomplete s");
**ERROR**
client token: N/A
diagnostics: User class threw exception: java.lang.RuntimeException:
java.lang.RuntimeException: class
org.elasticsearch.hadoop.mr.EsOutputFormat$EsOutputCommitter not
org.apache.hadoop.mapred.OutputCommitter
ApplicationMaster host: 192.168.x.y
ApplicationMaster RPC port: 0
queue: root.users.test
start time: 1485286033939
final status: FAILED
tracking URL: some URL
user: test
Exception in thread "main" org.apache.spark.SparkException:
Application application_1485258812942_0008 finished with failed status
at org.apache.spark.deploy.yarn.Client.run(Client.scala:1035)
at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1082)
at org.apache.spark.deploy.yarn.Client.main(Client.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
当我们在配置单元接口中创建外部表并将数据从配置单元表加载到外部表时,es hadoop工作正常。当我们在jar中包含相同的查询时,它不起作用。当我们创建普通的配置单元表时,相同的jar文件可以正常工作。这里的问题是当我们在jar文件中包含外部表时出现了下面的错误,有人能帮我解决这个问题吗?
暂无答案!
目前还没有任何答案,快来回答吧!