我正试图从我的eclipseide中执行一个正常工作的代码,而我却面临着无法处理的奇怪错误。试图总结我的问题:
用eclipse执行代码:一切正常。
捕获eclipse抛出的命令行来运行我的应用程序,并将其复制到shell中:一切正常。
现在,eclipse生成的运行我的应用程序的命令行 java -cp lots-of-jars -Dvm.params myPackage.MyMainClass app-params
.
我的目标是用oozie作为java操作来执行我的应用程序,所以我需要构建一个uberjar来减少myapp.jar中的大量jar。
为此,我将maven shade插件配置为:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.4.2</version>
<configuration>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>reference.conf</resource>
</transformer>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>es.mycompany.bigdata.OozieAction</mainClass>
</transformer>
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
<transformer implementation="org.apache.maven.plugins.shade.resource.PluginXmlResourceTransformer" />
</transformers>
</configuration>
</execution>
</executions>
</plugin>
我不得不添加一些变压器,因为我在启动我的应用程序时遇到了一些错误(不能创建fsshell spring对象,不能启动sparkcontext…),顺便说一下,我的应用程序的目的是下载一些azure blob,然后放入hdfs,用spark转换它们,最后添加到一个配置单元表。我用java开发了这个应用程序(包括spark部分),并使用spring来实现。
现在,我的上一个新问题发生在我尝试创建hivecontext时(我的spark上下文是正常的,因为如果我省略了hive部分,我的应用程序就会工作):
@Bean
@Lazy
@Scope("singleton")
public SQLContext getSQLContext(@Autowired JavaSparkContext sparkContext) {
return new HiveContext(sparkContext);
}
引发的错误是:
2017-04-02 20:20:18 WARN Persistence:106 - Error creating validator of type org.datanucleus.properties.CorePropertyValidator
ClassLoaderResolver for class "" gave error on creation : {1}
org.datanucleus.exceptions.NucleusUserException: ClassLoaderResolver for class "" gave error on creation : {1}
...
Caused by: org.datanucleus.exceptions.NucleusUserException: Persistence process has been specified to use a ClassLoaderResolver of name "datanucleus" yet this has not been found by the DataNucleus plugin mechanism. Please check your CLASSPATH and plugin specification.
at org.datanucleus.NucleusContext.<init>(NucleusContext.java:283)
at org.datanucleus.NucleusContext.<init>(NucleusContext.java:247)
at org.datanucleus.NucleusContext.<init>(NucleusContext.java:225)
at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.<init>(JDOPersistenceManagerFactory.java:416)
at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:301)
at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:202)
... 93 more
2017-04-02 20:20:18 WARN ExtendedAnnotationApplicationContext:550 - Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'getOozieJavaAction': Unsatisfied dependency expressed through field 'sqlContext'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'getSQLContext' defined in es.mediaset.technology.bigdata.config.FlatJsonToCsvAppConfig: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.spark.sql.SQLContext]: Factory method 'getSQLContext' threw exception; nested exception is java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
因为我的代码正确地运行在eclipse中,并使用这样的命令运行在eclipse之外
/usr/java/jdk1.8.0_121/bin/java -Dmode=responsive -Dspark.master=local[*] -Dfile.encoding=UTF-8 -classpath /home/cloudera/workspace-sts/oozie-eventhub-retriever/target/classes:/home/cloudera/workspace-sts/java-framework/target/classes:/home/cloudera/.m2/repository/com/microsoft/azure/azure-storage/5.0.0/azure-storage-5.0.0.jar:<...>:/etc/hive/conf.dist es.mycompany.technology.bigdata.OozieAction json2hive
我想我的遮光罩配置不对。但我不明白为什么,我也看不出我做错了什么。。。
谢谢
1条答案
按热度按时间bzzcjhmw1#
下面的stackoverflow q/a回答了这个问题:ApacheSparkHive,带有MavenShade的可执行jar
对于那些不了解如何“合并”datanucleus中所有plugin.xml文件的人,您可以选择这个:plugin.xml并将其粘贴到资源文件夹中。