maven shade插件和datanucleus问题

juud5qan  于 2021-06-26  发布在  Hive
关注(0)|答案(1)|浏览(667)

我正试图从我的eclipseide中执行一个正常工作的代码,而我却面临着无法处理的奇怪错误。试图总结我的问题:
用eclipse执行代码:一切正常。
捕获eclipse抛出的命令行来运行我的应用程序,并将其复制到shell中:一切正常。
现在,eclipse生成的运行我的应用程序的命令行 java -cp lots-of-jars -Dvm.params myPackage.MyMainClass app-params .
我的目标是用oozie作为java操作来执行我的应用程序,所以我需要构建一个uberjar来减少myapp.jar中的大量jar。
为此,我将maven shade插件配置为:

<plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>2.4.2</version>
            <configuration>
            </configuration>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>shade</goal>
                    </goals>
                    <configuration>
                        <filters>
                            <filter>
                                <artifact>*:*</artifact>
                                <excludes>
                                    <exclude>META-INF/*.SF</exclude>
                                    <exclude>META-INF/*.DSA</exclude>
                                    <exclude>META-INF/*.RSA</exclude>
                                </excludes>
                            </filter>
                        </filters>
                        <transformers>
                            <transformer
                                implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                                <resource>reference.conf</resource>
                            </transformer>
                            <transformer
                                implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                <mainClass>es.mycompany.bigdata.OozieAction</mainClass>
                            </transformer>

<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
<transformer implementation="org.apache.maven.plugins.shade.resource.PluginXmlResourceTransformer" />
                        </transformers>
                    </configuration>
                </execution>
            </executions>
        </plugin>

我不得不添加一些变压器,因为我在启动我的应用程序时遇到了一些错误(不能创建fsshell spring对象,不能启动sparkcontext…),顺便说一下,我的应用程序的目的是下载一些azure blob,然后放入hdfs,用spark转换它们,最后添加到一个配置单元表。我用java开发了这个应用程序(包括spark部分),并使用spring来实现。
现在,我的上一个新问题发生在我尝试创建hivecontext时(我的spark上下文是正常的,因为如果我省略了hive部分,我的应用程序就会工作):

@Bean
@Lazy
@Scope("singleton")
public SQLContext getSQLContext(@Autowired JavaSparkContext sparkContext) {
    return new HiveContext(sparkContext);
}

引发的错误是:

2017-04-02 20:20:18 WARN  Persistence:106 - Error creating validator of type org.datanucleus.properties.CorePropertyValidator
ClassLoaderResolver for class "" gave error on creation : {1}
org.datanucleus.exceptions.NucleusUserException: ClassLoaderResolver for class "" gave error on creation : {1}
...
Caused by: org.datanucleus.exceptions.NucleusUserException: Persistence process has been specified to use a ClassLoaderResolver of name "datanucleus" yet this has not been found by the DataNucleus plugin mechanism. Please check your CLASSPATH and plugin specification.
        at org.datanucleus.NucleusContext.<init>(NucleusContext.java:283)
        at org.datanucleus.NucleusContext.<init>(NucleusContext.java:247)
        at org.datanucleus.NucleusContext.<init>(NucleusContext.java:225)
        at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.<init>(JDOPersistenceManagerFactory.java:416)
        at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:301)
        at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:202)
        ... 93 more
2017-04-02 20:20:18 WARN  ExtendedAnnotationApplicationContext:550 - Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'getOozieJavaAction': Unsatisfied dependency expressed through field 'sqlContext'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'getSQLContext' defined in es.mediaset.technology.bigdata.config.FlatJsonToCsvAppConfig: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.spark.sql.SQLContext]: Factory method 'getSQLContext' threw exception; nested exception is java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient

因为我的代码正确地运行在eclipse中,并使用这样的命令运行在eclipse之外

/usr/java/jdk1.8.0_121/bin/java -Dmode=responsive -Dspark.master=local[*] -Dfile.encoding=UTF-8 -classpath /home/cloudera/workspace-sts/oozie-eventhub-retriever/target/classes:/home/cloudera/workspace-sts/java-framework/target/classes:/home/cloudera/.m2/repository/com/microsoft/azure/azure-storage/5.0.0/azure-storage-5.0.0.jar:<...>:/etc/hive/conf.dist es.mycompany.technology.bigdata.OozieAction json2hive

我想我的遮光罩配置不对。但我不明白为什么,我也看不出我做错了什么。。。
谢谢

bzzcjhmw

bzzcjhmw1#

下面的stackoverflow q/a回答了这个问题:ApacheSparkHive,带有MavenShade的可执行jar
对于那些不了解如何“合并”datanucleus中所有plugin.xml文件的人,您可以选择这个:plugin.xml并将其粘贴到资源文件夹中。

相关问题