我的spark submit(使用poi3.17)和hadoop类路径上的rapid miner jar(使用poi3.14)之间的poijava库的jar冲突
因此,我试图通过shell操作赋予jar优先级。
我了解到这可以通过java操作实现,但是我更喜欢使用shell操作。
附件是我的oozie工作流使用下面的这两个属性,但当我添加这些额外的属性时,我的工作流就挂起了
<property>
<name>oozie.launcher.mapreduce.task.classpath.user.precedence</name>
<value>true</value>
</property>
<property>
<name>oozie.launcher.mapreduce.job.classpath.user.precedence</name>
<value>true</value>
</property>
附件是完整的工作流程
<workflow-app xmlns='uri:oozie:workflow:0.5' name='Stuff' xmlns:sla="uri:oozie:sla:0.2">
<global>
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>mapreduce.job.queuename</name>
<value>${queueName}</value>
</property>
<property>
<name>oozie.launcher.mapreduce.task.classpath.user.precedence</name>
<value>true</value>
</property>
<property>
<name>oozie.launcher.mapreduce.job.classpath.user.precedence</name>
<value>true</value>
</property>
</configuration>
</global>
<credentials>
<credential name="hive2_credentials" type="hive2">
<property>
<name>hive2.jdbc.url</name>
<value>jdbc:hive2://${hive2_server}:${hive2_port}/default;ssl=true</value>
</property>
<property>
<name>hive2.server.principal</name>
<value>hive/${hive2_server}@${kerberos_realm}</value>
</property>
</credential>
</credentials>
<start to="spark2-submit"/>
<!-- Spark job to process the Vistalink Consumer - Shell-action -->
<action name="spark2-submit">
<shell xmlns="uri:oozie:shell-action:0.3">
<exec>spark_submit.sh</exec>
<argument>dev_submit.config</argument>
<argument>2018-08-08</argument>
<env-var>HADOOP_CONF_DIR=/etc/hadoop/conf</env-var>
<env-var>kerberos_realm=${kerberos_realm}</env-var>
<env-var>edgeUser=${edgeUser} </env-var>
<env-var>queueName=${queueName}</env-var>
<file>${keyTabLocation}/${keyTabName}</file>
<file>${nameNode}/abc.sh#abc.sh</file>
</shell>
<ok to="end"/>
<error to="end"/>
</action>
<kill name="fail">
<message>"WF: failed"</message>
</kill>
<end name='end'/>
</workflow-app>
我也尝试在shell操作中添加配置参数,但工作流只是挂起而不是事件启动。
我试图通过修改
export CLASSPATH=`echo $OLDCLASSPATH | sed 's|:/opt/cloudera/parcels/RAPIDMINER_LIBS-7.6.1/lib/radoop/rapidminer_libs-7.6.1.jar||g' | sed 's|:/opt/cloudera/parcels/RAPIDMINER_LIBS-7.6.1/lib/radoop/radoop_hive-v4.jar||g'`
export AUX_CLASSPATH=`echo $AUX_CLASSPATH | sed 's|/opt/cloudera/parcels/RAPIDMINER_LIBS-7.6.1/lib/radoop/rapidminer_libs-7.6.1.jar:||g' | sed 's|/opt/cloudera/parcels/RAPIDMINER_LIBS-7.6.1/lib/radoop/radoop_hive-v4.jar||g'`
export MR2_CLASSPATH=`echo $MR2_CLASSPATH | sed 's|/opt/cloudera/parcels/RAPIDMINER_LIBS-7.6.1/lib/radoop/rapidminer_libs-7.6.1.jar:||g' | sed 's|/opt/cloudera/parcels/RAPIDMINER_LIBS-7.6.1/lib/radoop/radoop_hive-v4.jar||g'`
export SCM_DEFINES_SCRIPTS=`echo $SCM_DEFINES_SCRIPTS | sed 's|:/opt/cloudera/parcels/RAPIDMINER_LIBS-7.6.1/meta/parcel_env.sh||g'`
export PARCEL_DIRNAMES=`echo $PARCEL_DIRNAMES | sed 's|RAPIDMINER_LIBS-7.6.1||g'`
在这之后,我检查了java类路径,通过引用java.class.path没有rapid miner
使用ApacheSpark2.3和CDH5.14
非常感谢您的帮助
暂无答案!
目前还没有任何答案,快来回答吧!