运行oozie工作流时出现异常

lokaqttq  于 2021-06-02  发布在  Hadoop
关注(0)|答案(0)|浏览(322)

我正在尝试运行我在hue中创建的一个简单工作流，它只调用一个没有参数的spark作业，但在运行它时遇到异常。这是在googledataproc集群上运行的，使用oozie4.2.0和hue3.9作为ui并运行它。
我已经确定mapreduce.framework.name是用于yarn的。到目前为止，我在网上找到了一些解决方案，但所有这些都围绕着工作流xml的问题。我已经用手重建了，它也在用同样的方法重建。
以下是我的工作流程代码：

<workflow-app name="Drug_File_Price_Generation" xmlns="uri:oozie:workflow:0.5">
<start to="spark-22ca"/>
<kill name="Kill">
    <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<action name="spark-22ca">
    <spark xmlns="uri:oozie:spark-action:0.1">
        <job-tracker>${jobTracker}</job-tracker>
        <name-node>${nameNode}</name-node>
        <master>local[*]</master>
        <mode>client</mode>
        <name>MySpark</name>
          <class>com.company.drug.GenerateDrugPriceFile</class>
        <jar>gs://customers/jar/DrugFileImport-assembly-1.0.jar</jar>
    </spark>
    <ok to="End"/>
    <error to="Kill"/>
</action>
<end name="End"/>

下面是我正在接收的日志的输出：

2016-03-22 19:27:27,927  INFO ActionStartXCommand:520 - SERVER[cluster-1-m.c.server-name.internal] USER[sayle.matthews] GROUP[-] TOKEN[] APP[Drug_File_Price_Generation] JOB[0000011-160322181838855-oozie-oozi-W] ACTION[0000011-160322181838855-oozie-oozi-W@:start:] Start action [0000011-160322181838855-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]

2016-03-22 19:27:27,930  INFO ActionStartXCommand:520 - SERVER[cluster-1-m.c.server-name.internal] USER[sayle.matthews] GROUP[-] TOKEN[] APP[Drug_File_Price_Generation] JOB[0000011-160322181838855-oozie-oozi-W] ACTION[0000011-160322181838855-oozie-oozi-W@:start:] [***0000011-160322181838855-oozie-oozi-W@:start:***]Action status=DONE

2016-03-22 19:27:27,930  INFO ActionStartXCommand:520 - SERVER[cluster-1-m.c.server-name.internal] USER[sayle.matthews] GROUP[-] TOKEN[] APP[Drug_File_Price_Generation] JOB[0000011-160322181838855-oozie-oozi-W] ACTION[0000011-160322181838855-oozie-oozi-W@:start:] [***0000011-160322181838855-oozie-oozi-W@:start:***]Action updated in DB!

2016-03-22 19:27:27,963  INFO WorkflowNotificationXCommand:520 - SERVER[cluster-1-m.c.server-name.internal] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000011-160322181838855-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000011-160322181838855-oozie-oozi-W

2016-03-22 19:27:27,963  INFO WorkflowNotificationXCommand:520 - SERVER[cluster-1-m.c.server-name.internal] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000011-160322181838855-oozie-oozi-W] ACTION[0000011-160322181838855-oozie-oozi-W@:start:] No Notification URL is defined. Therefore nothing to notify for job 0000011-160322181838855-oozie-oozi-W@:start:

2016-03-22 19:27:27,976  INFO ActionStartXCommand:520 - SERVER[cluster-1-m.c.server-name.internal] USER[sayle.matthews] GROUP[-] TOKEN[] APP[Drug_File_Price_Generation] JOB[0000011-160322181838855-oozie-oozi-W] ACTION[0000011-160322181838855-oozie-oozi-W@spark-22ca] Start action [0000011-160322181838855-oozie-oozi-W@spark-22ca] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]

2016-03-22 19:27:32,670  WARN ActionStartXCommand:523 - SERVER[cluster-1-m.c.server-name.internal] USER[sayle.matthews] GROUP[-] TOKEN[] APP[Drug_File_Price_Generation] JOB[0000011-160322181838855-oozie-oozi-W] ACTION[0000011-160322181838855-oozie-oozi-W@spark-22ca] Error starting action [spark-22ca]. ErrorType [TRANSIENT], ErrorCode [JA009], Message [JA009: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.]
org.apache.oozie.action.ActionExecutorException: JA009: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
    at org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:456)
    at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:436)
    at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1132)
    at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1286)
    at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:250)
    at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:64)
    at org.apache.oozie.command.XCommand.call(XCommand.java:286)
    at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:321)
    at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:250)
    at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
    at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:120)
    at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:82)
    at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:75)
    at org.apache.hadoop.mapred.JobClient.init(JobClient.java:475)
    at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:454)
    at org.apache.oozie.service.HadoopAccessorService$3.run(HadoopAccessorService.java:437)
    at org.apache.oozie.service.HadoopAccessorService$3.run(HadoopAccessorService.java:435)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
    at org.apache.oozie.service.HadoopAccessorService.createJobClient(HadoopAccessorService.java:435)
    at org.apache.oozie.action.hadoop.JavaActionExecutor.createJobClient(JavaActionExecutor.java:1329)
    at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1080)
    ... 10 more

编辑：根据要求，这里是我的mapred-site.xml文件：

<?xml version="1.0" ?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>mapreduce.job.maps</name>
    <value>20</value>
    <description>
      The default number of map tasks per job. Ignored when mapred.job.tracker
      is &amp;amp;amp;quot;local&amp;amp;amp;quot;.
    </description>
  </property>
  <property>
    <name>mapreduce.map.memory.mb</name>
    <value>3072</value>
    <source>programatically</source>
  </property>
  <property>
    <name>mapreduce.jobhistory.address</name>
    <value>cluster-1-m:10020</value>
    <description>MapReduce JobHistory Server IPC host:port</description>
  </property>
  <property>
    <name>mapreduce.jobhistory.webapp.address</name>
    <value>cluster-1-m:19888</value>
    <description>MapReduce JobHistory Server Web UI host:port</description>
  </property>
  <property>
    <name>mapreduce.reduce.memory.mb</name>
    <value>6144</value>
    <source>programatically</source>
  </property>
  <property>
    <name>yarn.app.mapreduce.am.command-opts</name>
    <value>-Xmx4915m</value>
    <source>programatically</source>
  </property>
  <property>
    <name>mapreduce.tasktracker.map.tasks.maximum</name>
    <value>4</value>
    <description>
      Property from MapReduce version 1 still used for TeraGen sharding.
    </description>
  </property>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
  <property>
    <name>mapreduce.input.fileinputformat.list-status.num-threads</name>
    <value>20</value>
    <description>
      The number of threads to use to list and fetch block locations for the
      specified input paths. Note: multiple threads should not be used if a
      custom non thread-safe path filter is used. Setting a larger value than
      the default of 1 can significantly improve job startup overhead,
      especially if using GCS as input with multi-level directories, such
      as in partitioned Hive tables.
    </description>
  </property>
  <property>
    <name>mapreduce.reduce.java.opts</name>
    <value>-Xmx4915m</value>
    <source>programatically</source>
  </property>
  <property>
    <name>yarn.app.mapreduce.am.resource.cpu-vcores</name>
    <value>2</value>
    <source>programatically</source>
  </property>
  <property>
    <name>mapreduce.reduce.cpu.vcores</name>
    <value>2</value>
    <source>programatically</source>
  </property>
  <property>
    <name>mapreduce.map.cpu.vcores</name>
    <value>1</value>
    <source>programatically</source>
  </property>
  <property>
    <name>yarn.app.mapreduce.am.resource.mb</name>
    <value>6144</value>
    <source>programatically</source>
  </property>
  <property>
    <name>mapreduce.job.reduces</name>
    <value>8</value>
    <description>
      The default number of reduce tasks per job. Typically set to 99%of the
      cluster's reduce capacity, so that if a node fails the reduces can still
      be executed in a single wave. Ignored when mapred.job.tracker is
      &amp;amp;amp;quot;local&amp;amp;amp;quot;.
    </description>
  </property>
  <property>
    <name>mapreduce.map.java.opts</name>
    <value>-Xmx2457m</value>
    <source>programatically</source>
  </property>
  <property>
    <name>mapreduce.job.working.dir</name>
    <value>/user/${user.name}</value>
    <description>
      The FileSystem working directory to use for relative paths.
    </description>
  </property>
  <property>
    <name>mapreduce.fileoutputcommitter.algorithm.version</name>
    <value>2</value>
    <description>
      Updated file output committer algorithm in Hadoop 2.7+. Significantly
      improves commitJob times when using the Google Cloud Storage connector.
      See https://issues.apache.org/jira/browse/MAPEDUCE-4815 for more details.
    </description>
  </property>
  <property>
    <name>mapred.local.dir</name>
    <value>/hadoop/mapred/local</value>
    <description>
      Directories on the local machine in which to store mapreduce temp files.
    </description>
  </property>
  <property>
    <name>mapred.bq.project.id</name>
    <value>instant-bonbon-111304</value>
    <description>
      Google Cloud Project ID to use for BigQuery operations.
    </description>
  </property>
  <property>
    <name>mapred.bq.output.buffer.size</name>
    <value>67108864</value>
    <description>
      The size in bytes of the output buffer to use when writing to BigQuery.
    </description>
  </property>
  <property>
    <name>mapred.bq.gcs.bucket</name>
    <value>dataproc-efab4f81-9a3a-4e62-ae58-db967541284c-us</value>
    <description>
      The GCS bucket holding temporary BigQuery data for the input connector.
    </description>
  </property>
</configuration>

编辑2：无法获取hue以导出job.properties文件，但这是我在命令行上测试时生成的文件。我相信生成的不会有太大的不同。

nameNode=hdfs://localhost:8020
jobTracker=localhost:8050
master=local[*]
oozie.wf.application.path=/home/sayle_matthews/workflow
oozie.use.system.libpath=true
hadoopooziehuegoogle-cloud-dataprocoozie-coordinator

暂无答案！

目前还没有任何答案，快来回答吧！

我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备2022004421号-2 NULL123 https://www.null123.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com