无法在apache tez上运行

xriantvc  于 2021-05-29  发布在  Hadoop
关注(0)|答案(2)|浏览(476)

谢谢你把这个和截图一起发布,我已经按照你提到的步骤做了,但是我不能正确的安装,我遇到的错误是
错误:无法找到或加载主类org.apache.tez.dag.app.dagappmaster
我已经下载了tez的构建版本。Apache·泰兹version:0.8.4、hadoopversion:2.6.0.
我的tez-site.xml是

<?xml version="1.0" encoding="UTF-8"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
    <configuration>
    <property>
    <name>tez.lib.uris</name>
    <value>${fs.defaultFS}/apps/tez-0.8.4,${fs.defaultFS}/apps/tez-0.8.4/lib/</value>
    </property>
    </configuration>

我的bashrc配置是:

export HADOOP_HOME=/usr/local/hadoop
    export HADOOP_INSTALL=$HADOOP_HOME
    export HADOOP_MAPRED_HOME=$HADOOP_HOME
    export HADOOP_COMMON_HOME=$HADOOP_HOME
    export HADOOP_HDFS_HOME=$HADOOP_HOME
    export YARN_HOME=$HADOOP_HOME
    export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
    export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
    export PATH=$PATH:/usr/local/spark/bin
    export HIVE_HOME=/usr/local/hive
    export PATH=$PATH:$HIVE_HOME/bin
    export CLASSPATH=$CLASSPATH:/usr/local/Hadoop/lib/*:.
    export CLASSPATH=$CLASSPATH:/usr/local/hive/lib/*:.
    export DERBY_HOME=/usr/local/derby
    export PATH=$PATH:$DERBY_HOME/bin
    export CLASSPATH=$CLASSPATH:$DERBY_HOME/lib/derby.jar:$DERBY_HOME/lib/derbytools.jar
    export HIVE_OPTS="-hiveconf mapreduce.map.memory.mb=4096 -hiveconf mapreduce.reduce.memory.mb=5120"
    export TEZ_HOME=/usr/local/apache-tez-0.8.4-bin
    export TEZ_CONF_DIR=$TEZ_HOME/conf
    export TEZ_JARS=$TEZ_HOME

    if [ -z "$HIVE_AUX_JARS_PATH" ]; then
    export HIVE_AUX_JARS_PATH="$TEZ_JARS"
    else
    export HIVE_AUX_JARS_PATH="$HIVE_AUX_JARS_PATH:$TEZ_JARS"
    fi

    export HADOOP_CLASSPATH=${TEZ_CONF_DIR}:${TEZ_JARS}/*:${TEZ_JARS}/lib/*
    export CLASSPATH=$CLASSPATH:${TEZ_CONF_DIR}:${TEZ_JARS}/*:${TEZ_JARS}/lib/*:.

我的mapreduce-site.xml是

<configuration>
       <!--<property>
          <name>mapreduce.framework.name</name>
          <value>yarn</value>
       </property> -->
       <property>
        <name>mapreduce.framework.name</name>
        <value>yarn-tez</value>
        <description>The runtime framework for executing MapReduce jobs.
                Can be one of local, classic or yarn.
        </description>
        </property>
    </configuration>

当我尝试运行示例程序时,它返回的跟踪是

Failing this attempt. Failing the application.
    16/07/27 12:52:00 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
    16/07/27 12:52:00 INFO client.DAGClientImpl: DAG completed. FinalState=FAILED
    16/07/27 12:52:00 INFO examples.OrderedWordCount: DAG diagnostics: [Application application_1469604082434_0001 failed 2 times due to AM Container for appattempt_1469604082434_0001_000002 exited with  exitCode: 1
    For more detailed output, check application tracking page:http://AnalyticsLinux.tcs.com:8088/proxy/application_1469604082434_0001/Then, click on links to logs of each attempt.
    Diagnostics: Exception from container-launch.
    Container id: container_1469604082434_0001_02_000001
    Exit code: 1
    Stack trace: ExitCodeException exitCode=1: 
        at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
        at org.apache.hadoop.util.Shell.run(Shell.java:455)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
        at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

当我看到http://localhost:8088在stderr下我找到了上面的一个。
请帮我解决这个问题。提前谢谢!!。

cnwbcb6i

cnwbcb6i1#

由于tez是一个孵化器项目,我们需要下载src并使用maven构建。
有关tez0.8.1alpha版本的详细步骤,请参见下面的链接。
https://acadgild.com/blog/integrating-apache-tez-with-hadoop/
为了不面对这个问题,在你的linux机器上安装git、node js和npm,然后开始构建,这样你就可以成功地构建。
除了上述步骤之外,还需要在~/.bashrc文件中添加参数

export TEZ_CONF_DIR=/usr/local/tez/conf
 export TEZ_JARS=/usr/local/tez/tez
 export HADOOP_CLASSPATH=${TEZ_CONF_DIR}:${TEZ_JARS}/*:${TEZ_JARS}/lib/*:${HADOOP_CLASSPATH}:${JAVA_JDBC_LIBS}:${MAPREDUCE_LIBS}
 export CLASSPATH=$CLASSPATH:${TEZ_CONF_DIR}:${TEZ_JARS}/*:${TEZ_JARS}/lib/*

然后尝试运行任何成功运行的mapreduce作业。

s4chpxco

s4chpxco2#

您在hdfs中输入了错误的tez包。
hdfs dfs-mkdir/application/tez-0.8.4
tar-zxvf apache-tez-0.8.4-bin.tar.gz
hdfs dfs-copyfromlocal apache-tez-0.8.4-bin/share/tez.tar.gz/application/tez-0.8.4

相关问题