wordcount执行

dgiusagp  于 2021-05-29  发布在  Hadoop
关注(0)|答案(2)|浏览(303)

我按照一个教程学习hadoop和java。我写的 Wordcount 中的程序 IntelliJ 作业成功了,我可以看到正确的输出文件。现在,我想在hadoop集群中运行这个应用程序,但是失败了。hadoop安装本身就很好,并且可以正常启动。这是我的目录,

$ hadoop fs -ls 

2018-08-18 09:15:44,012 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 2 items
drwxr-xr-x   - chaklader supergroup          0 2018-08-17 12:17 Wordcount
-rw-r--r--   1 chaklader supergroup     530989 2018-08-15 13:13 forum_users.tsv

这个 pom.xml 应用程序中的文件如下所示,

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.test</groupId>
    <artifactId>wordcount</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <packaging>jar</packaging>

    <name>wordcount</name>
    <url>http://maven.apache.org</url>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencies>
        <!-- Hadoop -->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-core</artifactId>
            <version>RELEASE</version>
        </dependency>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>RELEASE</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.codehaus.mojo</groupId>
                <artifactId>exec-maven-plugin</artifactId>
                <version>1.2.1</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>java</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <mainClass>com.test.hadoop.WordCount</mainClass>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

这是项目设置,

当我运行程序时,它会创建一个 JAR 即文件 wordcount.jar 我把它放在 Downloads 目录。最后,我执行命令在 Hadoop 集群,

$ hadoop jar  Downloads/wordcount.jar  /Users/chaklader/IdeaProjects/Wordcount/src/main/java/com/test/hadoop/WordCount  /user/chaklader/Wordcount/Input/input.txt  /user/chaklader/Wordcount/Output

Usage: WordCount needs two arguments <input> <output> files

错误提示 WordCount needs two arguments <input> <output> files . 我检查了所有的路径,似乎都是正确的。
这里有什么问题?

2vuwiymt

2vuwiymt1#

您应该提供wordcount类的包路径。改变这个

$ hadoop jar  Downloads/wordcount.jar  /Users/chaklader/IdeaProjects/Wordcount/src/main/java/com/test/hadoop/WordCount  /user/chaklader/Wordcount/Input/input.txt  /user/chaklader/Wordcount/Output

$ hadoop jar  Downloads/wordcount.jar  com.test.hadoop.WordCount /user/chaklader/Wordcount/Input/input.txt /user/chaklader/Wordcount/Output
ezykj2lf

ezykj2lf2#

教程中的命令不正确。应该是的

$ hadoop jar wordcount.jar Wordcount/Input/input.txt  Wordcount/Output

在我们创造了 JAR 文件,它不需要首先提供创建它的java类路径。

相关问题