找不到作业jar文件不能使用用户类

doinxwow  于 2021-06-02  发布在  Hadoop
关注(0)|答案(1)|浏览(351)

我试过浏览很多关于这个问题的博客，但是没有成功。我不知道我在这里犯了什么错误。。谁能帮帮我！
我的计划是：

package hadoopbook;

import java.io.IOException;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

public class WordCount {

    //Mapper
    public static class WcMapperDemo extends Mapper<LongWritable, Text, Text, IntWritable>{

        Text MapKey = new Text();
        IntWritable MapValue = new IntWritable();

        public void map(LongWritable key, Text Value, Context Context) throws IOException, InterruptedException{
            String Record = Value.toString();
            String[] Words = Record.split(",");

            for (String Word:Words){
                MapKey.set(Word);
                MapValue.set(1);
                Context.write(MapKey, MapValue);
            }   
        }
    }

    //Reducer
    public static class WcReducerDemo extends Reducer<Text, IntWritable, Text, IntWritable>{

        IntWritable RedValue = new IntWritable();

        public void reduce(Text key, Iterable<IntWritable> Values, Context Context) throws IOException, InterruptedException{
            int sum = 0;

            for (IntWritable Value:Values){
                sum = sum + Value.get();
            }
            RedValue.set(sum);
            Context.write(key, RedValue);
        }
    }

    //Driver
    public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException {
        // TODO Auto-generated method stub

        Configuration Conf = new Configuration();

        Job Job = new Job(Conf, "Word Count Job");

        Job.setJarByClass(WordCount.class);
        Job.setMapperClass(WcMapperDemo.class);
        Job.setReducerClass(WcReducerDemo.class);

        Job.setMapOutputKeyClass(Text.class);
        Job.setMapOutputValueClass(IntWritable.class);

        Job.setOutputKeyClass(Text.class);
        Job.setOutputValueClass(IntWritable.class);

        FileInputFormat.addInputPath(Job, new Path (args[0]));
        FileOutputFormat.setOutputPath(Job, new Path (args[1]));

        System.exit(Job.waitForCompletion(true) ? 0:1);
    }
}

我得到的错误是执行这个命令：

[cloudera@localhost Desktop]$ sudo -u hdfs hadoop jar WordCount.jar hadoopbook.WordCount /user/cloudera/InputFiles/Small/Words.txt /user/cloudera/Output

我的桌面文件夹里有wordcount.jar文件！

[cloudera@localhost Desktop]$ sudo -u hdfs hadoop jar WordCount.jar hadoopbook.WordCount /user/cloudera/InputFiles/Small/Words.txt /user/cloudera/Output
14/11/06 02:56:15 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
14/11/06 02:56:15 WARN mapred.JobClient: No job jar file set.  User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
14/11/06 02:56:15 INFO input.FileInputFormat: Total input paths to process : 1
14/11/06 02:56:16 INFO mapred.JobClient: Running job: job_201411040035_0017
14/11/06 02:56:17 INFO mapred.JobClient:  map 0% reduce 0%
14/11/06 02:56:29 INFO mapred.JobClient: Task Id : attempt_201411040035_0017_m_000000_0, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hadoopbook.WordCount$WcMapperDemo not found
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1617)
    at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:191)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:631)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
    at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.ClassNotFoundException: Class hadoopbook.WordCount$WcMapperDemo not found
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1523)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1615)
    ... 8 more

14/11/06 02:56:36 INFO mapred.JobClient: Task Id : attempt_201411040035_0017_m_000000_1, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hadoopbook.WordCount$WcMapperDemo not found
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1617)
    at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:191)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:631)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
    at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.ClassNotFoundException: Class hadoopbook.WordCount$WcMapperDemo not found
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1523)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1615)
    ... 8 more

14/11/06 02:56:42 INFO mapred.JobClient: Task Id : attempt_201411040035_0017_m_000000_2, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hadoopbook.WordCount$WcMapperDemo not found
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1617)
    at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:191)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:631)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
    at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.ClassNotFoundException: Class hadoopbook.WordCount$WcMapperDemo not found
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1523)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1615)
    ... 8 more

14/11/06 02:56:54 INFO mapred.JobClient: Job complete: job_201411040035_0017
14/11/06 02:56:54 INFO mapred.JobClient: Counters: 7
14/11/06 02:56:54 INFO mapred.JobClient:   Job Counters 
14/11/06 02:56:54 INFO mapred.JobClient:     Failed map tasks=1
14/11/06 02:56:54 INFO mapred.JobClient:     Launched map tasks=4
14/11/06 02:56:54 INFO mapred.JobClient:     Data-local map tasks=4
14/11/06 02:56:54 INFO mapred.JobClient:     Total time spent by all maps in occupied slots (ms)=33819
14/11/06 02:56:54 INFO mapred.JobClient:     Total time spent by all reduces in occupied slots (ms)=0
14/11/06 02:56:54 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
14/11/06 02:56:54 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0

我刚才试过了，但没有成功！我已经确保我的文件在桌面上！

[cloudera@localhost ~]$ sudo -u hdfs hadoop jar /home/cloudera/Desktop/WordCount.jar hadoopbook.WordCount /user/cloudera/InputFiles/Small/Words.txt /user/cloudera/Output
Exception in thread "main" java.io.IOException: Error opening job jar: /home/cloudera/Desktop/WordCount.jar
    at org.apache.hadoop.util.RunJar.main(RunJar.java:135)
Caused by: java.util.zip.ZipException: error in opening zip file
    at java.util.zip.ZipFile.open(Native Method)
    at java.util.zip.ZipFile.<init>(ZipFile.java:127)
    at java.util.jar.JarFile.<init>(JarFile.java:135)
    at java.util.jar.JarFile.<init>(JarFile.java:72)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:133)
hadoop

1条答案

按热度按时间
63lcw9qa

63lcw9qa1#

尝试以下命令

>hadoop jar <<absolute path to WordCount.jar>> hadoopbook.WordCount /user/cloudera/InputFiles/Small/Words.txt /user/cloudera/Output
赞(0）分享  回复(0）举报  2021-06-03
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备2022004421号-2 NULL123 https://www.null123.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com