提交mapreduce作业时出错

eivgtgni  于 2021-06-01  发布在  Hadoop
关注(0)|答案(1)|浏览(393)

我有2个csv文件,内容书的数据细节。我已经编写了java代码并制作了jar文件。
在执行jar文件时,我得到了下面的错误。如果你需要我可以分享完整的数据集和代码。请求帮助我的查询`

user@ubuntu:~$ hadoop jar /home/user/Desktop/project1_usecase3.jar /Input/BX-Book-Ratings.csv /Output
18/03/23 02:28:23 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/03/23 02:28:25 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
18/03/23 02:28:25 WARN mapreduce.JobSubmitter: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
18/03/23 02:28:26 INFO mapreduce.JobSubmitter: Cleaning up the staging area /tmp/hadoop-yarn/staging/user/.staging/job_1521796431998_0005
Exception in thread "main" org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://localhost:9000/input/BX-Book-Ratings.csv
    at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:321)
    at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:264)
    at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:385)
    at org.apache.hadoop.mapreduce.lib.input.DelegatingInputFormat.getSplits(DelegatingInputFormat.java:115)
    at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:597)
    at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:614)
    at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:492)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1293)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:1293)
    at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314)
    at com.edunextgen.maven.project1.NumBook.main(NumBook.java:106)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)`
46qrfjad

46qrfjad1#

记得吗 /Input/BX-Book-Ratings.csv 需要存在于hdfs上,而不是本地计算机上。如果你这样做了 hdfs dfs -ls /input ,你看到文件了吗?
如果没有,就做

hdfs dfs -copyFromLocal <csv-file-local-location> /Input/BX-Book-Ratings.csv

相关问题