mrjob.conf文件的位置是什么?

puruo6ea  于 2021-06-02  发布在  Hadoop
关注(0)|答案(2)|浏览(431)

我的hadoop流媒体作业失败了。我在oraclevm上有一个hadoop沙盒,带有python模块mrjob。需要在mrjob.conf中进行一些更改,如hadoop错误:启动作业时出错,错误的输入路径:文件不存在。streaming命令失败,但我找不到此conf文件。
有谁能指导我度过难关吗?
顺便说一句,我的实际问题是:

hduser@ubuntu:~$ python suncode/RatingsBreakdown.py -r hadoop --hadoop-streaming-jar /usr/local/hadoop/share/hadoop/tools/lib/hadoop-streaming-2.7.2.jar sundata/u.data

Using configs in /etc/mrjob.conf
Looking for hadoop binary in /usr/local/hadoop/bin...
Found hadoop binary: /usr/local/hadoop/bin/hadoop
Using Hadoop version 2.7.2
Creating temp directory /tmp/RatingsBreakdown.hduser.20170921.083051.964674
Copying local files to hdfs:///user/hduser/tmp/mrjob/RatingsBreakdown.hduser.20170921.083051.964674/files/...
Running step 1 of 1...
  session.id is deprecated. Instead, use dfs.metrics.session-id
  Initializing JVM Metrics with processName=JobTracker, sessionId=
  Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
  Cleaning up the staging area file:/app/hadoop/data/mapred/staging/hduser577870072/.staging/job_local577870072_0001
  Error launching job , bad input path : File does not exist: /app/hadoop/data/mapred/staging/hduser577870072/.staging/job_local577870072_0001/files/RatingsBreakdown.py#RatingsBreakdown.py
  Streaming Command Failed!
Attempting to fetch counters from logs...
Can't fetch history log; missing job ID
No counters found
Scanning logs for probable cause of failure...
Can't fetch history log; missing job ID
Can't fetch task logs; missing application ID
Step 1 of 1 failed: Command '['/usr/local/hadoop/bin/hadoop', 'jar', '/usr/local/hadoop/share/hadoop/tools/lib/hadoop-streaming-2.7.2.jar', '-files', 'hdfs:///user/hduser/tmp/mrjob/RatingsBreakdown.hduser.20170921.083051.964674/files/RatingsBreakdown.py#RatingsBreakdown.py,hdfs:///user/hduser/tmp/mrjob/RatingsBreakdown.hduser.20170921.083051.964674/files/mrjob.zip#mrjob.zip,hdfs:///user/hduser/tmp/mrjob/RatingsBreakdown.hduser.20170921.083051.964674/files/setup-wrapper.sh#setup-wrapper.sh', '-input', 'hdfs:///user/hduser/tmp/mrjob/RatingsBreakdown.hduser.20170921.083051.964674/files/u.data', '-output', 'hdfs:///user/hduser/tmp/mrjob/RatingsBreakdown.hduser.20170921.083051.964674/output', '-mapper', 'sh -ex setup-wrapper.sh python3 RatingsBreakdown.py --step-num=0 --mapper', '-reducer', 'sh -ex setup-wrapper.sh python3 RatingsBreakdown.py --step-num=0 --reducer']' returned non-zero exit status 512
emeijp43

emeijp431#

你可以用这个命令找到文件, sudo find / -name "mrjob.conf" .
mrjob.conf通常位于以下位置:
mrjob\u conf指定的位置
~/.mrjob.conf
/etc/mrjob.conf文件

i86rm4rw

i86rm4rw2#

您应该在以下位置查找mrjob.conf:
mrjob\u conf指定的位置。

~/.mrjob.conf
/etc/mrjob.conf

相关问题