我试图从mrjob的网站上对我的自定义数据运行基本示例。我已经使用流成功地运行了hadoop map reduce,我也成功地尝试了不使用hadoop的脚本,但是现在我尝试通过mrjob在hadoop上通过以下命令运行它。
./mapred.py -r hadoop --hadoop-bin /usr/bin/hadoop -o hdfs:///user/cloudera/wc_result_mrjob hdfs:///user/cloudera/books
mapred.py的源代码如下:
# ! /usr/bin/env python
from mrjob.job import MRJob
class MRWordFrequencyCount(MRJob):
def mapper(self, _, line):
yield "chars", len(line)
yield "words", len(line.split())
yield "lines", 1
def reducer(self, key, values):
yield key, sum(values)
if __name__ == '__main__':
MRWordFrequencyCount.run()
不幸的是,我得到以下错误:
no configs found; falling back on auto-configuration
no configs found; falling back on auto-configuration
creating tmp directory /tmp/mapred.cloudera.20140824.195414.420162
writing wrapper script to /tmp/mapred.cloudera.20140824.195414.420162/setup-wrapper.sh
STDERR: mkdir: `hdfs:///user/cloudera/tmp/mrjob/mapred.cloudera.20140824.195414.420162/files/': No such file or directory
Traceback (most recent call last):
File "./mapred.py", line 18, in <module>
MRWordFrequencyCount.run()
File "/usr/lib/python2.6/site-packages/mrjob/job.py", line 494, in run
mr_job.execute()
File "/usr/lib/python2.6/site-packages/mrjob/job.py", line 512, in execute
super(MRJob, self).execute()
File "/usr/lib/python2.6/site-packages/mrjob/launch.py", line 147, in execute
self.run_job()
File "/usr/lib/python2.6/site-packages/mrjob/launch.py", line 208, in run_job
runner.run()
File "/usr/lib/python2.6/site-packages/mrjob/runner.py", line 458, in run
self._run()
File "/usr/lib/python2.6/site-packages/mrjob/hadoop.py", line 238, in _run
self._upload_local_files_to_hdfs()
File "/usr/lib/python2.6/site-packages/mrjob/hadoop.py", line 265, in _upload_local_files_to_hdfs
self._mkdir_on_hdfs(self._upload_mgr.prefix)
File "/usr/lib/python2.6/site-packages/mrjob/hadoop.py", line 273, in _mkdir_on_hdfs
self.invoke_hadoop(['fs', '-mkdir', path])
File "/usr/lib/python2.6/site-packages/mrjob/fs/hadoop.py", line 109, in invoke_hadoop
raise CalledProcessError(proc.returncode, args)
subprocess.CalledProcessError: Command '['/usr/bin/hadoop', 'fs', '-mkdir', 'hdfs:///user/cloudera/tmp/mrjob/mapred.cloudera.20140824.195414.420162/files/']' returned non-zero exit status 1
在我看来,mrjob无法在hdfs中创建一些目录,但不幸的是,我不知道如何解决这个问题。
我的hadoop是clouderacdh5.1。快速启动。
提前感谢您的建议和建议。
编辑:
我尝试在cloudera cdh4.7上运行相同的代码。快速启动,效果很好。所以我修改的问题是:clouderacdh5.1。由mrjob框架支持?如果是这样,那么如何运行它?
1条答案
按热度按时间r1wp621o1#
我遇到了相同的错误,解决方法是更改:
self.invoke_hadoop(['fs', '-mkdir', path])
到self.invoke_hadoop(['fs', '-mkdir','-p', path])
修改的文件是:/usr/lib/python2.6/site-packages/mrjob/hadoop.py我的mrjob已经运行了几个月了,没有任何问题,所以我觉得这很好。
我自己也很想知道同样的选择。