我刚接触mongodb和hadoop。我正在尝试访问mongodb数据作为hadoopmapreduce作业的输入。我不太知道如何指定从哪个集合获取数据。这就是我所尝试的:
hadoop jar/usr/local/Cellar/hadoop/2.6.0/libexec/share/hadoop/tools/lib/hadoop-streaming-2.6.0.jar
-input user/test/input/
-output user/test/output/
-inputformat com.mongodb.hadoop.mapred.MongoInputFormat
-outputformat com.mongodb.hadoop.mapred.MongoOutputFormat
-io mongodb
-D mongo.input.uri=mongodb://localhost/my_dbs.collectionName
-D stream.io.identifier.resolver.class=com.mongodb.hadoop.streaming.io.MongoIdentifierResolver
-mapper /Users/wordcountMapper.py
-reducer /Users/wordcountReducer.py
-libjars /usr/local/Cellar/hadoop/2.6.0/libexec/share/hadoop/tools/lib/mongo-hadoop-streaming.jar
但我得到以下错误:
ERROR streaming.StreamJob: Unrecognized option: -D
Usage: $HADOOP_PREFIX/bin/hadoop jar hadoop-streaming.jar [options]
当我尝试这个的时候,我得到了另一个错误:
hadoop jar /usr/local/Cellar/hadoop/2.6.0/libexec/share/hadoop/tools/lib/hadoop-streaming-2.6.0.jar
-input user/input/
-output user/test/output
-inputformat com.mongodb.hadoop.mapred.MongoInputFormat
-outputformat com.mongodb.hadoop.mapred.MongoOutputFormat
-io mongodb -jobconf mongo.input.uri=mongodb://localhost/my_dbs.collectionName
-jobconf stream.io.identifier.resolver.class=com.mongodb.hadoop.streaming.io.MongoIdentifierResolver
-mapper /Users/wordcountMapper.py
-reducer /Users/wordcountReducer.py
-libjars /usr/local/Cellar/hadoop/2.6.0/libexec/share/hadoop/tools/lib/mongo-hadoop-streaming.jar
`ERROR streaming.StreamJob: Unrecognized option: -libjars
Usage: $HADOOP_PREFIX/bin/hadoop jar hadoop-streaming.jar [options]`
请帮忙。
1条答案
按热度按时间r3i60tvu1#
有关如何将mongodb连接到hadoop的更好方法,请查看此链接。
编辑:
或者,
您可以直接在驱动程序中编写jar,而不是使用-libjars选项: