mongo hadoop流媒体

fcg9iug3  于 2021-05-30  发布在  Hadoop
关注(0)|答案(1)|浏览(576)

我刚接触mongodb和hadoop。我正在尝试访问mongodb数据作为hadoopmapreduce作业的输入。我不太知道如何指定从哪个集合获取数据。这就是我所尝试的:

hadoop jar/usr/local/Cellar/hadoop/2.6.0/libexec/share/hadoop/tools/lib/hadoop-streaming-2.6.0.jar 
-input user/test/input/
-output user/test/output/
-inputformat com.mongodb.hadoop.mapred.MongoInputFormat
-outputformat com.mongodb.hadoop.mapred.MongoOutputFormat
-io mongodb
-D mongo.input.uri=mongodb://localhost/my_dbs.collectionName 
-D stream.io.identifier.resolver.class=com.mongodb.hadoop.streaming.io.MongoIdentifierResolver 
-mapper /Users/wordcountMapper.py 
-reducer /Users/wordcountReducer.py 
-libjars /usr/local/Cellar/hadoop/2.6.0/libexec/share/hadoop/tools/lib/mongo-hadoop-streaming.jar

但我得到以下错误:

ERROR streaming.StreamJob: Unrecognized option: -D
Usage: $HADOOP_PREFIX/bin/hadoop jar hadoop-streaming.jar [options]

当我尝试这个的时候,我得到了另一个错误:

hadoop jar /usr/local/Cellar/hadoop/2.6.0/libexec/share/hadoop/tools/lib/hadoop-streaming-2.6.0.jar 
-input user/input/ 
-output user/test/output 
-inputformat com.mongodb.hadoop.mapred.MongoInputFormat 
-outputformat com.mongodb.hadoop.mapred.MongoOutputFormat 
-io mongodb -jobconf mongo.input.uri=mongodb://localhost/my_dbs.collectionName 
-jobconf stream.io.identifier.resolver.class=com.mongodb.hadoop.streaming.io.MongoIdentifierResolver 
-mapper /Users/wordcountMapper.py 
-reducer /Users/wordcountReducer.py 
-libjars /usr/local/Cellar/hadoop/2.6.0/libexec/share/hadoop/tools/lib/mongo-hadoop-streaming.jar

`ERROR streaming.StreamJob: Unrecognized option: -libjars
Usage: $HADOOP_PREFIX/bin/hadoop jar hadoop-streaming.jar [options]`

请帮忙。

r3i60tvu

r3i60tvu1#

有关如何将mongodb连接到hadoop的更好方法,请查看此链接。
编辑:
或者,
您可以直接在驱动程序中编写jar,而不是使用-libjars选项:

args.add("-libjars");
args.add("/some/path/to/your/jar");

相关问题