rmr2 from.dfs导致r会话终止

irlmq6kh  于 2021-06-03  发布在  Hadoop
关注(0)|答案(0)|浏览(271)

我是hadoop的新手,尝试开始使用它。我已经“成功”在服务器上设置了一个单节点集群,并尝试开始使用r和rmr2包来运行mapreduce作业。但我甚至无法从hdfs中读取数据。当我从r执行以下命令时,一切似乎都很好:

> Sys.setenv(HADOOP_CMD="/usr/local/hadoop/bin/hadoop")
> Sys.setenv(HADOOP_STREAMING="/usr/local/hadoop/share/hadoop/tools/lib/hadoop-streaming-2.2.0.jar")
> 
> library(rmr2)
Loading required package: Rcpp
Loading required package: RJSONIO
Loading required package: bitops
Loading required package: digest
Loading required package: functional
Loading required package: stringr
Loading required package: plyr
Loading required package: reshape2
> rmr.options(backend="hadoop")
NULL
> 
> v <- keyval(1,2)
> to.dfs(v, "/user/dittmanc/test")
Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /usr/local/hadoop/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
14/01/15 10:26:35 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/01/15 10:26:36 INFO compress.CodecPool: Got brand-new compressor [.deflate]
[1] "/user/dittmanc/test"

使用以下命令从unix终端检查文件是否存在:

hdfs dfs -ls /user/dittmanc

文件在那里!
现在尝试读取文件:
from.dfs(“/user/dittmanc/test”)已弃用:不推荐使用此脚本执行hdfs命令。而是使用hdfs命令。

Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /usr/local/hadoop/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
14/01/15 10:34:20 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/01/15 10:34:21 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/01/15 10:34:23 INFO compress.CodecPool: Got brand-new decompressor [.deflate]

执行此操作将导致r会话错误异常终止,而不需要任何进一步的消息。使用执行代码

rmr.options(backend="local")

代码运行良好。所以这似乎是hdfs配置错误之类的。
谢谢你的帮助。

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题