我正在从hdfs读取r中的数据。当我使用 sparklyr 正在破译错误消息…因为我不是java程序员。
考虑这个例子：
在r里做这个
创建鲍鱼数据框架-鲍鱼是一个用于机器学习示例的数据集

load pivotal R package #contains abalone data and create dataframe
if (!require(PivotalR)){ 
  install.packages(PivotalR) }

data(abalone)

# sample of data

head(abalone)

# export data to a CSV file

if (!require(readr)){ 
  install.packages(readr) }
write_csv(abalone,'abalone.csv')

在命令行执行此操作

hdfs dfs -put abalone.csv abalone.csv

# check to see if the file is on the hdfs

hdfs dfs -ls

在r中执行此操作此设置是为了使用当前版本的spark，您可能需要将spark\u更改为home

library(sparklyr)
    library(SparkR)
    sc = spark_connect(master = 'yarn-client',
                       spark_home = '/usr/hdp/current/spark-client',
                       app_name = 'sparklyr',
                       config = list(
                         "sparklyr.shell.executor-memory" = "1G",
                         "sparklyr.shell.driver-memory"   = "4G",
                         "spark.driver.maxResultSize"     = "2G" # may need to transfer a lot of data into R 
    )
    )

读入我们刚刚写给hdfs的abalone文件。您必须更改路径以匹配您的路径。

df <- spark_read_csv(sc,name='abalone',path='hdfs://pnhadoop/user/stc004/abalone.csv',delimiter=",",
                         header=TRUE)

我得到以下错误：

Error: java.lang.IllegalArgumentException: invalid method csv for object 63
        at sparklyr.Invoke$.invoke(invoke.scala:113)
        at sparklyr.StreamHandler$.handleMethodCall(stream.scala:89)
        at sparklyr.StreamHandler$.read(stream.scala:55)
        at sparklyr.BackendHandler.channelRead0(handler.scala:49)
        at sparklyr.BackendHandler.channelRead0(handler.scala:14)
        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:244)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
        at java.lang.Thread.run(Thread.java:745)

不知道发生了什么。我用过 spark_read_csv 以前没有错误。我不知道如何破译java错误。思想？