我想把一个 data.frame
到 Spark DataFrame
:
library(sparklyr)
conf <- list()
conf$`sparklyr.cores.local` <- 4
conf$`sparklyr.shell.driver-memory` <- "16G"
conf$spark.memory.fraction <- 0.9
sc <- spark_connect(master = "local",
config = conf)
load("./data/dfm_training.RData")
load("./data/training.RData")
df <- training[,c(1,2,3,18)]
df$TextLength <- nchar(df$txt)
df <- df[df$TextLength > 0,]
df.tokens.df <- cbind(nchar = df$TextLength, label = df$training2_CFT,
convert(dfm_training, to = "data.frame"))
## to spark
sdf <- copy_to(sc, df.tokens.df)
但是,我得到以下错误: Error: C stack usage 133776899 is too close to the limit
我试过切换内核的数量和使用的内存的数量,但没有改变。
暂无答案!
目前还没有任何答案,快来回答吧!