Spark-Cassandra-连接器:toArray不起作用

rqcrx0a6  于 2022-11-23  发布在  Cassandra
关注(0)|答案(2)|浏览(178)

我正在Scala中使用spark-cassandra-connector,我想从cassandra读取数据并通过toArray方法显示它。但是,我得到一个错误消息,它不是类的成员,但在API中指明了它。有人能帮我找到错误吗?
以下是我的文件:
build.sbt:

name := "Simple_Project"

version := "1.0"

scalaVersion := "2.11.8"

assemblyMergeStrategy in assembly := {
 case PathList("META-INF", xs @ _*) => MergeStrategy.discard
 case x => MergeStrategy.first
}

libraryDependencies += "org.apache.spark" %% "spark-core" % "2.0.0-preview"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.0.0-preview"
resolvers += "Spark Packages Repo" at "https://dl.bintray.com/spark-packages/maven"
libraryDependencies += "datastax" % "spark-cassandra-connector" % "2.0.0-M2-s_2.11"

SimpleScala.scala:

import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
import org.apache.spark.sql._
import org.apache.spark.sql.functions._
import com.datastax.spark.connector._
import com.datastax.spark.connector.rdd._
import org.apache.spark.sql.cassandra._
import org.apache.spark.sql.SQLContext
import com.datastax.spark.connector.cql.CassandraConnector._

object SimpleApp {

  def main(args: Array[String]) {

    val conf = new SparkConf().setAppName("Simple Application")
    conf.set("spark.cassandra.connection.host", "127.0.0.1")
    val sc = new SparkContext(conf)

    val rdd_2 = sc.cassandraTable("test_2", "words")

    rdd_2.toArray.foreach(println)
   }
 }

cqlsh的函数:

CREATE KEYSPACE test_2 WITH REPLICATION = {'class': 'SimpleStrategy', 'replication_factor': 1 };

CREATE TABLE test_2.words (word text PRIMARY KEY, count int);
INSERT INTO test_2.words (word, count) VALUES ('foo', 20);
INSERT INTO test_2.words (word, count) VALUES ('bar', 20);

错误消息:

[info] Loading global plugins from /home/andi/.sbt/0.13/plugins
[info] Resolving org.scala-sbt.ivy#ivy;2.3.0-sbt-2cc8d2761242b072cedb0a04cb39435[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Loading project definition from /home/andi/test_spark/project
[info] Updating {file:/home/andi/test_spark/project/}test_spark-build...
[info] Resolving org.scala-sbt.ivy#ivy;2.3.0-sbt-2cc8d2761242b072cedb0a04cb39435[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Set current project to Simple_Project (in build file:/home/andi/test_spark/)
[info] Compiling 1 Scala source to /home/andi/test_spark/target/scala-2.11/classes...
[error] /home/andi/test_spark/src/main/scala/SimpleApp.scala:50: value toArray is not a member of com.datastax.spark.connector.rdd.CassandraTableScanRDD[com.datastax.spark.connector.CassandraRow]
[error] rdd_2.toArray.foreach(println)
[error]       ^
[error] one error found
[error] (compile:compileIncremental) Compilation failed

非常感谢安迪

n3h0vuf2

n3h0vuf21#

自Spark Cassandra连接器2.0.0版本以来,CassandraTableScanRDD.toArray方法已被弃用并删除。此方法在1.6.0版本之前一直存在。您可以改用collect方法。

e37o9pze

e37o9pze2#

不幸的是,文档Spark Cassandra Connector仍然使用toArray。

rdd_2.collect.foreach(println)

相关问题