我在databricks笔记本上使用spark,cassandra,spark cassandra连接器,根据他们的网站,我们可以使用'deletefromcassandra'删除行:https://github.com/datastax/spark-cassandra-connector/blob/master/doc/5_saving.md, https://datastax-oss.atlassian.net/browse/sparkc-349 下面是我的python脚本:
def read_table(tableName,kespace, columns):
dfData = (spark
.read
.format("org.apache.spark.sql.cassandra")
.options(table = tableName, keyspace = kespace)
.load()
.select(*columns))
return dfData
emails='abc@test.com'.split(",")
df = read_table(my_table, my_keyspace,"*").where(col("email").isin(emails))
df.rdd.deleteFromCassandra(my_keyspace, my_table)
但它失败了:
AttributeError: 'RDD' object has no attribute 'deleteFromCassandra'
注意到他们提供的所有示例都在scala中,这是否意味着函数deletefromcassandra在python中不可用?
2条答案
按热度按时间cig3rfwq1#
使用stock spark cassandra连接器是不可能的,因为python绑定只支持Dataframe。但PySparkCassandra应该是可能的,这也可以在SparkPackages网站上找到
--packages anguenot:pyspark-cassandra:2.4.0
. 像这样:kulphzqa2#
希望这能解决-