如何使用spark cassandra连接器保存javabean?

wlsrxk51  于 2021-07-13  发布在  Spark
关注(0)|答案(1)|浏览(345)

我已经阅读了spark文档,但不确定如何使用spark cassandra连接器将javabean保存到表中?

public class NewImageMetadataRow implements Serializable {

    private final String merchant;
    private final String productId;
    private final String url;
    private final int width;
    private final int height;

    public NewImageMetadataRow(NewImageMetadataRow row) {
        this.merchant = row.getMerchant();
        this.productId = row.getProductId();
        this.url = row.getUrl();
        this.width = row.getWidth();
        this.height = row.getHeight();
    }

    public String getMerchant() {
        return merchant;
    }

    public String getProductId() {
        return productId;
    }

    public String getUrl() {
        return url;
    }

    public int getWidth() {
        return width;
    }

    public int getHeight() {
        return height;
    }
}

我有rdd RDD[NewImageMetadataRow] 我想这样保存:

myRDD.saveToCassandra(keyspace, "imagemetadatav3", SomeColumns("merchant", "productid", "url"))

这会导致以下错误:

java.lang.IllegalArgumentException: requirement failed: Columns not found in com.mridang.image.NewImageMetadataRow: [merchant, productid, url]
    at scala.Predef$.require(Predef.scala:281)
    at com.datastax.spark.connector.mapper.DefaultColumnMapper.columnMapForWriting(DefaultColumnMapper.scala:106)
    at com.datastax.spark.connector.mapper.MappedToGettableDataConverter$$anon$1.<init>(MappedToGettableDataConverter.scala:35)
    at com.datastax.spark.connector.mapper.MappedToGettableDataConverter$.apply(MappedToGettableDataConverter.scala:26)
    at com.datastax.spark.connector.writer.DefaultRowWriter.<init>(DefaultRowWriter.scala:16)
    at com.datastax.spark.connector.writer.DefaultRowWriter$$anon$1.rowWriter(DefaultRowWriter.scala:30)
    at com.datastax.spark.connector.writer.DefaultRowWriter$$anon$1.rowWriter(DefaultRowWriter.scala:28)
    at com.datastax.spark.connector.writer.TableWriter$.apply(TableWriter.scala:423)
    at com.datastax.spark.connector.RDDFunctions.saveToCassandra(RDDFunctions.scala:35)

根据我的理解(以及糟糕的scalafoo),它似乎无法从javabean中推断属性名。
另一个问题是,我的表中的列名都是小写的,删除了空格和连字符,即getter对应的cassandra列 getProductIdproductid .
(如果我用的是Jackson,我可以简单地加上 JsonProperty 注解。我想知道我能用卡萨德拉Map绘制器做同样的事情吗

nfzehxib

nfzehxib1#

这需要一点戳周围,但结果是这样的:

val columns: RowWriterFactory[NewImageMetadataRow] =
  CassandraJavaUtil.mapToRow(classOf[NewImageMetadataRow])

myRDD.saveToCassandra(keyspace, "imagemetadatav3")(CassandraConnector(sc), columns)

bean中的字段需要是公共的,并用 @CqlName 注解。

@CqlName("merchant")
public final String merchant;

相关问题