sparksql with scalapb:在输出proto格式中使用maptype会在调用tobytestring时产生scala.matcherror

fnvucqvd 于 2021-05-27 发布在 Spark

关注(0)|答案(1)|浏览(463)

以下是我的输出消息格式：

message EditorialTextAdEnforcementData {
  int32 customerId = 1;
  int32 source = 2;
  DecisionDetails decisionDetails = 3;
  int32 flagsEnforceOption = 4;
  int32 categoryEnforceOption = 5;
  int32 applyBypass = 6;
  map<int32, string> categories = 7;
  bcl.DateTime rowDateSource = 8;
  int32 accountId = 9;
  int64 adId = 10;
  int64 orderId = 11;
  int32 adType = 12;
  int32 campaignType = 13;
  bool hasImage = 14;
  bool isNewAdType = 15;
}

在准备输出数据集时，我使用以下方法包含一个虚拟Map

val output =...
        .withColumn(UC.Categories, map(lit("1"), lit("test"))).as[EditorialTextAdEnforcementData]
 output.show()

在这一步之前都很好。但是，当我打电话时：

output.map(_.toByteString).show()

我得到以下错误：

Exception in thread "main" scala.MatchError: MapType(IntegerType,StringType,false) (of class org.apache.spark.sql.types.MapType)
    at org.apache.spark.sql.catalyst.expressions.objects.MapObjects.doGenCode(objects.scala:836)
    at org.apache.spark.sql.catalyst.expressions.Expression$$anonfun$genCode$2.apply(Expression.scala:108)
    at org.apache.spark.sql.catalyst.expressions.Expression$$anonfun$genCode$2.apply(Expression.scala:105)
    at scala.Option.getOrElse(Option.scala:121)

如果我删除categories字段，它似乎可以正常工作。请让我知道如何将Map类型字段编写为protobuf消息

apache-spark apache-spark-sql scalapb

来源：https://stackoverflow.com/questions/62875322/sparksql-with-scalapb-using-maptype-in-output-proto-format-gives-scala-matcherr

1条答案

按热度按时间

nwnhqdif1#

这是一个未解决的问题https://github.com/scalapb/sparksql-scalapb/issues/79).
作为解决方法，我们可以使用map作为消息中的封装结构字段，如下所示：https://developers.google.com/protocol-buffers/docs/proto3#backwards_compatibility. 具体如下：

message Categories {
  int32 key = 1;
  string value = 2;
}

message EditorialTextAdEnforcementData {
  int32 customerId = 1;
  int32 source = 2;
  repeated DecisionDetails decisionDetails = 3;
  int32 flagsEnforceOption = 4;
  int32 categoryEnforceOption = 5;
  int32 applyBypass = 6;
  repeated Categories categories = 7;
  bcl.DateTime rowDateSource = 8;
  int32 accountId = 9;
  int64 adId = 10;
  int64 orderId = 11;
  int32 adType = 12;
  int32 campaignType = 13;
  bool hasImage = 14;
  bool isNewAdType = 15;
}

赞(0）回复(0）举报 2021-05-27

我来回答

sparksql with scalapb:在输出proto格式中使用maptype会在调用tobytestring时产生scala.matcherror

1条答案

相关问题

热门标签

最新问答