从包含map(键值对)的复杂json创建avro模式

rseugnpd  于 2021-05-29  发布在  Hadoop
关注(0)|答案(1)|浏览(315)

我有一个json文档,希望创建一个avro模式,用于数据序列化和反序列化。
我已经编写了下面定义的json文档的avro模式,但是当我根据该模式序列化json数据时,schemaparser抛出了一个异常。虽然我读了很多关于avro及其数据类型的文章,但还是无法克服这个问题。
对于这个问题,我指定了json文档、avro模式和schemaparser抛出的异常。
1) json文档

{
"category": "test",
"values": [
    {
        "subscriberid": 87392,
        "simserialnumber": 923,
        "MCC": 33,
        "MNC": [
            {
                "mn": {"key1":"kunal","key2":"gupta"},
                "mc": 44
            }
        ],
        "countryiso": "IN",
        "operatorname": "vodadone"
    }
]
}

2) avro模式

{
 "type": "record",
 "namespace": "testavro.schema",
 "name": "test",
 "fields": [
{
  "type": "string",
  "name": "data_version"
},
{
  "type": "string",
  "name": "ip_address"
},
{
  "type": "string",
  "name": "category"
},
{
  "type": {
    "items": {
      "fields": [
        {
          "type": "int",
          "name": "simserialnumber"
        },
        {
          "type": "string",
          "name": "countryiso"
        },
        {
          "type": "int",
          "name": "MCC"
        },
        {
          "type": "int",
          "name": "subscriberid"
        },
        {
          "type": {
            "items": {
              "fields": [
                {
                  "fields": [
                    {
                      "type": "string",
                      "name": "key2"
                    },
                    {
                      "type": "string",
                      "name": "key1"
                    }
                  ],
                  "type": "record",
                  "name": "mn"
                },
                {
                  "type": "int",
                  "name": "mc"
                }
              ],
              "type": "record",
              "name": "MNC_records"
            },
            "type": "array"
          },
          "name": "MNC"
        },
        {
          "type": "string",
          "name": "operatorname"
        }
      ],
      "type": "record",
      "name": "values_records"
    },
    "type": "array"
  },
  "name": "values"
}
]
}

3) schemaparserexception异常

SchemaParseException: Type property "{u'items': {u'fields': [{u'type': u'int', u'name': u'simserialnumber'}, {u'type': u'string', u'name': u'countryiso'}, {u'type': u'int', u'name': u'MCC'}, {u'type': u'int', u'name': u'subscriberid'}, {u'type': {u'items': {u'fields': [{u'fields': [{u'type': u'string', u'name': u'key2'}, {u'type': u'string', u'name': u'key1'}], u'type': u'record', u'name': u'mn'}, {u'type': u'int', u'name': u'mc'}], u'type': u'record', u'name': u'MNC_records'}, u'type': u'array'}, u'name': u'MNC'}, {u'type': u'string', u'name': u'operatorname'}], u'type': u'record', u'name': u'values_records'}, u'type': u'array'}" not a valid Avro schema: Items schema ({u'fields': [{u'type': u'int', u'name': u'simserialnumber'}, {u'type': u'string', u'name': u'countryiso'}, {u'type': u'int', u'name': u'MCC'}, {u'type': u'int', u'name': u'subscriberid'}, {u'type': {u'items': {u'fields': [{u'fields': [{u'type': u'string', u'name': u'key2'}, {u'type': u'string', u'name': u'key1'}], u'type': u'record', u'name': u'mn'}, {u'type': u'int', u'name': u'mc'}], u'type': u'record', u'name': u'MNC_records'}, u'type': u'array'}, u'name': u'MNC'}, {u'type': u'string', u'name': u'operatorname'}], u'type': u'record', u'name': u'values_records'}) not a valid Avro schema: Type property "{u'items': {u'fields': [{u'fields': [{u'type': u'string', u'name': u'key2'}, {u'type': u'string', u'name': u'key1'}], u'type': u'record', u'name': u'mn'}, {u'type': u'int', u'name': u'mc'}], u'type': u'record', u'name': u'MNC_records'}, u'type': u'array'}" not a valid Avro schema: Items schema ({u'fields': [{u'fields': [{u'type': u'string', u'name': u'key2'}, {u'type': u'string', u'name': u'key1'}], u'type': u'record', u'name': u'mn'}, {u'type': u'int', u'name': u'mc'}], u'type': u'record', u'name': u'MNC_records'}) not a valid Avro schema: Type property "record" not a valid Avro schema: Could not make an Avro Schema object from record. (known names: [u'testavro.schema.MNC_records', u'testavro.schema.test', u'testavro.schema.values_records']) (known names: [u'testavro.schema.MNC_records', u'testavro.schema.test', u'testavro.schema.values_records'])

请帮帮我,克服这个问题对我来说太好了。我花了一整天的时间在这个json和avro模式上,但是没有成功。

ajsxfq5m

ajsxfq5m1#

看起来里面有个拼写错误 items 的类型 MNC 你的领域 values_record . Package mn 的定义在新对象中起作用:

{
  "type": "record",
  "namespace": "testavro.schema",
  "name": "test",
  "fields": [
    {
      "type": "string",
      "name": "data_version"
    },
    {
      "type": "string",
      "name": "ip_address"
    },
    {
      "type": "string",
      "name": "category"
    },
    {
      "type": {
        "items": {
          "fields": [
            {
              "type": "int",
              "name": "simserialnumber"
            },
            {
              "type": "string",
              "name": "countryiso"
            },
            {
              "type": "int",
              "name": "MCC"
            },
            {
              "type": "int",
              "name": "subscriberid"
            },
            {
              "type": {
                "items": {
                  "fields": [
                    {
                      "type": {
                        "type": "record",
                        "fields": [
                          {
                            "type": "string",
                            "name": "key2"
                          },
                          {
                            "type": "string",
                            "name": "key1"
                          }
                        ],
                        "name": "Mn"
                      },
                      "name": "mn"
                    },
                    {
                      "type": "int",
                      "name": "mc"
                    }
                  ],
                  "type": "record",
                  "name": "MNC_records"
                },
                "type": "array"
              },
              "name": "MNC"
            },
            {
              "type": "string",
              "name": "operatorname"
            }
          ],
          "type": "record",
          "name": "values_records"
        },
        "type": "array"
      },
      "name": "values"
    }
  ]
}

相关问题