将表转换为json，省略重复的头行

wlp8pajw  于 2021-06-04  发布在  Kafka
关注(0)|答案(1)|浏览(334)

我有一个Kafka命令，将给你一个输出在下面的格式。

GROUP                         TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                                                     HOST            CLIENT-ID
connect-mm-leads-mapture-map DEV.mmvm.V2 0          9               9               0               connector-consumer-mm-leads-mapture-map-0-296b0e23-0995-4220-b493-fb006f41760a /10.00.000.00   connector-consumer-mm-leads-mapture-map-0
connect-mm-leads-mapture-map DEV.mmLeads.V1 1          7               7               0               connector-consumer-mm-leads-mapture-map-0-296b0e23-0995-4220-b493-fb006f41760a /10.01.01.01   connector-consumer-mm-leads-mapture-map-0
GROUP             TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                                         HOST            CLIENT-ID
connect-ccc-7562w DEV3.MMAutoshipTemplates.V1    0          594013          594021          8               connector-consumer-kkk-34ff-0-dfsfdsf-dsfd-1 /10.01.01.02   connector-consumer-ccc-7562w-0
connect-ccc-7562w DEV3.MMProducts.V1             0          4884            4885            1               connector-consumer-kkk-234f-2444vf-2-s-dsffdd-11 /10.01.01.03   connector-consumer-ccc-7562w-0
connect-kkk-7562w DEV3.mmLeads.V1
GROUP                              TOPIC              PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                                                                                                          HOST            CLIENT-ID
_company-centerpart-0-command _company-command 0          20              20              0               _company-centerpart-0-command-0d76f6e2-7ff3-47fa-a63c-0f36aedda046-StreamThread-1-consumer-66e0cc27-469e-4735-80cf-c930f27f6e5e /10.01.01.04   _company-centerpart-0-command-0d76f6e2-7ff3-47fa-a63c-0f36aedda046-StreamThread-1-consumer

但是我想要json格式的输出，内容如下。
期望输出应为：

[
 {
   "GROUP": "connect-mm-leads-mapture-map",
   "TOPIC": "DEV.mmvm.V2",
   "PARTITION": "0",
   "CURRENT-OFFSET": "9",
   "LOG-END-OFFSET": "9",
   "LAG": "0",
   "CONSUMER-ID": "connector-consumer-mm-leads-mapture-map-0-296b0e23-0995-4220-b493-fb006f41760a",
   "HOST": "/10.00.000.00",
   "CLIENT-ID": "connector-consumer-mm-leads-mapture-map-0"
 },
 {
   "GROUP": "connect-mm-leads-mapture-map",
   "TOPIC": "DEV.mmLeads.V1",
   "PARTITION": "1",
   "CURRENT-OFFSET": "7",
   "LOG-END-OFFSET": "7",
   "LAG": "0",
   "CONSUMER-ID": "connector-consumer-mm-leads-mapture-map-0-296b0e23-0995-4220-b493-fb006f41760a",
   "HOST": "/10.01.01.01",
   "CLIENT-ID": "connector-consumer-mm-leads-mapture-map-0"
 },
 {
   "GROUP": "connect-ccc-7562w",
   "TOPIC": "DEV3.MMAutoshipTemplates.V1",
   "PARTITION": "0",
   "CURRENT-OFFSET": "594013",
   "LOG-END-OFFSET": "594021",
   "LAG": "8",
   "CONSUMER-ID": "connector-consumer-kkk-34ff-0-dfsfdsf-dsfd-1",
   "HOST": "/10.01.01.02",
   "CLIENT-ID": "connector-consumer-ccc-7562w-0"
 },
 {
   "GROUP": "connect-ccc-7562w",
   "TOPIC": "DEV3.MMProducts.V1",
   "PARTITION": "0",
   "CURRENT-OFFSET": "4884",
   "LOG-END-OFFSET": "4885",
   "LAG": "1",
   "CONSUMER-ID": "connector-consumer-kkk-234f-2444vf-2-s-dsffdd-11",
   "HOST": "/10.01.01.03",
   "CLIENT-ID": "connector-consumer-ccc-7562w-0"
 },
 {
   "GROUP": "connect-kkk-7562w",
   "TOPIC": "DEV3.mmLeads.V1",
   "PARTITION": "",
   "CURRENT-OFFSET": "",
   "LOG-END-OFFSET": "",
   "LAG": "",
   "CONSUMER-ID": "",
   "HOST": "",
   "CLIENT-ID": ""
 },

 {
   "GROUP": "_company-centerpart-0-command",
   "TOPIC": "_company-command",
   "PARTITION": "0",
   "CURRENT-OFFSET": "20",
   "LOG-END-OFFSET": "20",
   "LAG": "0",
   "CONSUMER-ID": "_company-centerpart-0-command-0d76f6e2-7ff3-47fa-a63c-0f36aedda046-StreamThread-1-consumer-66e0cc27-469e-4735-80cf-c930f27f6e5e",
   "HOST": "/10.01.01.04",
   "CLIENT-ID": "_company-centerpart-0-command-0d76f6e2-7ff3-47fa-a63c-0f36aedda046-StreamThread-1-consumer"
 }
]

我正在使用下面的命令，并能够转换成预期的json与一些额外的无意义的数据。
命令：

./kafka-consumer-groups.sh --bootstrap-server "xxxxxxx:9092" --command-config /root/config.properties --describe --all-groups| jq -sR '[sub("\n$";"") | splits("\n") | sub("^ +";"") | [splits(" +")]] | .[1] as $header | .[2:] | [.[] | [. as $x | range($header | length) | {"key": $header[.], "value": $x[.]}] | from_entries]'

命令输出：

[
 {
   "GROUP": "connect-mm-leads-mapture-map",
   "TOPIC": "DEV.mmvm.V2",
   "PARTITION": "0",
   "CURRENT-OFFSET": "9",
   "LOG-END-OFFSET": "9",
   "LAG": "0",
   "CONSUMER-ID": "connector-consumer-mm-leads-mapture-map-0-296b0e23-0995-4220-b493-fb006f41760a",
   "HOST": "/10.00.000.00",
   "CLIENT-ID": "connector-consumer-mm-leads-mapture-map-0"
 },
 {
   "GROUP": "connect-mm-leads-mapture-map",
   "TOPIC": "DEV.mmLeads.V1",
   "PARTITION": "1",
   "CURRENT-OFFSET": "7",
   "LOG-END-OFFSET": "7",
   "LAG": "0",
   "CONSUMER-ID": "connector-consumer-mm-leads-mapture-map-0-296b0e23-0995-4220-b493-fb006f41760a",
   "HOST": "/10.01.01.01",
   "CLIENT-ID": "connector-consumer-mm-leads-mapture-map-0"
 },
 {
   "GROUP": "GROUP",
   "TOPIC": "TOPIC",
   "PARTITION": "PARTITION",
   "CURRENT-OFFSET": "CURRENT-OFFSET",
   "LOG-END-OFFSET": "LOG-END-OFFSET",
   "LAG": "LAG",
   "CONSUMER-ID": "CONSUMER-ID",
   "HOST": "HOST",
   "CLIENT-ID": "CLIENT-ID"
 },
 {
   "GROUP": "connect-ccc-7562w",
   "TOPIC": "DEV3.MMAutoshipTemplates.V1",
   "PARTITION": "0",
   "CURRENT-OFFSET": "594013",
   "LOG-END-OFFSET": "594021",
   "LAG": "8",
   "CONSUMER-ID": "connector-consumer-kkk-34ff-0-dfsfdsf-dsfd-1",
   "HOST": "/10.01.01.02",
   "CLIENT-ID": "connector-consumer-ccc-7562w-0"
 },
 {
   "GROUP": "connect-ccc-7562w",
   "TOPIC": "DEV3.MMProducts.V1",
   "PARTITION": "0",
   "CURRENT-OFFSET": "4884",
   "LOG-END-OFFSET": "4885",
   "LAG": "1",
   "CONSUMER-ID": "connector-consumer-kkk-234f-2444vf-2-s-dsffdd-11",
   "HOST": "/10.01.01.03",
   "CLIENT-ID": "connector-consumer-ccc-7562w-0"
 },
 {
   "GROUP": "connect-kkk-7562w",
   "TOPIC": "DEV3.mmLeads.V1",
   "PARTITION": "",
   "CURRENT-OFFSET": "",
   "LOG-END-OFFSET": "",
   "LAG": "",
   "CONSUMER-ID": "",
   "HOST": "",
   "CLIENT-ID": ""
 },
 {
   "GROUP": "GROUP",
   "TOPIC": "TOPIC",
   "PARTITION": "PARTITION",
   "CURRENT-OFFSET": "CURRENT-OFFSET",
   "LOG-END-OFFSET": "LOG-END-OFFSET",
   "LAG": "LAG",
   "CONSUMER-ID": "CONSUMER-ID",
   "HOST": "HOST",
   "CLIENT-ID": "CLIENT-ID"
 },
 {
   "GROUP": "_company-centerpart-0-command",
   "TOPIC": "_company-command",
   "PARTITION": "0",
   "CURRENT-OFFSET": "20",
   "LOG-END-OFFSET": "20",
   "LAG": "0",
   "CONSUMER-ID": "_company-centerpart-0-command-0d76f6e2-7ff3-47fa-a63c-0f36aedda046-StreamThread-1-consumer-66e0cc27-469e-4735-80cf-c930f27f6e5e",
   "HOST": "/10.01.01.04",
   "CLIENT-ID": "_company-centerpart-0-command-0d76f6e2-7ff3-47fa-a63c-0f36aedda046-StreamThread-1-consumer"
 }
]

如下所示，意味着应移除的（不需要的）输出更少：

{
   "GROUP": "GROUP",
   "TOPIC": "TOPIC",
   "PARTITION": "PARTITION",
   "CURRENT-OFFSET": "CURRENT-OFFSET",
   "LOG-END-OFFSET": "LOG-END-OFFSET",
   "LAG": "LAG",
   "CONSUMER-ID": "CONSUMER-ID",
   "HOST": "HOST",
   "CLIENT-ID": "CLIENT-ID"
 },

注意：命令输出可能有许多行。
谢谢。

JSONapache-kafkajq

1条答案

按热度按时间
egmofgnx

egmofgnx1#

为了将两个数组的对应元素配对，并从中构造一个对象，我们使用 transpose 内置与 add .
将原始输入转换成一个巨大的json脚本通常是不必要的，更不用说这种做法有多么糟糕了。
鉴于 --null-input 以及 --raw-input 命令行上的选项，此jq程序将生成预期的输出：

[ input | splits(" +") ] as $header | [
  inputs
  | [ splits(" +") ]
  | select(. != $header)
  | [ $header, . ]
  | [ transpose[] | { (.[0]): .[1] } ]
  | add
]

在线演示

赞(0）分享  回复(0）举报  2021-06-04
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备2022004421号-2 NULL123 https://www.null123.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com