bash文本解析配置单元ddl

a7qyws3x 于 2021-06-26 发布在 Hive

关注(0)|答案(1)|浏览(420)

下面是一组重复的文本，它本质上是来自两个hadoop配置单元表tablea1和tablea2的descripe扩展输出，并显示了它的属性。

Detailed Table Information      Table(tableName:tablea1, dbName:default, owner:eedc_hdp_s_d-itm-e, createTime:1519807981, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:col1, type:int, comment:null), FieldSchema(name:col2, type:int, comment:null)], location:hdfs://DBDP-Dev/apps/hive/warehouse/tablea1, inputFormat:org.apache.hadoop.mapred.TextInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, parameters:{serialization.format=1}), bucketCols:[], sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[], skewedColValueLocationMaps:{}), storedAsSubDirectories:false), partitionKeys:[], parameters:{totalSize=0, rawDataSize=0, numRows=0, COLUMN_STATS_ACCURATE={"BASIC_STATS":"true"}, numFiles=0, transient_lastDdlTime=1519807981}, viewOriginalText:null, viewExpandedText:null, tableType:MANAGED_TABLE)

Detailed Table Information      Table(tableName:tablea2, dbName:default, owner:eedc_hdp_s_d-itm-e, createTime:1519807982, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:col3, type:int, comment:null), FieldSchema(name:col4, type:int, comment:null)], location:hdfs://DBDP-Dev/apps/hive/warehouse/tablea2, inputFormat:org.apache.hadoop.mapred.TextInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, parameters:{serialization.format=1}), bucketCols:[], sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[], skewedColValueLocationMaps:{}), storedAsSubDirectories:false), partitionKeys:[], parameters:{totalSize=0, rawDataSize=0, numRows=0, COLUMN_STATS_ACCURATE={"BASIC_STATS":"true"}, numFiles=0, transient_lastDdlTime=1519807982}, viewOriginalText:null, viewExpandedText:null, tableType:MANAGED_TABLE)
Time taken: 0.08 seconds, Fetched: 4 row(s)

我试图从上面的数据生成一个tablename | columnname，如下所示

tablea1|col1
tablea1|col2
tablea2|col3
tablea2|col4

我能够生成两个命令来生成每个列

grep -o 'Table(tableName:[^,]*' sample_file  | awk -F ':' '{ print $2}'

给出第一列

tablea1
tablea2

grep -o 'FieldSchema(name:[^,]*' sample_file | awk -F ':' '{ print $2}' | uniq

给出第二列

col1
col2
col3
col4

但我无法进一步取得预期的成果

tablea1|col1
tablea1|col2
tablea2|col3
tablea2|col4

你能帮忙吗。还是有更简单的方法？

Hive ddl dynamic-sql bash parsing

来源：https://stackoverflow.com/questions/49027501/bash-text-parsing-a-hive-ddl

1条答案

按热度按时间

13z8s7eq1#

使用 -n 选择grep和 -o 同时携带火柴的行号。
然后使用 join 通过与行号连接作为键来获得所需的输出。refer:join to 理解使用的参数。

grep -on 'Table(tableName:[^,]*' sample_file  | awk -F ':' '{ OFS="|";print $1,$3}'  >file1

grep -on 'FieldSchema(name:[^,]*' sample_file | awk -F ':' '{ OFS="|";print $1,$3}'  >file2

join -t "|" -1 1 -2 1 -o '1.2,2.2'   <(sort file1) <(sort file2)

我们也可以写一个 REGEXP 或者 awk 一个班轮得到想要的结果，但我觉得上面会更干净。

赞(0）回复(0）举报 2021-06-26

我来回答

bash文本解析配置单元ddl

1条答案

相关问题

热门标签

最新问答