regex 如何编写正则表达式从stderr日志文件中提取hadoopmr计数器数据

3zwtqj6y  于 2023-02-10  发布在  Hadoop
关注(0)|答案(1)|浏览(246)

如何编写正则表达式从stderr日志文件中提取hadoopMR计数器数据,
如何查找具有\t\t线对数据所有\t线,我编写了一个规则(re.findall(r'(\t[a-zA-Z\s]+)\n(.*?)\n\t\w+', text, re.S|re.M)),但它不正确

这是stderr日志示例

23/01/16 14:26:13 INFO mortbay.log: Conf is not init.
23/01/16 14:26:14 INFO mapreduce.Job: Counters: 246
    File System Counters
        FILE: Number of bytes read=104971581500
        FILE: Number of bytes written=287906526786
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=758223470025
        HDFS: Number of bytes written=97994290043
        HDFS: Number of read operations=24275
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=2000
        VIEWFS: Number of bytes read=0
        VIEWFS: Number of bytes written=0
        VIEWFS: Number of read operations=0
        VIEWFS: Number of large read operations=0
        VIEWFS: Number of write operations=0
    Job Counters 
        Killed map tasks=3
        Killed reduce tasks=2
        Launched map tasks=6427
        Launched reduce tasks=1002
        Other local map tasks=33
        Data-local map tasks=3746
        Rack-local map tasks=2648
        Total time spent by all maps in occupied slots (ms)=358061940
        Total time spent by all reduces in occupied slots (ms)=858021936
        Total time spent by all map tasks (ms)=119353980
        Total time spent by all reduce tasks (ms)=107252742
        Total vcore-milliseconds taken by all map tasks=119353980
        Total vcore-milliseconds taken by all reduce tasks=107252742
        Total megabyte-milliseconds taken by all map tasks=305546188800
        Total megabyte-milliseconds taken by all reduce tasks=878614462464
    Map-Reduce Framework
        Map input records=30951997
        Map output records=30951997
        Shuffled Maps =6425000
        Failed Shuffles=46
        Merged Map outputs=6425000
    File Input Format Counters 
        Bytes Read=0
    File Output Format Counters 
        Bytes Written=0
23/01/16 14:26:14 INFO streaming.StreamJob: Output directory:
+ [[ 0 -ne 0 ]]
+ exit 0
omqzjyyz

omqzjyyz1#

您可以匹配以单个制表符开头的行,后跟以2个制表符和一个非空格字符开头的一行或多行:
您也只能使用re.M

^\t[a-zA-Z].*(?:\n\t\t\S.*)+

参见regex101 demo
如果匹配2个制表符后行的其余部分也可以为空,则只能为.*

^\t[a-zA-Z].*(?:\n\t\t.*)+

参见另一个regex101 demo

相关问题