用于sqoop deamon日志的regex

7vux5j2d 于 2021-06-01 发布在 Hadoop

关注(0)|答案(1)|浏览(310)

我正在尝试为sqoop日志创建正则表达式。
日志如下：

> Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12- 
1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
18/12/06 07:03:04 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
18/12/06 07:03:05 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
18/12/06 07:03:05 WARN sqoop.ConnFactory: Parameter --driver is set to an explicit driver however appropriate connection manager is not being set (via --connection-manager). Sqoop is going to fall back to org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly which connection manager should be used next time.
18/12/06 07:03:05 INFO manager.SqlManager: Using default fetchSize of 1000
18/12/06 07:03:05 INFO tool.CodeGenTool: Beginning code generation
18/12/06 07:03:06 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM ROH319P4 AS t WHERE 1=0
18/12/06 07:03:06 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/lib/hadoop-mapreduce
Note: /tmp/sqoop-root/compile/92b93a5009481a238e86271708bb80e0/ROH319P4.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
18/12/06 07:03:10 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/92b93a5009481a238e86271708bb80e0/ROH319P4.jar
18/12/06 07:03:10 INFO mapreduce.ImportJobBase: Beginning import of ROH319P4
18/12/06 07:03:10 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
18/12/06 07:03:11 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
18/12/06 07:03:11 INFO client.RMProxy: Connecting to ResourceManager at ip-172-27-88-6.ap-south-1.compute.internal/172.27.88.6:8032
18/12/06 07:03:14 INFO db.DBInputFormat: Using read commited transaction isolation
18/12/06 07:03:14 INFO mapreduce.JobSubmitter: number of splits:1
18/12/06 07:03:14 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1536239303820_0582
18/12/06 07:03:15 INFO impl.YarnClientImpl: Submitted application application_1536239303820_0582
18/12/06 07:03:15 INFO mapreduce.Job: The url to track the job: http://ip-172-27-88-6.ap-south-1.compute.internal:20888/proxy/application_1536239303820_0582/
18/12/06 07:03:15 INFO mapreduce.Job: Running job: job_1536239303820_0582
18/12/06 07:03:22 INFO mapreduce.Job: Job job_1536239303820_0582 running in uber mode : false
18/12/06 07:03:22 INFO mapreduce.Job:  map 0% reduce 0%
18/12/06 07:03:28 INFO mapreduce.Job:  map 100% reduce 0%
18/12/06 07:03:28 INFO mapreduce.Job: Job job_1536239303820_0582 completed successfully
18/12/06 07:03:28 INFO mapreduce.Job: Counters: 30

File System Counters
    FILE: Number of bytes read=0
    FILE: Number of bytes written=189523
    FILE: Number of read operations=0
    FILE: Number of large read operations=0
    FILE: Number of write operations=0
    HDFS: Number of bytes read=87
    HDFS: Number of bytes written=4997
    HDFS: Number of read operations=4
    HDFS: Number of large read operations=0
    HDFS: Number of write operations=2
Job Counters 
    Launched map tasks=1
    Other local map tasks=1
    Total time spent by all maps in occupied slots (ms)=742431
    Total time spent by all reduces in occupied slots (ms)=0
    Total time spent by all map tasks (ms)=4057
    Total vcore-milliseconds taken by all map tasks=4057
    Total megabyte-milliseconds taken by all map tasks=23757792
Map-Reduce Framework
    Map input records=38
    Map output records=38
    Input split bytes=87
    Spilled Records=0
    Failed Shuffles=0
    Merged Map outputs=0
    GC time elapsed (ms)=62
    CPU time spent (ms)=1260
    Physical memory (bytes) snapshot=295165952
    Virtual memory (bytes) snapshot=7060725760
    Total committed heap usage (bytes)=340262912
File Input Format Counters 
    Bytes Read=0
File Output Format Counters 
    Bytes Written=4997
18/12/06 07:03:28 INFO mapreduce.ImportJobBase: Transferred 4.8799 KB in 17.1259 seconds (291.781 bytes/sec)
18/12/06 07:03:28 INFO mapreduce.ImportJobBase: Retrieved 38 records.

我尝试创建的正则表达式：
^（\d{4}/\d{2}/\d{2}）\s+（\d{2}.\d{2}.\d{2}）\s+（\s+）\s+（\s+）\s+（\s+）\s+（*）$
所以我的目标是获取如下格式的行：
18/12/06 07:03:06 info orm.compilationmanager:hadoop\u mapred\u home是/usr/lib/hadoop mapreduce
有人能帮我弄清楚正则表达式吗？

hadoop sqoop regex

来源：https://stackoverflow.com/questions/53699495/regex-for-sqoop-deamon-logs

1条答案

按热度按时间

jutyujz01#

你的正则表达式几乎是正确的，除了，
第一 \d{4} 应该是的 \d{2} 因为年份只有两位数。如果你认为它可以有4到2个数字，你可以使用
\d{2,4} / 字符需要像这样在regex中转义 \/ 而不是 . 你需要使用 : 因为你的时间是分开的。尽管 . 将匹配任何字符，甚至冒号，但我认为你应该更好地精确和使用 : 所以你更新了regex，它成功地匹配了日志文件中的行，

^(\d{2}\/\d{2}\/\d{2})\s+(\d{2}:\d{2}:\d{2})\s+(\S+)\s+(\S+)\s+(.*)$

在这里检查
如果你不想要捕获组，非捕获组会更好，

^(?:\d{2}\/\d{2}\/\d{2})\s+(?:\d{2}:\d{2}:\d{2})\s+(?:\S+)\s+(?:\S+)\s+(?:.*)$

没有组

赞(0）回复(0）举报 2021-06-01

我来回答

用于sqoop deamon日志的regex

1条答案

相关问题

热门标签

最新问答