我一直在使用logstash管道进行测试,以处理具有以下格式的oracle多行审核日志:
Audit file /u01/app/oracle/admin/DEVINST/adump/DEVINST_ora_15460_20201001230100743853143795.aud
Oracle Database 12c Standard Edition Release 12.2.0.1.0 - 64bit DEV
Build label: RDBMS_12.2.0.1.0_LINUX.X64_170125
ORACLE_HOME: /u01/app/oracle/product/12.2.0/dbhome_1
System name: Linux
Node name: testdevserver
Release: 3.10.0-862.14.4.el7.x86_64
Version: #1 SMP Fri Sep 21 09:07:21 UTC 2018
Machine: x86_64
Instance name: DEVINST
Redo thread mounted by this instance: 1
Oracle process number: 57
Unix process pid: 15460, image: oracle@testdevserver (TNS V1-V3)
Thu Oct 1 23:01:00 2020 +00:00
LENGTH : '275'
ACTION :[7] 'CONNECT'
DATABASE USER:[1] '/'
PRIVILEGE :[6] 'SYSDBA'
CLIENT USER:[9] 'test_user'
CLIENT TERMINAL:[5] 'pts/0'
STATUS:[1] '0'
DBID:[10] '1762369616'
SESSIONID:[10] '4294967295'
USERHOST:[21] 'testdevserver'
CLIENT ADDRESS:[0] ''
ACTION NUMBER:[3] '100'
Thu Oct 1 23:01:00 2020 +00:00
LENGTH : '296'
ACTION :[29] 'SELECT STATUS FROM V$INSTANCE'
DATABASE USER:[1] '/'
PRIVILEGE :[6] 'SYSDBA'
CLIENT USER:[9] 'test_user'
CLIENT TERMINAL:[5] 'pts/0'
STATUS:[1] '0'
DBID:[10] '1762369616'
SESSIONID:[10] '4294967295'
USERHOST:[21] 'testdevserver'
CLIENT ADDRESS:[0] ''
ACTION NUMBER:[1] '3'
我的 /etc/logstash/conf.d/25-filter.conf
:
filter {
grok {
match => { "message" => "(?<day>^[A-Za-z]{3}) (?<month>[A-Za-z]{3}) (?<monthday>[0-9]{2}) (?<hour>[0-9]{2}):(?<min>[0-9]{2}):(?<sec>[0-9]{2}) (?<year>[0-9]{4}) %{GREEDYDATA:message}" }
overwrite => [ "message" ]
add_tag => [ "oracle_audit" ]
}
grok {
match => { "ACTION :\[[0-9]*\] '(?<ora_audit_action>.*)'.*DATABASE USER:\[[0-9]*\] '(?<ora_audit_dbuser>.*)'.*PRIVILEGE :\[[0-9]*\] '(?<ora_audit_priv>.*)'.*CLIENT USER:\[[0-9]*\] '(?<ora_audit_osuser>.*)'.*CLIENT TERMINAL:\[[0-9]*\] '(?<ora_audit_term>.*)'.*STATUS:\[[0-9]*\] '(?<ora_audit_status>.*)'.*DBID:\[[0-9]*\] '(?<ora_audit_dbid>.*)'.*SESSIONID:\[[0-9]*\] '(?<ora_audit_sessionid>.*)'.*USERHOST:\[[0-9]*\] '(?<ora_audit_dbhost>.*)'.*CLIENT ADDRESS:\[[0-9]*\] '(?<ora_audit_clientaddr>.*)'.*ACTION NUMBER:\[[0-9]*\] '(?<ora_audit_actionnum>.*)'" }
}
grok {
match => { "source" => [ ".*/[a-zA-Z0-9_#$]*_[a-z0-9]*_(?<ora_audit_derived_pid>[0-9]*)_[0-9]*\.aud" ] }
}
mutate {
add_field => { "timestamp" => "%{year}-%{day}-%{monthday} %{hour}:%{min}:%{sec}" }
}
date {
locale => "en"
match => [ "timestamp", "YYYY-MMM-dd HH:mm:ss" ]
}
mutate {
remove_field => [ "timestamp", "year", "day", "monthday", "hour", "min", "sec" ]
}
}
我的 /etc/logstash/conf.d/000-file-in.conf
文件:
input {
file {
path => [ "/tmp/testora" ]
start_position => "beginning"
codec => multiline
{
pattern => "^[A-Za-z]{3} [A-Za-z]{3} [0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2} [0-9]{4}"
negate => "true"
what => "previous"
}
}
}
然后我通过运行: /usr/share/logstash/bin/logstash -r -f /etc/logstash/conf.d/
:
....
[INFO ] 2020-10-05 11:16:30.656 [Converge PipelineAction::Reload<main>] reload - Reloading pipeline {"pipeline.id"=>:main}
[INFO ] 2020-10-05 11:16:30.662 [Converge PipelineAction::Reload<main>] observingtail - QUIT - closing all files and shutting down.
{
"ora_audit_dbid" => "1762369616",
"ora_audit_actionnum" => "3",
"ora_audit_sessionid" => "4294967295",
"@version" => "1",
"tags" => [
[0] "multiline",
[1] "_grokparsefailure",
[2] "_dateparsefailure"
],
"path" => "/tmp/testora",
"ora_audit_action" => [
[0] "275'\nACTION :[7] 'CONNECT'\nDATABASE USER:[1] '/'\nPRIVILEGE :[6] 'SYSDBA'\nCLIENT USER:[9] 'test_user'\nCLIENT TERMINAL:[5] 'pts/0'\nSTATUS:[1] '0'\nDBID:[10] '1762369616'\nSESSIONID:[10] '4294967295'\nUSERHOST:[21] 'testdevserver'\nCLIENT ADDRESS:[0] ''\nACTION NUMBER:[3] '100'\nThu Oct 1 23:01:00 2020 +00:00\nLENGTH : '296",
[1] "SELECT STATUS FROM V$INSTANCE"
],
"@timestamp" => 2020-10-05T01:16:30.889Z,
"ora_audit_priv" => "SYSDBA",
"message" => "Audit file /u01/app/oracle/admin/DEVINST/adump/DEVINST_ora_15460_20201001230100743853143795.aud\nOracle Database 12c Standard Edition Release 12.2.0.1.0 - 64bit DEV\nBuild label: RDBMS_12.2.0.1.0_LINUX.X64_170125\nORACLE_HOME: /u01/app/oracle/product/12.2.0/dbhome_1\nSystem name: Linux\nNode name: testdevserver\nRelease: 3.10.0-862.14.4.el7.x86_64\nVersion: #1 SMP Fri Sep 21 09:07:21 UTC 2018\nMachine: x86_64\nInstance name: DEVINST\nRedo thread mounted by this instance: 1\nOracle process number: 57\nUnix process pid: 15460, image: oracle@testdevserver (TNS V1-V3)\nThu Oct 1 23:01:00 2020 +00:00\nLENGTH : '275'\nACTION :[7] 'CONNECT'\nDATABASE USER:[1] '/'\nPRIVILEGE :[6] 'SYSDBA'\nCLIENT USER:[9] 'test_user'\nCLIENT TERMINAL:[5] 'pts/0'\nSTATUS:[1] '0'\nDBID:[10] '1762369616'\nSESSIONID:[10] '4294967295'\nUSERHOST:[21] 'testdevserver'\nCLIENT ADDRESS:[0] ''\nACTION NUMBER:[3] '100'\nThu Oct 1 23:01:00 2020 +00:00\nLENGTH : '296'\nACTION :[29] 'SELECT STATUS FROM V$INSTANCE'\nDATABASE USER:[1] '/'\nPRIVILEGE :[6] 'SYSDBA'\nCLIENT USER:[9] 'test_user'\nCLIENT TERMINAL:[5] 'pts/0'\nSTATUS:[1] '0'\nDBID:[10] '1762369616'\nSESSIONID:[10] '4294967295'\nUSERHOST:[21] 'testdevserver'\nCLIENT ADDRESS:[0] ''\nACTION NUMBER:[1] '3'",
"ora_audit_dbhost" => "testdevserver",
"host" => "myhost",
"ora_audit_dbuser" => "/",
"ora_audit_osuser" => "test_user",
"ora_audit_term" => "pts/0",
"ora_audit_status" => "0"
}
不幸的是,这不是我所期望的。不知何故,它没有正确地分块消息和解析消息。我期待的是:
"ora_audit_action" => "CONNECT",
"ora_audit_dbuser" => "/",
"ora_audit_dbid" => "1762369616",
"ora_audit_status" => "0",
"ora_audit_osuser" => "test_user",
"ora_audit_priv" => "SYSDBA",
"ora_audit_term" => "pts/0",
"ora_audit_sessionid" => "4294967295",
"ora_audit_dbhost" => "testdevserver",
"ora_audit_clientaddr" => "",
"ora_audit_actionnum" => "3",
"host" => "myhost",
"@version" => "1",
"@timestamp" => 2020-10-05T01:16:30.889Z,
"message" => "Thu Oct 1 23:01:00 2020 +00:00\nLENGTH : '275'\nACTION :[7] 'CONNECT'\nDATABASE USER:[1] '/'\nPRIVILEGE :[6] 'SYSDBA'\nCLIENT USER:[9] 'test_user'\nCLIENT TERMINAL:[5] 'pts/0'\nSTATUS:[1] '0'\nDBID:[10] '1762369616'\nSESSIONID:[10] '4294967295'\nUSERHOST:[21] 'testdevserver'\nCLIENT ADDRESS:[0] ''\nACTION NUMBER:[3] '100'\nThu Oct 1 23:01:00 2020 +00:00\nLENGTH : '296'\nACTION :[29] 'SELECT STATUS FROM V$INSTANCE'\nDATABASE USER:[1] '/'\nPRIVILEGE :[6] 'SYSDBA'\nCLIENT USER:[9] 'test_user'\nCLIENT TERMINAL:[5] 'pts/0'\nSTATUS:[1] '0'\nDBID:[10] '1762369616'\nSESSIONID:[10] '4294967295'\nUSERHOST:[21] 'testdevserver'\nCLIENT ADDRESS:[0] ''\nACTION NUMBER:[1] '3'"
我看到了其中的一些信息,但是我也希望它能吐出2个“大块”的消息(与模式匹配) ^[A-Za-z]{3} [A-Za-z]{3} [0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2} [0-9]{4}
作为这条线的起点 Thu Oct 1 23:01:00 2020 +00:00
)-相反,它看起来好像只看到一个?我想我的模式匹配可能是这里的问题,如果有人能提供任何提示,谢谢。
另外,我不确定是什么导致了这些错误 [1] "_grokparsefailure", [2] "_dateparsefailure"
-显然,这不是解析的东西很正确,但我只是不知道如何去做。
帮助:(
谢谢,j
3条答案
按热度按时间jbose2ul1#
我认为下面的模式应该适用于您的多行编解码器,并给出所需的分组。
此外,下面的格罗克过滤器应该给你的多行所需的故障。您可能需要根据喜好重命名密钥。
管道输出样本如下
mqxuamgl2#
找到下面匹配的grok模式
Thu Oct 1 23:01:00 2020 +00:00
```(?%{DAY} %{MONTH} %{MONTHDAY} %{TIME})
zvokhttg3#
根据您的要求,您必须在第二个grok过滤器中修改lengh和action参数的grok模式。您正在创建相同的字段或audit操作,这样它将显示数据并形成一个数组,以便更好地为这两个参数创建单独的字段
代替删除字段,您可以使用prune filter将必要的字段列入白名单。
在您的代码中,您正在分离时间戳并对其进行重组,但在过滤器部分,您正在删除它并使用default@timestamp字段,如果您没有要求,那么您可以删除不必要的代码。