logstash 如何使filebeat配置从文件中收集特定行

nwnhqdif  于 2023-11-15  发布在  Logstash
关注(0)|答案(1)|浏览(132)

我有一个文件日志和甲骨文,我需要收集整个块,而不是整个文件。该块是从动作字段到动作数
日志文件如下

Thu Nov  9 10:20:24 2023 +01:00
LENGTH : '373'
ACTION :[122] 'select 'export nls_nchar_characterset="'||value||'"' from nls_database_parameters where parameter='NLS_NCHAR_CHARACTERSET''
DATABASE USER:[1] '/'
PRIVILEGE :[6] 'SYSDBA'
CLIENT USER:[6] 'oracle'
CLIENT TERMINAL:[0] ''
STATUS:[1] '0'
DBID:[10] '3256053857'
SESSIONID:[10] '4294967295'
USERHOST:[12] 's01vl9926909'
CLIENT ADDRESS:[0] ''
ACTION NUMBER:[1] '3'

Thu Nov  9 10:20:24 2023 +01:00
LENGTH : '1575'
ACTION :[1323] 'select 'export db_patchset_new="'||max_ver||'"' from (
select length(regexp_replace(substr(description,0,decode(instr(description,'('),0,length(description),instr(description,'('))),'[^0.0-9]', '')) max_length,
       max(regexp_replace(substr(description,0,decode(instr(description,'('),0,length(description),instr(description,'('))),'[^0.0-9]', '')) max_ver
from   (select description from ( select da.description, da.action_time apply, nvl(dr.action_time, da.action_time-1) rollback
                                  from (select description, max(action_time) action_time from dba_registry_sqlpatch where action='APPLY'    group by description) da,
                                       (select description, max(action_time) action_time from dba_registry_sqlpatch where action='ROLLBACK' group by description) dr
                                  where da.description=dr.description(+)
                                )
        where apply > rollback
       )
where  (upper(description) like '%DATABASE%PATCH%' or upper(description) like '%DATABASE%RELEASE%') and upper(description) not like '%JAVA%' and upper(description) not like '%JVM%'
group by length(regexp_replace(substr(description,0,decode(instr(description,'('),0,length(description),instr(description,'('))),'[^0.0-9]', ''))
order by 1 desc
)
where rownum=1'
DATABASE USER:[1] '/'
PRIVILEGE :[6] 'SYSDBA'
CLIENT USER:[6] 'oracle'
CLIENT TERMINAL:[0] ''
STATUS:[1] '0'
DBID:[10] '3256053857'
SESSIONID:[10] '4294967295'
USERHOST:[12] 's01vl9926909'
CLIENT ADDRESS:[0] ''
ACTION NUMBER:[1] '3'

字符串
我使用了include_lines模式来收集这些特定的行,但我这里的问题是,在字段“ACTION”中,有时会有包含长查询的操作,而我的conf只收集第一行,而不是整个查询。
这里的例子我的事件不收集整个查询它停止在字

{"@timestamp":"2023-11-09T13:34:21.491Z","@metadata":{"beat":"filebeat","type":"_doc","version":"7.17.0"},"message":"ACTION:'select 'export db_patchset_new="'||max_ver||'"' from ('","fields":{"env":"staging"},"host":{"name":"s01vl9926909"},"event":{"timezone":"+01:00"},"log":{"offset":811,"file":{"path":"/apps/oracle/diag/rdbms/q08775kp1/Q08775KP10/audit/Q08775KP10_ora_29807_20231109143420100545459814.aud"}},"input":{"type":"log"},"ecs":{"version":"1.12.0"}


在第一次我的文件input_Oracle.yml是像下面;它工作正常,但问题是动作字段
我的输入文件配置在第一时间工作正常

enabled: true
  tags: ["linux-Oracle"]
  #ignore_older: 4h
  close_inactive: 30s

  paths:
   /apps/oracle/diag/rdbms/*/*/audit/*.aud

  include_lines: ['^LENGTH :', '^ACTION :', '^DATABASE USER:', '^PRIVILEGE :', '^CLIENT USER:', '^CLIENT TERMINAL:', '^STATUS:', '^DBID:', '^SESSIONID:', '^USERHOST:', '^CLIENT ADDRESS:', '^ACTION NUMBER:']


然后我尝试“多行消息”,但它不适合我

type: log
  enabled: true
  tags: ["linux-Oracle"]
  #ignore_older: 4h
  close_inactive: 30s

  paths:
    - /apps/oracle/diag/rdbms/*/*/audit/*.aud

  multiline:
    pattern: '^ACTION :'
    negate: true
    match: after

  include_lines: ['^LENGTH :', '^DATABASE USER:', '^PRIVILEGE :', '^CLIENT USER:', '^CLIENT TERMINAL:', '^STATUS:', '^DBID:', '^SESSIONID:', '^USERHOST:', '^CLIENT ADDRESS:', '^ACTION NUMBER:']`


有没有办法做到这一点的合并线!!

d7v8vwbk

d7v8vwbk1#

Tldr;

你使用多行解析器确实是正确的,但是我认为你做了一个轻微的配置错误。

解决方案:

我相信这个能帮到你。
我正在匹配一个大写字母开头的单词。

filebeat.inputs:
- type: filestream
  id: srt
  paths:
    - /usr/share/filebeat/data.log
  include_lines: ['^LENGTH :', '^ACTION :', '^DATABASE USER:', '^PRIVILEGE :', '^CLIENT USER:', '^CLIENT TERMINAL:', '^STATUS:', '^DBID:', '^SESSIONID:', '^USERHOST:', '^CLIENT ADDRESS:', '^ACTION NUMBER:']
  parsers:
  - multiline:
      type: pattern
      pattern: '^[A-Z]+'
      negate: true
      match: after

output.console:
  pretty: true

字符串
它给了我以下对象:

{
  "@timestamp": "2023-11-09T14:19:08.184Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "8.6.2"
  },
  "message": "LENGTH : '373'"
}
{
  "@timestamp": "2023-11-09T14:19:08.184Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "8.6.2"
  },
  "message": "ACTION :[122] 'select 'export nls_nchar_characterset=\"'||value||'\"' from nls_database_parameters where parameter='NLS_NCHAR_CHARACTERSET''"
}
{
  "@timestamp": "2023-11-09T14:19:08.184Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "8.6.2"
  },
  "message": "DATABASE USER:[1] '/'"
}
{
  "@timestamp": "2023-11-09T14:19:08.184Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "8.6.2"
  },
  "message": "PRIVILEGE :[6] 'SYSDBA'"
}
{
  "@timestamp": "2023-11-09T14:19:08.184Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "8.6.2"
  },
  "message": "CLIENT USER:[6] 'oracle'"
}
{
  "@timestamp": "2023-11-09T14:19:08.184Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "8.6.2"
  },
  "message": "CLIENT TERMINAL:[0] ''"
}
{
  "@timestamp": "2023-11-09T14:19:08.184Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "8.6.2"
  },
  "message": "STATUS:[1] '0'"
}
{
  "@timestamp": "2023-11-09T14:19:08.184Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "8.6.2"
  },
  "message": "DBID:[10] '3256053857'"
}
{
  "@timestamp": "2023-11-09T14:19:08.184Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "8.6.2"
  },
  "message": "SESSIONID:[10] '4294967295'"
}
{
  "@timestamp": "2023-11-09T14:19:08.184Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "8.6.2"
  },
  "message": "USERHOST:[12] 's01vl9926909'"
}
{
  "@timestamp": "2023-11-09T14:19:08.184Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "8.6.2"
  },
  "message": "CLIENT ADDRESS:[0] ''"
}
{
  "@timestamp": "2023-11-09T14:19:08.184Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "8.6.2"
  },
  "message": "ACTION NUMBER:[1] '3'\n"
}
{
  "@timestamp": "2023-11-09T14:19:08.184Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "8.6.2"
  },
  "message": "LENGTH : '1575'"
}
{
  "@timestamp": "2023-11-09T14:19:08.184Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "8.6.2"
  },
  "message": "ACTION :[1323] 'select 'export db_patchset_new=\"'||max_ver||'\"' from (\nselect length(regexp_replace(substr(description,0,decode(instr(description,'('),0,length(description),instr(description,'('))),'[^0.0-9]', '')) max_length,\n       max(regexp_replace(substr(description,0,decode(instr(description,'('),0,length(description),instr(description,'('))),'[^0.0-9]', '')) max_ver\nfrom   (select description from ( select da.description, da.action_time apply, nvl(dr.action_time, da.action_time-1) rollback\n                                  from (select description, max(action_time) action_time from dba_registry_sqlpatch where action='APPLY'    group by description) da,\n                                       (select description, max(action_time) action_time from dba_registry_sqlpatch where action='ROLLBACK' group by description) dr\n                                  where da.description=dr.description(+)\n                                )\n        where apply > rollback\n       )\nwhere  (upper(description) like '%DATABASE%PATCH%' or upper(description) like '%DATABASE%RELEASE%') and upper(description) not like '%JAVA%' and upper(description) not like '%JVM%'\ngroup by length(regexp_replace(substr(description,0,decode(instr(description,'('),0,length(description),instr(description,'('))),'[^0.0-9]', ''))\norder by 1 desc\n)\nwhere rownum=1'"
}
{
  "@timestamp": "2023-11-09T14:19:08.184Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "8.6.2"
  },
  "message": "DATABASE USER:[1] '/'"
}
{
  "@timestamp": "2023-11-09T14:19:08.184Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "8.6.2"
  },
  "message": "PRIVILEGE :[6] 'SYSDBA'"
}
{
  "@timestamp": "2023-11-09T14:19:08.184Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "8.6.2"
  },
  "message": "CLIENT USER:[6] 'oracle'"
}
{
  "@timestamp": "2023-11-09T14:19:08.184Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "8.6.2"
  },
  "message": "CLIENT TERMINAL:[0] ''"
}
{
  "@timestamp": "2023-11-09T14:19:08.184Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "8.6.2"
  },
  "message": "STATUS:[1] '0'"
}
{
  "@timestamp": "2023-11-09T14:19:08.185Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "8.6.2"
  },
  "message": "DBID:[10] '3256053857'"
}
{
  "@timestamp": "2023-11-09T14:19:08.185Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "8.6.2"
  },
  "message": "SESSIONID:[10] '4294967295'"
}
{
  "@timestamp": "2023-11-09T14:19:08.185Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "8.6.2"
  },
  "message": "USERHOST:[12] 's01vl9926909'"
}
{
  "@timestamp": "2023-11-09T14:19:08.185Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "8.6.2"
  },
  "message": "CLIENT ADDRESS:[0] ''"
}

相关问题