仅跟踪更新的文件行

l7mqbcuq  于 2021-06-03  发布在  Flume
关注(0)|答案(2)|浏览(381)

我只想跟踪文件的更新行,而不是整个内容
tail-f/文件路径
显示文件中的所有行。我只需要显示添加到文件中的新行,有人能帮我吗?
例如,我只想在文件中看到更新的行,例如,如果文件有10行tail-f显示终端上的10行如果再添加5行,我应该只能看到新的5行,而不是所有的15行
编辑我已配置flume将日志数据发送到hbase我正在使用“tail-f/path to file”,每次更新文件时都会显示所有行。只有更新的日志数据(例如添加5行)才应发送到hbase,否则会有数据冗余。
向查亚问好

euoag5mw

euoag5mw1#

如果日志事件来自java日志框架,那么我建议使用log4j2flume appender。这将确保最新的事件很快到达Flume。

h79rfbju

h79rfbju2#

我假设那个文件是一个日志文件?
所以,也许你不想想出一种方法来记住上次写的内容,只显示最新的内容,你可能想使用一个日志系统[比如syslogd,或者它的更新版本],并告诉它在文件中记录这两个内容并将其发送到flume?
否则,这里有一个肮脏的黑客:创建一个“shownew.sh”文件,包含:


# try to be as "atomic" as possible: we will all do with a copy of ${1}, to "freeze" time

cp -p "${1}" "${1}.cur"  #very important. "freezes" the state of $1

if [ -f "${1}.old ]; then

   diff "${1}.old" "${1}.cur" | grep '^> ' | sed -e 's/^> //'

else

   cat "${1}.cur" #show the file at the time of invocation

fi

mv -f "${1}.cur" "${1}.old"  #we just showed "${1}.cur" (or the diff between ${1}.cur and a previous ${1}.old=.
  # so we now move that ${1}.cur $^{1}.old, for the next iteration
  #We used a ${1}.cur instead of ${1} because ${1} may be updated at any time, and it's possible we copy a "$1" updated since the display of differences! By using ${1}.cur instead, this won't be a problem

# edit: after the OP's comment he wants to tail -f the file too:

# and now we showed the diffs since $1.old, we continue to display what is new in $1, using tail -f:

# since we showed ${1}.cur (now known as ${1}.old}, $1 may have changed?

diff "${1}" "${1}.old" | grep '^> ' | sed -e 's/> //' 

# and now we tail -f on $1 to show what's incoming, until the user press ctrl+C

tail -n 0 -f "${1} 

# we showed the complete ${1}, this becomes the new ${1}.old

cp "${1}" "${1}.old"

在第一次调用时, shownew.sh /some/file :如果是您第一次调用它,它将显示它的全部内容 /some/file .
每次调用脚本时: shownew.sh /some/file :它将只显示现在在“${1}”中的行和以前在“${1}.old”中没有的行。。。我希望这就是你想要的?

相关问题