Getting Data In

forwarder creating multiline event from big file transferred via rsycn on slow channel

gots
Path Finder

We have big file with events in each line of file. Every minute file transfers via rsync to forwarder with new events.
Events format:

timestamp="XXXX" login="YYY" action="ZZZ"

Size of file about 10Mb, with 15000 lines.
Sometimes (1-2 times per minute) forwarder compile one event from multiple (usually 2) lines.
Configs from forwarder:
inputs.conf:

[monitor:///file.log]
disabled = false
followTail = 1
index = f_logs
sourcetype = f_logs
multiline_event_extra_waittime = true
time_before_close = 10

props.conf:

[f_logs]
TIME_PREFIX = timestamp=\"
TIME_FORMAT = %s
SHOULD_LINEMERGE = true
MAX_EVENTS = 1

What i have to do to solve this problem?

0 Karma

anand_singh17
Path Finder

Two ways to solve:

option 1-
Let the forwarder create multi-line. Use "transaction" keyword to group similar based on common field you prepare for.

Option2-
try using trail method for the log. Let rsync write to file.log, but use another crontab to send data to another file - rysnclistner.log.
Now listen to this log file.

Reason:
1. Changes the frequency of reading data for fowarder
2. You can use it in form of Queue and data purges as new entries keeps come. (FIFO)

Hope any of the option can help. But most important, we also would like to see some sample _raw to be able to decide.

Thank You

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...