Getting Data In

how to tell forwarder to wait for a cr/nl

a212830
Champion

Hi,

I have a logfile which is single-line and well structured, but the feed doesn't always write out entire lines. I might get 10 lines (with cr/lf) and then half a line, until the next "batch" of data comes in. As a result, I'm getting events with only half of the data sometimes. How do i tell the forwarder to wait until a cr/lf comes in, and don't process it until that happens?

0 Karma

yannK
Splunk Employee
Splunk Employee

If this is a problem of lines partially written, check the option time_before_close in the inputs.conf
and increase the time to wait before considering a line to be finalized.

http://docs.splunk.com/Documentation/Splunk/6.0/admin/Inputsconf

a212830
Champion

OK. It works, but it almost seems like it goes into "sleep" and then wakes up and reads the file, which, while it works, leaves me behind a batch of entries. Is there a way to tell it to break before the beginning of every line, but only if it contains a timestamp? The file is very well structured - timestamp|field|value|field|value

Here's a sample:

1385823300000|132210|NormalizedMemoryInfo|Free|287534508|Memory|testdevice|Enhanced-MemoryPool: Processor 3012.1

1385823300000|132210|NormalizedMemoryInfo|Utilization|23.67089250345821|Memory|testdevice2|Enhanced-MemoryPool: Processor 3012.1

0 Karma

jtacy
Builder

Please move that config to inputs.conf in the affected stanza and restart the indexer.

0 Karma

a212830
Champion

Tried it, but it was flagged as a possible type error after starting up the indexer.

Here's the props.conf entry:

[snmpinfo]
MAX_TIMESTAMP_LOOKAHEAD = 30
SHOULD_LINEMERGE = true
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = 1
TIME_PREFIX = ^
TIME_FORMAT = %s%3N
time_before_close = 310

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...