Light forwarder events coming in as a bundle instead of individual

[anantshah]

Hello,

We are on Splunk 4.1.3. We have configured light forwarders on 4 windows 2008 r2 servers to send IIS logs to indexer. The events are coming in on the indexer but they are bundled to gether. For example, multiple entries (close to 200) from the IIS log come in as 1 event instead of a event for each entry in the log file.

[rrich]

For what it's worth, i ran into this exact issue by accidentally pointing the forwarder at a TCP input port rather than a receiver port.

Biggest clue is if you see metadata (e.g. sourcetype strings, log filenames) and nulls encoded as \x00 in the events.

[Simeon · Accepted solution]

The problem you are having resembles a line breaking issue. Depending on how you have configured your IIS logs, you may need to create a regex to properly break the events. In most scenarios, Splunk will automatically recognize where to break events as well as merge them. You can review more detail on how to configure the line breaking here:

http://www.splunk.com/base/Documentation/latest/Admin/Indexmulti-lineevents

[gkanapathy]

basically, there's a problem with the out of the box IIS sourcetype. pick a different sourcetype name for your IIS logs and the problem will probably go away. Alternatively, add CHECK_FOR_HEADER = false and priority = 101 to the sourcetype definition on the LWF.

[anantshah]

The issue was the events were coming in as iis-2 and iis3 on the indexer so the default props for iss were not getting applied which automatically deal with the multilines.

The issue was resolved by configuring a props.conf file on the light forwarder that ignores the file header for iis source type.

[anantshah]

The issue i am facing is its not following the default rule which is an event for every line \r\n. The same light forwarder is forwarding to a second instance of splunk (4.1.2) without any issues.

[ziegfried]

Sounds like there is an issue with line-merging or timestamp recognition for the monitored files.

What's the sourcetype of those events?
How do they look like?