How do I monitor a file containing more than 1000 lines?

[leon24]

Hi all,

I have a log file with many rows. However, I tried to change the settings in /etc/system/local/props.conf

My props.conf is below.

[sap_availability]
MAX_EVENTS = 10000
TRUNCATE = 0

However, I'm still seeing only 234 lines out of the many lines. How do I allow Splunk to index all the lines in the file? I've tried to search the forum for answers and these are settings I've added to props.conf but to no avail.

Thanks all for the help in advance!

Regards

[kristian_kolb · Accepted solution]

MAX_EVENTS refers to the maximum number of lines in a multi-line event, not the number of lines in the log file. TRUNCATE refers to at what point Splunk should cut off individual lines.

Many times, problems like this is related to improper detection of timestamps (which in turn affects the breaking of the log file into individual events). The default behaviour is to break the file when a timestamp is detected.

Consider using the following settings in props.conf

SHOULD_LINEMERGE = false
LINE_BREAKER = some regex for when to break between events
TIME_FORMAT = strptime formatting 
TIME_PREFIX = string that precedes the timestamp
MAX_TIMESTAMP_LOOKAHEAD = number, indicating how far into an event Splunk will look for the timestamp.

Read more in the docs for props.conf and timestamp recognition in general

http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf
http://docs.splunk.com/Documentation/Splunk/latest/Data/HowSplunkextractstimestamps

Posting some sample events may lead to a more qualified answer.

Hope this helps,

K