Getting Data In

How do I monitor a file containing more than 1000 lines?

leon24
Explorer

Hi all,

I have a log file with many rows. However, I tried to change the settings in /etc/system/local/props.conf

My props.conf is below.

[sap_availability]
MAX_EVENTS = 10000
TRUNCATE = 0

However, I'm still seeing only 234 lines out of the many lines. How do I allow Splunk to index all the lines in the file? I've tried to search the forum for answers and these are settings I've added to props.conf but to no avail.

Thanks all for the help in advance!

Regards

Tags (1)
0 Karma
1 Solution

kristian_kolb
Ultra Champion

MAX_EVENTS refers to the maximum number of lines in a multi-line event, not the number of lines in the log file. TRUNCATE refers to at what point Splunk should cut off individual lines.

Many times, problems like this is related to improper detection of timestamps (which in turn affects the breaking of the log file into individual events). The default behaviour is to break the file when a timestamp is detected.

Consider using the following settings in props.conf

SHOULD_LINEMERGE = false
LINE_BREAKER = some regex for when to break between events
TIME_FORMAT = strptime formatting 
TIME_PREFIX = string that precedes the timestamp
MAX_TIMESTAMP_LOOKAHEAD = number, indicating how far into an event Splunk will look for the timestamp.

Read more in the docs for props.conf and timestamp recognition in general

http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf
http://docs.splunk.com/Documentation/Splunk/latest/Data/HowSplunkextractstimestamps

Posting some sample events may lead to a more qualified answer.

Hope this helps,

K

View solution in original post

kristian_kolb
Ultra Champion

MAX_EVENTS refers to the maximum number of lines in a multi-line event, not the number of lines in the log file. TRUNCATE refers to at what point Splunk should cut off individual lines.

Many times, problems like this is related to improper detection of timestamps (which in turn affects the breaking of the log file into individual events). The default behaviour is to break the file when a timestamp is detected.

Consider using the following settings in props.conf

SHOULD_LINEMERGE = false
LINE_BREAKER = some regex for when to break between events
TIME_FORMAT = strptime formatting 
TIME_PREFIX = string that precedes the timestamp
MAX_TIMESTAMP_LOOKAHEAD = number, indicating how far into an event Splunk will look for the timestamp.

Read more in the docs for props.conf and timestamp recognition in general

http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf
http://docs.splunk.com/Documentation/Splunk/latest/Data/HowSplunkextractstimestamps

Posting some sample events may lead to a more qualified answer.

Hope this helps,

K

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...