line breaking help

[zachvida]

957978 11:23:33 (INTEL) IN: "IFBFE4F44" user@hostlx8.domain  $  
957979 11:24:07 (MLM) IN: "MATLAB" user@hostlx1.domain  $  
957980 11:24:07 (MLM) IN: "Statistics_Toolbox" user@hostx1.domain  $  
957981 11:24:17 (MLM) DENIED: "MATLAB" user@hostx1.domain  (User/host on EXCLUDE list for feature. (-38,348))$  
957982 11:24:17 (MLM) OUT: "MATLAB" user@hostx1.domain  $  
957983 11:27:05 (MLM) DENIED: "Statistics_Toolbox" user@hostx1.domain  (User/host on EXCLUDE list for feature. (-38,348))$  
957984 11:27:05 (MLM) OUT: "Statistics_Toolbox" user@hostx1.domain  $  
957985 11:27:14 (INTEL) OUT: "IFBFE4F44" user@hostlx36.domain  $  
957986 11:27:14 (INTEL) OUT: "FCompL" user@hostlx36.domain  $  
957987 11:27:14 (INTEL) IN: "FCompL" user@hostlx36.domain  $
957988 11:27:14 (INTEL) IN: "IFBFE4F44" user@hostlx36.domain  $

Using the list mode of vim to display hidden characters hence the $.
:set list

/splunk/etc/system/local/props.conf

[license-logs]
SHOULD_LINEMERGE = false
LINE_BREAKER = (\n)

(We also tried, (\s*\n) )

Results in events in splunk being indexed as:

957978 11:23:33 (INTEL) IN: "IFBFE4F44" user@hostlx8.domain  $  
957979 11:24:07 (MLM) IN: "MATLAB" user@hostlx1.domain  $  
957980 11:24:07 (MLM) IN: "Statistics_Toolbox" user@hostx1.domain  $  
957981 11:24:17 (MLM) DENIED: "MATLAB" user@hostx1.domain  (User/host on EXCLUDE list for feature. (-38,348))$  
957982 11:24:17 (MLM) OUT: "MATLAB" user@hostx1.domain  $ 
957983 11:27:05 (MLM) DENIED: "Statistics_Toolbox" user@hostx1.domain  (User/host on EXCLUDE list for feature. (-38,348))$  
957984 11:27:05 (MLM) OUT: "Statistics_Toolbox" user@hostx1.domain  $  
957985 11:27:14 (INTEL) OUT: "IFBFE4F44" user@hostlx36.domain  $  
957986 11:27:14 (INTEL) OUT: "FCompL" user@hostlx36.domain  $  
957987 11:27:14 (INTEL) IN: "FCompL" user@hostlx36.domain  $
957988 11:27:14 (INTEL) IN: "IFBFE4F44" user@hostlx36.domain  $

EDIT:The events are being grouped on timestamp. I want each line to always be its own event.

EDIT2: As suggested I put LINE_BREAKER = ([\r\n]+) this still didn't put each line in its own event. Splunk was restarted

[zachvida · Accepted solution]

So the solution turned out be that props.conf needed the sourcetype of my input not the index. Along with some time format settings:
[sourcetype]
SHOULD_LINEMERGE = false
TIME_PREFIX = ^
MAX_TIMESTAMP_LOOKAHEAD = 8
TIME_FORMAT = %H:%M:%S
TZ = -0400

[Gilberto_Castil]

The reason the events are not showing properly may be because the data has not changed since the first time it was indexed. Splunk is defensive by design and that means that once data has been indexed, it becomes immutable –it cannot be changed.

A Splunk monitor keeps track the offset where it last read a file, as well as other tidy bits like the inode and modification time. If the file is live, meaning it is being populated with data, the monitor should continue to update from the last known offset and then the events should be properly broken at a normal break line going forward. On the other hand, if the file is at rest, meaning it is no longer being updated with new data, you will not see a change.

You have two good options to re-index the file:

1. You can create a test index and then modify your monitor input to point to that new repository. That will re-index the data. This is, by the way, a good practice in order to test your data inputs. If you were to erase the complete context of the index, you would not be bound by the contents of the index –knowing anything stored in there can be considered volatile.

2. In testing, I often open the file being monitored and add a single blank line at the very top of the file and then save. That does two things: a) it fractures the read offset of the file monitor and b) changes the modification time of the file. The upshot benefit is that the data contents of the file are re-indexed.

There are other things you can do to get the data to re-index but anything else could have undesired results.

If these recommendations do not work, you may need more precision on the regular expression for your line breaker.

[Gilberto_Castil]

The default line breaker in Splunk is ([\r\n]+) and that works in most cases. In this particular case, you may not need to specifically call out for this as a regular expression. It may be that all you need in the props.conf entry is the following:

[license-logs]
SHOULD_LINEMERGE = false

If you wanted to ensure of this, you could add a line breaker expression for good measure. That is, of course, a redundancy since that is the default line breaker when the SHOULD_LINEMERGE property is set to false.

[license-logs]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)

One thing to note is that if you fat finger the LINE_BREAKER entry, you will get a consolidated record with all the lines -as you may have noted above. You see, Splunk is looking to match that fat-fingered, erroneous regular expression and will attempt to consolidate until a match is found, or the default size of record is reached.

[Ayn]

Yes? What is the question?