I am indexing a file of single line log events and some lines are getting chunked together into one event. Trying to figure out how to correct it.
Here's some example log data that is getting indexed together in each event:
a1-server1 App-> 2011-10-05 12:35:23 - a1-server1 - [99.99.99.99] user3(Active Directory Users)[gg-group] - Key Exchange number 23 occured for user with NCIP 192.168.212.63
a1-server1 App-> 2011-10-05 12:35:23 - a1-server1 - [99.99.99.99] user4(Active Directory Users)[gg-group] - Key Exchange number 5 occured for user with NCIP 192.168.212.221
a1-server1 App-> 2011-10-05 12:35:23 - a1-server1 - [99.99.99.99] user5(Active Directory Users)[gg-group] - Key Exchange number 2 occured for user with NCIP 192.168.212.193
a1-server1 App-> 2011-10-05 12:33:46 - a1-server1 - [99.99.99.99] auser2(Active Directory Users)[gg-group] - Key Exchange number 29 occured for user with NCIP 192.168.212.139
a1-server1 App-> 2011-10-05 12:33:46 - a1-server1 - [99.99.99.99] auser1(Active Directory Users)[gg-group] - Key Exchange number 16 occured for user with NCIP 192.168.212.108
a1-server1 App-> 2011-10-05 12:28:43 - a1-server1 - [99.99.99.99] auser(Active Directory Users)[gg-group] - Logout from 99.99.99.99 (session:ff6f38ff)
a1-server1 App-> 2011-10-05 12:28:43 - a1-server1 - [99.99.99.99] auser(Active Directory Users)[gg-group] - Closed connection after 12513 seconds, with 4502 bytes read (in 19 chunks) and 19391 bytes written (in 43 chunks)
a1-server1 App-> 2011-10-05 12:28:43 - a1-server1 - [99.99.99.99] auser(Active Directory Users)[gg-group] - Network Connect: ACL count = 833.
a1-server1 App-> 2011-10-05 12:28:43 - a1-server1 - [99.99.99.99] auser(Active Directory Users)[gg-group] - Network Connect: Session ended for user with IP 192.168.212.21
Initially, in props.conf I tried setting line_merge to false:
[source::source-to-break]
SHOULD_LINEMERGE = false
When that didn't work I tried specifying a line break before the server name at the beginning of each event:
[source::source-to-break]
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE = a[0-9]-server[0-9]\sApp[\w'-]>\s
But that also is not working. I'm beginning to think this has something to do w/ the time, but not sure.
Try the stanza below:
[source::source-to-break]
TIME_PREFIX = a\d+\-server\d+\s+App\-\>\s+
TIME_FORMAT = %Y-%m-%d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 20
LINE_BREAKER = ([\r\n]+)(?=a\d+\-server\d+\s+App\-\>\s+\d+\-\d+\-\d+\s+)
SHOULD_LINEMERGE = false
This will attempt to break right before a1-server1, extract the timestamp and format it properly. If this works please do not forget to vote 🙂
Try the stanza below:
[source::source-to-break]
TIME_PREFIX = a\d+\-server\d+\s+App\-\>\s+
TIME_FORMAT = %Y-%m-%d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 20
LINE_BREAKER = ([\r\n]+)(?=a\d+\-server\d+\s+App\-\>\s+\d+\-\d+\-\d+\s+)
SHOULD_LINEMERGE = false
This will attempt to break right before a1-server1, extract the timestamp and format it properly. If this works please do not forget to vote 🙂
took me a while to get this implemented, but I just did and it worked. thanks so much!