I'm a newbie to Splunk so any advice would be appreciated. I am trying to forward a log file that has multiline events. Each event begins with the pattern, "timestamp date alert". So I simply just want to break before that line.
09:12:47 08/09/2012 alert
alert-type: Post
alert-name: post_email1x
alert-message: ok,
alert-monitor: Test Monitor - Please Ignore
alert-group: Test Group:
alert-id: 200689661
alert-monitor-id: SiteScope/TestGroup/1:1
action-name: post_email1x
alert-url: http://some.url.com
alert-postData:
alert-replyStatus: ok
alert-replySize: 54
alert-replyDuration: 577
alert-replyContent: SITESCOPE HTTP REQUEST HTTP REQUEST HTTP REQUEST HTTP REQUEST HTTP REQUEST HTTP REQUEST
POST /cgi-bin/sis_postalert?sis_sendemail|some.url.com HTTP/1.1
User-Agent: Mozilla/4.0 (compatible; MSIE 4.01; Windows NT)
Accept: */*
Content-Type: application/x-www-form-urlencoded
Host: some.url.com
Content-Length: 861
siteScope=some.url.com&monitor+name=DBQry%3A+-+Check+AMCOM+dblink&monitor+internalID=200&group+name=name&group+path=Application+Monitors%3A+name&group+internal+path=SiteScope%2FApplicationMonitors&status=connect+error%2C+null&sample=1127834&class=DatabaseMonitor&tag=&time=8%3A22+AM+8%2F9%2F12&date=08%2F09%2F2012&mseqtime=1344514951279&category=error&frequency=60&schedule=&TEMPLATENAME=Email%2CMCIT_DefaultEmailV1&SUBJECT=%22SiteScope+Alert%2C+error%2C+DBQry%3A+CDRT+-+Check+AMCOM+dblink%2C+connect+error%2C+null+%28uhsismonsts1.umhs.med.umich.edu%29%22&BODY=This+alert+is+from+SiteScope+at+http%3A%2F%2Fuhsismonsts1.umhs.med.umich.edu%3A8443%2FSiteScope%0AMonitor%3A++DBQry%3A+CDRT+-+Check+AMCOM+dblink%0AGroup%3A++++CDRT%0AStatus%3A+++connect+error%2C+null%0ASample+%23%3A+1127834%0A%0ATime%3A+++++8%3A22+AM+8%2F9%2F12
SITESCOPE BLANK LINE
SITESCOPE HTTP RESPONSE HEADERS
HTTP/1.1 200 OK
Date: Thu, 09 Aug 2012 12:22:31 GMT
Server: IBM_HTTP_Server
Content-Length: 54
Content-Type: text/html
SITESCOPE HTTP RESPONSE BODY
[some html stuff goes here]
09:14:06 08/09/2012 alert
alert-type: Post
alert-name: post_email1x
alert-message: ok,
alert-monitor: Test Monitor - Please Ignore
alert-group: Test Group:
alert-id: 200689661
alert-monitor-id: SiteScope/TestGroup/1:1
action-name: post_email1x
alert-url: http://some.url.com
alert-postData:
alert-replyStatus: ok
alert-replySize: 54
alert-replyDuration: 577
alert-replyContent: SITESCOPE HTTP REQUEST HTTP REQUEST HTTP REQUEST HTTP REQUEST HTTP REQUEST HTTP REQUEST
POST /cgi-bin/sis_postalert?sis_sendemail|some.url.com HTTP/1.1
User-Agent: Mozilla/4.0 (compatible; MSIE 4.01; Windows NT)
Accept: */*
Content-Type: application/x-www-form-urlencoded
Host: some.url.com
Content-Length: 861
siteScope=some.url.com&monitor+name=DBQry
SITESCOPE BLANK LINE
SITESCOPE HTTP RESPONSE HEADERS
HTTP/1.1 200 OK
Date: Thu, 09 Aug 2012 12:22:31 GMT
Server: IBM_HTTP_Server
Content-Length: 54
Content-Type: text/html
SITESCOPE HTTP RESPONSE BODY
[some html goes here]
etc. etc. etc
It seems like a trivial problem to solve. I've attempted to go through the docs and splunk site to solve this problem in addition to experimenting with different settings in the props.conf file but cannot solve this problem. The following is what is in the local/props.conf file:
[sitescope_alerts]
DATETIME_CONFIG = NONE
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE = ^(\d{2}:\d{2}:\d{2} \d{2}\/\d{2}\/\d{4}\s\balert\b)
However, Splunk is forwarding 3 events. It seems to be breaking after it finds a date:
Event 1
09:12:47 08/09/2012 alert
alert-type: Post
alert-name: post_email1x
alert-message: ok,
alert-monitor: Test Monitor - Please Ignore
alert-group: Test Group:
alert-id: 200689661
alert-monitor-id: SiteScope/TestGroup/1:1
action-name: post_email1x
alert-url: http://some.url.com
alert-postData:
alert-replyStatus: ok
alert-replySize: 54
alert-replyDuration: 577
alert-replyContent: SITESCOPE HTTP REQUEST HTTP REQUEST HTTP REQUEST HTTP REQUEST HTTP REQUEST HTTP REQUEST
POST /cgi-bin/sis_postalert?sis_sendemail|some.url.com HTTP/1.1
User-Agent: Mozilla/4.0 (compatible; MSIE 4.01; Windows NT)
Accept: */*
Content-Type: application/x-www-form-urlencoded
Host: some.url.com
Content-Length: 861
siteScope=some.url.com&monitor+name=DBQry%3A+-+Check+AMCOM+dblink&monitor+internalID=200&group+name=name&group+path=Application+Monitors%3A+name&group+internal+path=SiteScope%2FApplicationMonitors&status=connect+error%2C+null&sample=1127834&class=DatabaseMonitor&tag=&time=8%3A22+AM+8%2F9%2F12&date=08%2F09%2F2012&
Event 2
mseqtime=1344514951279&category=error&frequency=60&schedule=&TEMPLATENAME=Email%2CMCIT_DefaultEmailV1&SUBJECT=%22SiteScope+Alert%2C+error%2C+DBQry%3A+CDRT+-+Check+AMCOM+dblink%2C+connect+error%2C+null+%28uhsismonsts1.umhs.med.umich.edu%29%22&BODY=This+alert+is+from+SiteScope+at+http%3A%2F%2Fuhsismonsts1.umhs.med.umich.edu%3A8443%2FSiteScope%0AMonitor%3A++DBQry%3A+CDRT+-+Check+AMCOM+dblink%0AGroup%3A++++CDRT%0AStatus%3A+++connect+error%2C+null%0ASample+%23%3A+1127834%0A%0ATime%3A+++++8%3A22+AM+8%2F9%2F12
SITESCOPE BLANK LINE
SITESCOPE HTTP RESPONSE HEADERS
HTTP/1.1 200 OK
Event 3
Date: Thu, 09 Aug 2012 12:22:31 GMT
Server: IBM_HTTP_Server
Content-Length: 54
Content-Type: text/html
SITESCOPE HTTP RESPONSE BODY
[some html goes here]
Splunk recognises the timestamp formatting, so you don't need DATETIME_CONFIG=NONE
The only other difference is the regex - This works for me - not sure if the formatting of the log has changed by posting it here though
[sitescope_alert]
BREAK_ONLY_BEFORE = ^\d{2}:\d{2}:\d{2} \d{2}/\d{2}/\d{4}\s+alert$
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = true
Also - is [some html goes here] actually part of the log message?
If its a really big log entry, it'll split after 256 lines anyway unless you set MAX_EVENTS to something bigger
update: this is more efficient
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?=\d{2}:\d{2}:\d{2} \d{2}/\d{2}/\d{4}\s+alert[\r\n]+)
Have you tried the Splunk for Sitescope App? Might make life easier for you.
http://splunk-base.splunk.com/apps/39629/splunk-for-sitescope
Splunk recognises the timestamp formatting, so you don't need DATETIME_CONFIG=NONE
The only other difference is the regex - This works for me - not sure if the formatting of the log has changed by posting it here though
[sitescope_alert]
BREAK_ONLY_BEFORE = ^\d{2}:\d{2}:\d{2} \d{2}/\d{2}/\d{4}\s+alert$
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = true
Also - is [some html goes here] actually part of the log message?
If its a really big log entry, it'll split after 256 lines anyway unless you set MAX_EVENTS to something bigger
update: this is more efficient
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?=\d{2}:\d{2}:\d{2} \d{2}/\d{2}/\d{4}\s+alert[\r\n]+)
ok...I get it now 🙂 I modified the props.conf on the indexer and all is well.
ty!
Aha - universal forwarder.
Where are your config files ?
http://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings%3F
linebreaking happens at parsing time - so the props.conf needs to be on the indexer, not the universal forwarder
Thanks for the reply!
The Splunk universal forwarder is still breaking on date. We would really like to create a dashboard on our alerting; reading this alert.log file is key.
A colleague of mine solved a similar problem via writing a script to parse out what he wanted into a file and set the forwarder to read that file. I'd rather not do this but it works.