Struggling with extracting multiline events from SiteScope Alert.log file

[dlovett]

I'm a newbie to Splunk so any advice would be appreciated. I am trying to forward a log file that has multiline events. Each event begins with the pattern, "timestamp date alert". So I simply just want to break before that line.

---

09:12:47 08/09/2012 alert
alert-type: Post
alert-name: post_email1x
alert-message: ok,
alert-monitor: Test Monitor - Please Ignore
alert-group: Test Group:
alert-id: 200689661
alert-monitor-id: SiteScope/TestGroup/1:1
action-name: post_email1x
alert-url: http://some.url.com
alert-postData:
alert-replyStatus: ok
alert-replySize: 54
alert-replyDuration: 577
alert-replyContent: SITESCOPE HTTP REQUEST HTTP REQUEST HTTP REQUEST HTTP REQUEST HTTP REQUEST HTTP REQUEST
POST /cgi-bin/sis_postalert?sis_sendemail|some.url.com HTTP/1.1
User-Agent: Mozilla/4.0 (compatible; MSIE 4.01; Windows NT)
Accept: */*
Content-Type: application/x-www-form-urlencoded
Host: some.url.com
Content-Length: 861
siteScope=some.url.com&monitor+name=DBQry%3A+-+Check+AMCOM+dblink&monitor+internalID=200&group+name=name&group+path=Application+Monitors%3A+name&group+internal+path=SiteScope%2FApplicationMonitors&status=connect+error%2C+null&sample=1127834&class=DatabaseMonitor&tag=&time=8%3A22+AM+8%2F9%2F12&date=08%2F09%2F2012&mseqtime=1344514951279&category=error&frequency=60&schedule=&TEMPLATENAME=Email%2CMCIT_DefaultEmailV1&SUBJECT=%22SiteScope+Alert%2C+error%2C+DBQry%3A+CDRT+-+Check+AMCOM+dblink%2C+connect+error%2C+null+%28uhsismonsts1.umhs.med.umich.edu%29%22&BODY=This+alert+is+from+SiteScope+at+http%3A%2F%2Fuhsismonsts1.umhs.med.umich.edu%3A8443%2FSiteScope%0AMonitor%3A++DBQry%3A+CDRT+-+Check+AMCOM+dblink%0AGroup%3A++++CDRT%0AStatus%3A+++connect+error%2C+null%0ASample+%23%3A+1127834%0A%0ATime%3A+++++8%3A22+AM+8%2F9%2F12
SITESCOPE BLANK LINE
SITESCOPE HTTP RESPONSE HEADERS
HTTP/1.1 200 OK
Date: Thu, 09 Aug 2012 12:22:31 GMT
Server: IBM_HTTP_Server
Content-Length: 54
Content-Type: text/html
SITESCOPE HTTP RESPONSE BODY
[some html stuff goes here]

09:14:06 08/09/2012 alert
alert-type: Post
alert-name: post_email1x
alert-message: ok,
alert-monitor: Test Monitor - Please Ignore
alert-group: Test Group:
alert-id: 200689661
alert-monitor-id: SiteScope/TestGroup/1:1
action-name: post_email1x
alert-url: http://some.url.com
alert-postData:
alert-replyStatus: ok
alert-replySize: 54
alert-replyDuration: 577
alert-replyContent: SITESCOPE HTTP REQUEST HTTP REQUEST HTTP REQUEST HTTP REQUEST HTTP REQUEST HTTP REQUEST
POST /cgi-bin/sis_postalert?sis_sendemail|some.url.com HTTP/1.1
User-Agent: Mozilla/4.0 (compatible; MSIE 4.01; Windows NT)
Accept: */*
Content-Type: application/x-www-form-urlencoded
Host: some.url.com
Content-Length: 861
siteScope=some.url.com&monitor+name=DBQry
SITESCOPE BLANK LINE
SITESCOPE HTTP RESPONSE HEADERS
HTTP/1.1 200 OK
Date: Thu, 09 Aug 2012 12:22:31 GMT
Server: IBM_HTTP_Server
Content-Length: 54
Content-Type: text/html
SITESCOPE HTTP RESPONSE BODY
[some html goes here]

etc. etc. etc

---

It seems like a trivial problem to solve. I've attempted to go through the docs and splunk site to solve this problem in addition to experimenting with different settings in the props.conf file but cannot solve this problem. The following is what is in the local/props.conf file:

[sitescope_alerts]

DATETIME_CONFIG = NONE
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE = ^(\d{2}:\d{2}:\d{2} \d{2}\/\d{2}\/\d{4}\s\balert\b)

However, Splunk is forwarding 3 events. It seems to be breaking after it finds a date:

Event 1

09:12:47 08/09/2012 alert
alert-type: Post
alert-name: post_email1x
alert-message: ok,
alert-monitor: Test Monitor - Please Ignore
alert-group: Test Group:
alert-id: 200689661
alert-monitor-id: SiteScope/TestGroup/1:1
action-name: post_email1x
alert-url: http://some.url.com
alert-postData:
alert-replyStatus: ok
alert-replySize: 54
alert-replyDuration: 577
alert-replyContent: SITESCOPE HTTP REQUEST HTTP REQUEST HTTP REQUEST HTTP REQUEST HTTP REQUEST HTTP REQUEST
POST /cgi-bin/sis_postalert?sis_sendemail|some.url.com HTTP/1.1
User-Agent: Mozilla/4.0 (compatible; MSIE 4.01; Windows NT)
Accept: */*
Content-Type: application/x-www-form-urlencoded
Host: some.url.com
Content-Length: 861
siteScope=some.url.com&monitor+name=DBQry%3A+-+Check+AMCOM+dblink&monitor+internalID=200&group+name=name&group+path=Application+Monitors%3A+name&group+internal+path=SiteScope%2FApplicationMonitors&status=connect+error%2C+null&sample=1127834&class=DatabaseMonitor&tag=&time=8%3A22+AM+8%2F9%2F12&date=08%2F09%2F2012&

Event 2

mseqtime=1344514951279&category=error&frequency=60&schedule=&TEMPLATENAME=Email%2CMCIT_DefaultEmailV1&SUBJECT=%22SiteScope+Alert%2C+error%2C+DBQry%3A+CDRT+-+Check+AMCOM+dblink%2C+connect+error%2C+null+%28uhsismonsts1.umhs.med.umich.edu%29%22&BODY=This+alert+is+from+SiteScope+at+http%3A%2F%2Fuhsismonsts1.umhs.med.umich.edu%3A8443%2FSiteScope%0AMonitor%3A++DBQry%3A+CDRT+-+Check+AMCOM+dblink%0AGroup%3A++++CDRT%0AStatus%3A+++connect+error%2C+null%0ASample+%23%3A+1127834%0A%0ATime%3A+++++8%3A22+AM+8%2F9%2F12
SITESCOPE BLANK LINE
SITESCOPE HTTP RESPONSE HEADERS
HTTP/1.1 200 OK

Event 3

Date: Thu, 09 Aug 2012 12:22:31 GMT
Server: IBM_HTTP_Server
Content-Length: 54
Content-Type: text/html
SITESCOPE HTTP RESPONSE BODY
[some html goes here]

[sdaniels]

Have you tried the Splunk for Sitescope App? Might make life easier for you.

http://splunk-base.splunk.com/apps/39629/splunk-for-sitescope

[jonuwz · Accepted solution]

Splunk recognises the timestamp formatting, so you don't need DATETIME_CONFIG=NONE

The only other difference is the regex - This works for me - not sure if the formatting of the log has changed by posting it here though

[sitescope_alert]
BREAK_ONLY_BEFORE = ^\d{2}:\d{2}:\d{2} \d{2}/\d{2}/\d{4}\s+alert$
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = true

Also - is [some html goes here] actually part of the log message?

If its a really big log entry, it'll split after 256 lines anyway unless you set MAX_EVENTS to something bigger

update: this is more efficient

SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?=\d{2}:\d{2}:\d{2} \d{2}/\d{2}/\d{4}\s+alert[\r\n]+)

[dlovett]

ok...I get it now 🙂 I modified the props.conf on the indexer and all is well.

ty!

[jonuwz]

Aha - universal forwarder.

Where are your config files ?
http://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings%3F

linebreaking happens at parsing time - so the props.conf needs to be on the indexer, not the universal forwarder

[dlovett]

Thanks for the reply!

• I removed the HTML to simply the formatting of my original post.
• Good point on MAX_EVENTS. Each alert is less than 256 lines. each contains three date/timestamps.
• What you wrote is definitely more efficient. ty 🙂

The Splunk universal forwarder is still breaking on date. We would really like to create a dashboard on our alerting; reading this alert.log file is key.

A colleague of mine solved a similar problem via writing a script to parse out what he wanted into a file and set the forwarder to read that file. I'd rather not do this but it works.