Can splunk search use rex to pick through 6 lines to get (beginning to end) ERROR condition (I included the lines).

[lindsaygw]

Here is the 6 lines in a log file that all come out together in the log but they are each different lines (not wrapped): I need to key the words "exceeded, max, stopped, snooze and "ERROR [Thread".

2013-04-03 18:55:37,065 ERROR [Thread-15] com.documentum.services.thread.CTSThreadPoolManagerImpl - Thread: Thread-24 exceeded max allowable time for single task
2013-04-03 18:55:37,065 ERROR [Thread-15] com.documentum.services.thread.CTSThreadPoolManagerImpl - Time diff: 38635091 max allowable diff: 36000000
2013-04-03 18:55:37,065 ERROR [Thread-15] com.documentum.services.thread.CTSThreadPoolManagerImpl - Time exceeded by 2635 sec
2013-04-03 18:55:37,065 ERROR [Thread-15] com.documentum.services.thread.CTSThreadPoolManagerImpl - Please note that this max allowable time is configurable inCTSServerService.xml file. Modify the following tags to have a higher value 9060
2013-04-03 18:55:37,065 ERROR [Thread-15] com.documentum.services.thread.CTSThreadPoolManagerImpl - Stopping all threads...
2013-04-03 18:55:37,065 ERROR [Thread-15] com.documentum.services.thread.CTSThreadPoolManagerImpl - Will snooze now to allow running threads finish their job

Basically the App folks want an alert from a saved search that pulls in all 6 of these lines (there is info on each line that they want to see so to speak). All 6 lines make up an error thread (could be any thread).

I have this so far and the rex pipe will not put any data into my field, but the search works. So my regex needs to be able to not just think of this as 1 line I guess (I am not a regex guru yet...far from it).

\bE(\w*)R\b\s.*| max | exceeded | Stopping | snooze

In splunk search I got this relaibly catching just what I want, but it wont work if I try to pipe this through rex: I want the alert from the search to look pretty with a table format and rename command and time conversion. I just cant get rex to create a field or get REX to give me back anything other than one line at a time (IE: 6 searches).

index=application sourcetype=documentum host=myserver01 ERROR AND "exceeded max" OR "max allowable" OR "Time exceeded" OR "Stopping all threads" OR snooze

[bmacias84]

I think you want to use Transaction. Transaction will build all six lines (events) into a single event. There are a few aproaches that would work.

Read the section Data classification: Event types and transactions. Event types, taggings, and transactions all have possibility of solving your problem.

Hope this help or gets you started. Dont forget to accept answers and/or vote them up.

Cheers,

[lindsaygw]

Just to post and let you know I appreciate your reply (for sure), I wanted to give a status. The status is that the documentum folks didnt really need all 5 lines to make one condition (they didnt know what they really wamted...LOL). In the end we did a regular search for 1 line that meant the most and was reliable.

Anyway, I still think this is something that I need to know how to do and will take and read these "sections". I will test it and reply back with my results. My problem is that it may take a week to peck at it with my workload. I will then answer and vote it.

Thanks again!

[lindsaygw]

ignore this part of my post (bE(w)Rbs.| max | exceeded | Stopping | snooze).

I can get 5 individual "easy" searches that work to catch each line. Do I need to just create 6 and then use all 6 to produce a report? Do I need to use sub search? Do I pipe rex over and over in 1 search (I know that will not do well. I get results but cant make a table view that brings it all in one view (all 5 lines equal 1 alert that looks good).