Need to treat multiple lines in log file as a single unit, joined by an element

[jarrowwx]

I need to index the logs for a web service. For each end-user's interactions with the system, a collection of web service calls are likely to be made. In the raw files, each is a separate line, and different sessions will be woven in together, because it is a multi-user system. The one thing they will all have in common is the session ID. For example:

13:28:09,080 01/31 0317BF7117D4C4A2F80CE25BB76F8EB8 ... ConfigurationService.lookupApplicationConfigurationProperties ...
13:28:09,086 01/31 D4CC24E705670088920490171F76A079 ... NetworkService.queryOutages ...
13:28:09,234 01/31 0317BF7117D4C4A2F80CE25BB76F8EB8 ... ConfigurationService.uivrLookupBroadcastMessages ...
13:28:09,298 01/31 D4CC24E705670088920490171F76A079 ... AppointmentService.queryAppointment ...
13:28:09,385 01/31 B5D41CFAE762291AEDD43CE424B751F8 ... IdentifyService.dtmfLookupCustomerByTN ...

What I need to do is select across multiple rows. In the above example, where the session ID starts with 0317, those lines are for one customer. I need to be able to compose a search that will treat all of those lines as being together, so I can specify criteria across different services. A single customer session may be composed of 10 services, and I need to select all of the customer sessions that included services X, Y, and Z, and for each of those services, I want to put restrictions in there, such as field X.FOO=someValue, Y.BAR=somethingElse, and Z.BLIP=argh

Is this even doable?

Thanks for the help!

[David · Accepted solution]

It sounds like what you want is the transaction command: http://www.splunk.com/base/Documentation/latest/SearchReference/Transaction

You would need to define a field that matches your session ID using perl-compatible regular expressions with either props.conf or the rex command, such as:

YourSearch | rex field=_raw "\d\d,\d\d\d \d\d/\d\d (?<SessionID>\S*)"

That will package all the lines for a particular user as one event, and give you access to things such as:

YourSearch | rex field=_raw "\d\d,\d\d\d \d\d/\d\d (?<SessionID>\S*)" | search NetworkService.queryOutages 

to find only sessions that query network service outages or:

YourSearch | rex field=_raw "\d\d,\d\d\d \d\d/\d\d (?<SessionID>\S*)" 
  | search NetworkService.queryOutages Query.Result=Error

to get sessions where they both queried network service outages and where one of the different actions they took had an error. You also get the duration field, which tells you how long the session lasted, and the eventcount field, which tells you how many individual lines were packaged into that transaction.

If you're not familiar with PCRE syntax and rex/props.conf, you can run with what I put above, or start your search with the following:

• http://www.splunk.com/base/Documentation/latest/SearchReference/rex
• http://www.splunk.com/base/Documentation/latest/admin/Propsconf
• http://www.splunk.com/wiki/Community:RegexSyntaxInSplunk

I hope that all is helpful.