Transaction Search Using different fields across multiple logs

[manwin]

I am facing a problem with doing a transaction search across multiple logs (11 different sourcetypes) based on the example below.

I will simplify the scenario to just 4 sourcetypes for illustration purposes

1. sourcetype="web-access-logs" Common fields: Username, IP_address
2. sourcetype="application-logs" Common fields: Username, transaction_id
3. sourcetype="third-tier-app-logs" Common field: transaction_id
4. sourcetype="bmc-error-logs" Common field: Username

Is there a way to craft a transaction search to pull out all events?

In sourcetype="web-access-logs" I will want to pull out all activities based on the IP_address, as there will be no Username before the User logs into the portal and can only be tracked via the IP address until he logs in and the Username will appear, only then will we be able to link in the Username into the search.

Ultimately we are looking at having all the results using the indirect linkages to form a transaction for troubleshooting purposes

I tried a search as below

sourcetype="*" | transaction fields IP_address,Username,transaction_id connected=f maxspan=1h maxpause=35m |search transaction_id="123456789"

Unfortunately it is not working.

[Lowell · Accepted solution]

I do see one minor typo in your transaction command.

You can do either:

transaction fields="IP_address,Username,transaction_id" connected=f maxspan=1h maxpause=35m

or

transaction IP_address Username transaction_id connected=f maxspan=1h maxpause=35m

but you don't want the term fields just sitting there before you list your fields. I'm guessing this would look literally look for a field named "fields", which isn't what you want.

[Stephen_Sorkin · Accepted solution]

If your retrieval task is to see an entire transaction (either one or a handful of them), a subsearch is essential for performance. As gkanapathy and Lowell commented, you're pulling every event off disk and building a bunch of transactions just to throw most of them away.

You probably want a search like:

[search sourcetype="application-logs" transaction_id=<transaction_id> | dedup Username | fields Username] | transaction Username,transaction_id connected=f ...

[gkanapathy]

As Lowell, says, transaction is for mass assembly of many transactions. If you're trying to look up a single transaction based on a field, it's much more efficient to use subsearches or use forms and drilldown to select the specific events that make up a transaction.

[Lowell · Accepted solution]

I do see one minor typo in your transaction command.

You can do either:

transaction fields="IP_address,Username,transaction_id" connected=f maxspan=1h maxpause=35m

or

transaction IP_address Username transaction_id connected=f maxspan=1h maxpause=35m

but you don't want the term fields just sitting there before you list your fields. I'm guessing this would look literally look for a field named "fields", which isn't what you want.

[Lowell]

As with any complex search/view, I suggested either breaking the problem into smaller pieces if possible. Also, start with a simple search you can get working, then build on top of it. Only add a single new component (aka sourcetype) at a time, test and review your results and make sure to save off a working copy as you go (I find it helpful to simply copy-n-paste the search in a text editor along with some short notes about what's working and what's not. You can use splunk to find your previous searches, but without knowing which one was working, it can be quite time consuming I've found.)

[manwin]

Thanks I will test that out, the main reason why I need the transaction search is because the results is used to pull a user's transaction from the application which is spanning across different systems (11 of them in total) pulling out the entire transaction into a single view to understand the flow from start to end in a single view.

[Lowell]

Do you need to look at many transactions at once for analysis purposes? If not, a much simpler approach may be to build a form-search based view (or sets of views) and let a user drive the investigation instead of trying to do everything in a single search... just a thought.