How to speed up log search time with transaction

[willadams]

I have Cisco ESA logs coming into Splunk and extractions are working as they are meant to. The logs are sent by syslog and each line of the event appears to be an entry in the index. So that I can combine a transaction in ESA I use the following spl query (mid is the common field):

Index=foo | transaction mid

This will give me the required block for a particular mid (message id) that have come through the ESA device.

I can do searches from here. For example:

 index=foo | transaction mid | search spam_status = positive

For a 15 minute search, this is quite quick. However, for a 24 hour block (particularly if I am trying to do some reporting on the number of say spam is negative, positive) the search takes a very very long time. In fact, if I stop the search I end up with 0 results. For example:

index=foo | transaction mid | stats coumt by spam_status

Is there a way I can do this transaction without waiting an eternity for the search to do something (if it ever finishes)?

If I shortcut the search say with:

index=foo mid=123456 | transaction mid | stats coumt by spam_status

Then the search is fast. However, this depends on knowing the mid which will change every day so os not feasible, especially doing reports from the available logs.

[marcluescher]

There are some options available, some more some less difficult to implement.

The fastest way it to configure the new CEL logs file forwarding for the ESA servers and then inject those files into Splunk. Should you need more detailed information then ytou mainly use CEL mail log as the sourcefile adn you then aggregate the information with transcational data from the classical mail_logs.

Searching CEL is very fast, even the first Release still has some issuie when a field contains a space.

I hope that helps.

Marc

[to4kawa]

index=foo | stats count by spam_status mid

If you can use transaction, basically stats can be used.

[PavelP]

transaction is a slow command because it breaks map reduce. You can speed up transaction search by using following options (https://docs.splunk.com/Documentation/SplunkCloud/8.0.2001/SearchReference/transaction):
endswith, maxspan, maxevents, startswith

try these first:
maxspan=1m maxevents=20