Splunk Search

Index vs Search Time for Busy Web Logs

Kyle_Brandt
Path Finder

I am moving my web log reporting to Splunk. Even when I don't log static content I have about 1.5 Million events per hour with about 20 fields per entry I want to extract.

In the interest of speeding this up I have made one search time regex which contains all the fields instead of multiple regexes.

In the end I would like to be able to run multiple report like "eventtype="HAProxy Web Logs" | chart count by HTTP_CLIENT_IP | sort count desc" over 24 hour periods. However, even running one of these reports for the period of 1 hour takes several minutes.

In order to optimize searches and reporting on this amount of data am I going to have to create index time field extraction? Are there other optimizations I should be looking into?

0 Karma
1 Solution

gkanapathy
Splunk Employee
Splunk Employee

Index time field extractions will not help and in fact will probably make things slower. To be able to efficiently report, you will need to use summary indexing. Summary indexing is not too hard to set up, but it does cost you flexibility as you essentially decide what sort of things you'd report on first, then build the summaries.

View solution in original post

gkanapathy
Splunk Employee
Splunk Employee

Index time field extractions will not help and in fact will probably make things slower. To be able to efficiently report, you will need to use summary indexing. Summary indexing is not too hard to set up, but it does cost you flexibility as you essentially decide what sort of things you'd report on first, then build the summaries.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...