Splunk Search

Splunk returns some queries much faster than others

mgimelfarb
Explorer

I am working with access_combined_wcookie data (essentially Nginx log files) in Splunk. An example of a record is below:

5/25/14 2:44:08.000 AM  xxx.xxx.xxx.xxx - - [25/May/2014:02:44:08 -0500] "GET /somepath/ HTTP/1.1" 200 9696 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +bingbot.htm)" "-"

I'm specifically interested in being able to run queries on the GET uri. So, that I can issue the following for instance:

/somepath/ | chart count by date_mday 

I have now loaded over 60M records into Splunk and keep adding more each day.

I see that sometimes Splunk automatically indexes the "/somepath/" value and when I enter the query I get an immediate answer. But when enter the following query:

/some-path/ | chart count by date_mday 

(essentially the same as above, but with a hyphen or dash), I have to wait a while for Splunk to generate the results. It seems that the outcome is volume-related: on smaller sets I get the result immediately, almost as if it were cached, where as the larger result sets take a long time (disproportionally so).

Is there any way for me to control that behavior? Is the smaller volume results get cached in memory and thus adding more RAM to the machine (VM in this case) would help get faster results?

Tags (3)
1 Solution

jhupka
Path Finder

In terms of why Search A is way faster than Search B is Splunk is doing two different searches under the hood. In the first Search A, Splunk essentially looks for "somepath" - The Splunk Indexer is only going to return events that have "somepath" in them. But in Search B, Splunk will likely breakup your terms because of the dash and look for "some" AND "path" So in a way your search could be looking for many more events than necessary.

A good thing to do is look at the Job Inspector and see what Splunk is doing under the hood. Also, you might try this Search C and see if its performance is on par with Search A:

TERM(some-path) | chart count by date_mday

The TERM search operator tells Splunk to use the contents as a single term instead of splitting up your search terms at the dash.

View solution in original post

jhupka
Path Finder

In terms of why Search A is way faster than Search B is Splunk is doing two different searches under the hood. In the first Search A, Splunk essentially looks for "somepath" - The Splunk Indexer is only going to return events that have "somepath" in them. But in Search B, Splunk will likely breakup your terms because of the dash and look for "some" AND "path" So in a way your search could be looking for many more events than necessary.

A good thing to do is look at the Job Inspector and see what Splunk is doing under the hood. Also, you might try this Search C and see if its performance is on par with Search A:

TERM(some-path) | chart count by date_mday

The TERM search operator tells Splunk to use the contents as a single term instead of splitting up your search terms at the dash.

mgimelfarb
Explorer

@jhupka, this worked really well. I was able to also use
TERM(some-path/*) | chart count by date_mday for some more advanced queries. The speed has in fact improved dramatically. Thank you!

0 Karma

rsennett_splunk
Splunk Employee
Splunk Employee

Yes. Totally true. I was so focused on the details of the search structure I didn't address your actual question.

But the bottom line is... you only want to have it check the uri_stem, not the whole event. If you have /some-path-with-more-words/ and that's part of the cookie... or the referer, well, you've just caused Splunk to check in all sorts of places (and possibly bring back stuff you don't want) rather than just the uri_stem for that exact value.

Using a "power grep" when you don't have to isn't going to tell you anything about performance.

With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!
0 Karma

rsennett_splunk
Splunk Employee
Splunk Employee

Taking your search literally... you are doing a "power grep" for what is essentially a specific uri_stem and counting the occurrences by date_mday. Now, perhaps you're using a snippet of your search and you are actually telling Splunk which index to look at, and which sourcetype... but by putting the phrase "/somepath/" in there as the only info on what you want to find you are leaving some key info out and forcing Splunk to run around searching for that pattern... 60 Million times.

We can do better. And you can see the difference by checking the job inspector.

First, since you mention "access_combined_wcookie", can I assume that you're using that sourcetype and that you've allowed Splunk to create the default access_extractions?

If so... you have fields you can work with right of the bat.

GET is a method (that's one of the fields created) and the uri is what comes after. Splunk has extracted just the "stem" for you as a field called uri_stem There is probably one called uri_query as well... which is the next field after a space. together that is the uri, which may also be a field.

Show me a count of all the occurrences of the uri_stem /somepath/ when the method is GET
I'd do this:

index=blah sourcetype=access_combined_wcookie uri_stem="/somepath/" method="GET"|chart blah, blah blah...

Let Splunk do the work...
If you really want to see the difference in response time, start with an hour of data... check the job inspector. it will show you how many records it brought back and how long it took. Widen the scope, and check again.

With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!

rsennett_splunk
Splunk Employee
Splunk Employee

take a look in the $SPLUNK_HOME/etc/system/default/transforms.conf file... there are transforms you can manipulate to further break down the uri_path into root, file etc... I use that to good effect on these type of logs. you don't have to do it yourself...

With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!

mgimelfarb
Explorer

Looks like the field is actually called uri_path in this particular case, in case anyone else is trying the same with Nginx logs

0 Karma

mgimelfarb
Explorer

@rsennett_splunk, this is a great idea. Even though I haven't accepted this answer, I think that this is a great response, and I will try this next. I would actually like to go further and start splitting uri_stem into more subfields based on regex, which in actuality is more like /some-path/some-more-path/blah so that I can count the occurrences of blah within a given /some-path/ combinations. But that's for another question and is a bit OT.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...