Getting Data In

Tracing Response time of apache using splunk

vpraveenindia
Engager

I am tracking the response time(in seconds) of the pages served by apache using "%T" and i would like to track all the request which are taking more than "10" seconds to serve to the clients. Please let me know if there is a way to achieve.

Tags (2)
0 Karma
1 Solution

gcoles
Communicator

In your case the time (in seconds) field comes three positions after the request details. Since the request is always enclosed in quotes in apache logs, I like to use them as an anchor point for the regex, rather than the start of the log message (it allows for a simpler regex). Also, you have three numeric fields following a double-quote and preceding another double-quote, so I would write a regex like this:

\" \d+ \d+ (?P< response_time >\d+(?:\.\d+)?)

Note that this forum won't let me use the correct syntax above, so if you use this regex with either rex or the field extraction UI, remove the spaces on either side of response_time. I am not sure if the %T field may include decimal points or not, so I made the regex support them optionally.

You can use the regex with either rex (inline pipe to rex), or you can save this field extraction for your apache sourcetype (Manager > Fields > Field Extractions).

Here is what a search might look like with Field Extraction configured:

index=xxx sourcetype=access_combined response_time>10

And with inline rex:

index=xxx sourcetype=access_combined | rex "\" \d+ \d+ (?P< response_time >\d+(?:\.\d+)?)" | where response_time>10

I should note that you also have the %D parameter in your apache configuration, which is the response time in microseconds. I would personally use this field as it has better resolution and its at the end of the log, which makes it easier to anchor within the string:

(?P< response_time >\d+)$

Then just adjust your where/search syntax for the precision, response_time>10000000.

As I mentioned before, make sure to remove the spaces around 'response_time' above, when you use it in any of the splunk UI's.

View solution in original post

0 Karma

gcoles
Communicator

In your case the time (in seconds) field comes three positions after the request details. Since the request is always enclosed in quotes in apache logs, I like to use them as an anchor point for the regex, rather than the start of the log message (it allows for a simpler regex). Also, you have three numeric fields following a double-quote and preceding another double-quote, so I would write a regex like this:

\" \d+ \d+ (?P< response_time >\d+(?:\.\d+)?)

Note that this forum won't let me use the correct syntax above, so if you use this regex with either rex or the field extraction UI, remove the spaces on either side of response_time. I am not sure if the %T field may include decimal points or not, so I made the regex support them optionally.

You can use the regex with either rex (inline pipe to rex), or you can save this field extraction for your apache sourcetype (Manager > Fields > Field Extractions).

Here is what a search might look like with Field Extraction configured:

index=xxx sourcetype=access_combined response_time>10

And with inline rex:

index=xxx sourcetype=access_combined | rex "\" \d+ \d+ (?P< response_time >\d+(?:\.\d+)?)" | where response_time>10

I should note that you also have the %D parameter in your apache configuration, which is the response time in microseconds. I would personally use this field as it has better resolution and its at the end of the log, which makes it easier to anchor within the string:

(?P< response_time >\d+)$

Then just adjust your where/search syntax for the precision, response_time>10000000.

As I mentioned before, make sure to remove the spaces around 'response_time' above, when you use it in any of the splunk UI's.

0 Karma

vpraveenindia
Engager

You are simply awesome!

0 Karma

vpraveenindia
Engager

Apologies for the delay.. below is the snippet from the apache log and the Log format

10.0.0.2 - - [03/Jan/2013:13:50:31 +0000] "GET /admin HTTP/1.1" 200 293 0 "-" "http://test.test.com" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; InfoPath.2)" 1338

"%h %l %u %t \"%r\" %>s %b %T \"%{LoggedIn}e\" \"%{Referer}i\" \"%{User-Agent}i\" %D"

0 Karma

gcoles
Communicator

Do you already have a Field Extraction set up for the response_time field? If you do, you can simply use greater and less than signs in the search:

index=weblogs response_time>10

gcoles
Communicator

No problem, can you paste in a sample log record or your apache LogFormat directive?Apache log files typically don't include field=value pairs (just values), so regular expressions must be position-sensitive (eg populate this field with the fifth element in the message).

0 Karma

vpraveenindia
Engager

Thanks for your reply, unfortunately i didn't have that setup already, would you be able to help me with that please? may be a small hint will do.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...