How Do I Capture Apache Logs into the x-forwarder-for value to search in Splunk?

[heats]

Trying to capture the IP address out of the apache logs and into the x-forwarded-for field in Splunk

I've added the following line to httpd.conf:

# Include generic snippets of statements
Include /etc/httpd/conf.d/*.conf

And I've created extended_logging.conf and adding the following formatting syntax:

LogFormat "%v %p %h %a %l %u %t %D %m \"%U%q\" \"%U\" \"%q\" %>s %b \"%{Referer}i\" \"%{User-agent}i\" \"%{X-Forwarded-For}i\" \"%{Cookie}i\"" extended

I've restarted the Apache service and the splunk service on the host but searching Splunk on that host I am still not seeing the x-forwarded-for value pop up in Interesting Fields in the GUI.

This box is a RHEL6 box. Has anyone gotten x-forwarded-for working for Apache logs that may know of a step I missed?

Thanks!

[cpetterborg]

From Settings -> Fields -> Field Extractions -> New you can create an automatic field extraction. The application needs to be specified, along with either source or sourcetype as the means of deciding what date you will be doing the field extraction on. I'd suggest sourcetype and then select the sourcetype of the data you are needing to extract the x_forwarded_for field in. Leave the Type on Inline and then enter the following in the Extraction/Transform textbox (you may have to modify it slightly if your data is different from mine):

x_forwarded_for:"(?P<x_forwarded_for>\d+\.\d+\.\d+\.\d+)"

That regular expression (regex) will extract the IP address from the data and put it in a field called x_forwarded_for. Save that and then go search your data. You should then have the field automatically extracted for all the data with the sourcetype that you have selected.

Here are some additional resources for doing this kind of thing (some references may be a bit older, but can give you an idea of what to do):

http://docs.splunk.com/Documentation/SplunkCloud/6.6.0/Knowledge/Aboutfields
http://docs.splunk.com/Documentation/Splunk/latest/Knowledge/ExtractfieldsinteractivelywithIFX
https://www.splunk.com/view/SP-CAAADUY
http://docs.splunk.com/Documentation/Splunk/latest/Knowledge/Createandmaintainsearch-timefieldextrac...

[heats]

Thanks! I'm about to try this - does this mean that no changes need to be made to the servers themselves as far as the formatting I've done for conf.d and extended_logging.conf? Or this is in addition to that? It would be great if this method worked without having to do special configurations of the servers.

[cpetterborg]

As long as the logs contain the x_forwarded_for information in them, this should work for you, but that is dependent on if it is already in the logs. Most Apache logs don't contain this information by default. We have some Apache servers that are configured to output the info to the logs, and some are not. YMMV. Check the log as it exists before making those changes.

[cpetterborg]

I just did an automatic field extraction to get ours to have the x-forwarded-for field extracted properly. Yes, it is an additional field extraction, but that is what Splunk is good at, right?

If you need any help with the field extraction regex, just let me know with a comment and I'll add it here.

[heats]

I'm actually not even sure what the automatic field extraction is and how it works. So additional help with how it works and the regex would be greatly appreciated! Thanks so much.