Getting Data In

Coalesce and CIM Compliant Fields

BP9906
Builder

Hello,
From a reporting perspective, I have apache logs in a company standard format. Due to load balancing configuration, we have 3 possible fields where the source ip is noted.

These fields are as follows:
clientip (standard source IP field)
X_FORWARDED_IP (x-forwarded-for http header)
ns_client_ip (load balancer's view of source ip)

Since all 3 fields exist in sourcetype=access_combined (apache) logs, how do I coalesce the fields to "src" to make it CIM compliant?

I will mention that Apache logs a hyphen "-" for null field values for the above too.

Thanks for your help.

0 Karma
1 Solution

woodcock
Esteemed Legend

Inside props.conf do this:

EVAL-src = case((clientip != "-"), clientip,  (X_FORWARDED_IP != "-"), X_FORWARDED_IP, (ns_client_ip != "-"), ns_client_ip)

View solution in original post

woodcock
Esteemed Legend

Inside props.conf do this:

EVAL-src = case((clientip != "-"), clientip,  (X_FORWARDED_IP != "-"), X_FORWARDED_IP, (ns_client_ip != "-"), ns_client_ip)

BP9906
Builder

Thanks! Does this go on both indexer and search head?

0 Karma

maraman_splunk
Splunk Employee
Splunk Employee

this is a search time settings so will have no effect on a indexer (but should be in a TA which will be deployed on both SH and IDX)

0 Karma

domenico_perre
Path Finder

Looks like the regex is being changed when I post, here is a working one

\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}

Try this regex as a search time extraction.

0 Karma

BP9906
Builder

The Regex is fine for IP addresses, but there's no if logic to assess which IP to use since its common to have both clientip and X_FORWARDED_IP present.

0 Karma

domenico_perre
Path Finder

Can you put an example which may make it easier to understand 🙂

0 Karma

domenico_perre
Path Finder

Is there a time where all fields will be the same?

0 Karma

BP9906
Builder

The X_FORWARDED_IP commonly matches clientip or SOURCE_IP, but depending on the load balancer configuration either clientip is "-" or SOURCE_IP is "-", they never match.

0 Karma

domenico_perre
Path Finder

Regex is your friend .

Create a search time field extraction for the following

\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}

This will grab the IP address. click extract new fields and then I prefer to write My own regex.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...