Getting Data In

Problem Removing Leading Zeros From IP Address

rbacon
Path Finder

Splunk is indexing a CSV file that contains an IP address and it looks something like this:

"Windows 7","SSHEFFIER8GDAOC","010.003.002.059","101BD9089D18"

The IP has leading zeros which I need removed and prefer to do so at index time, so based on what I've seen in the forums here I added the following SEDCMD line to the relevant props.conf file:

[my:sourcetype]
...(some field extractions)....
EXTRACT-LD_IPAddress = (?:[^"\n]"){17}(?P[^"]+)
SEDCMD = s/(src=|dst=)0
([^.]+.)0*([^.]+.)0*([^.]+.)0*(\d+)/\1\2\3\4\5/g

This seems to have no effect on the data. I double checked that the props.conf file was deployed to the indexers. Is there something wrong with the way I did the SEDCMD (what is src & dst?)? Could it be that the SEDCMD needs to be placed before the EXTRACT lines in props.conf? Also, I wonder if the quotes around the IP address could be affecting this?

Thanks for your help.

Tags (2)
0 Karma

rbacon
Path Finder

So here's what's being deployed in the props.conf file to the universal forwarders. I'm getting the data but the SEDCMD isn't removing the leading zeros from IP addresses.

[source::file location]

[sourcetype]
FIELD_DELIMITER=,
FIELD_QUOTE = "
DATETIME_CONFIG = CURRENT
INDEXED_EXTRACTIONS = csv
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = false
SEDCMD-StripLeadingZeros = s/(src=|dst=)0*([^.]+.)0*([^.]+.)0*([^.]+.)0*(\d+)/\1\2\3\4\5/g

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

If you're applying INDEXED_EXTRACTIONS=csv then these events are cooked even on universal forwarders... so anything you do with those events would have to be configured there. See http://wiki.splunk.com/Community:HowIndexingWorks section 4, you're going through the structuredparsing queue on the forwarder.

rbacon
Path Finder

So I copied the SEDCMD line to the props.conf belonging to the SplunkForwarder and reloaded the deploy server. The changes were imported and it reread the CSV file, but it did not remove the leading zeros from the IP address.

0 Karma

rbacon
Path Finder

Could it be something else? I'm sort of out of ideas at this point. I even tried removing the leading zeros at search time but had a problem (I think maybe the quotes around the IP address, see below). I'd rather have this done at index time. Should I try to use a transform?

(this didn't work either ... | rex field=youripfield mode=sed "s/.0+/./g")

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

I've just taken another look at your question - make sure you use SEDCMD-something = ... and not SEDCMD = ....

0 Karma

rbacon
Path Finder

Hi Martin. Is the arbitrary? I changed it to SEDCMD-RemoveLeadingZeros but this didn't help either.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Yeah, that's arbitrary to avoid multiple SEDCMDs overwriting each other.

0 Karma

rbacon
Path Finder

Still no luck! I'm going bald from pulling my hair out!

0 Karma

rbacon
Path Finder

Also, no indication of what happened in the splunkd.log of the SplunkForwarder.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Copy the index-time settings to the forwarder, yes. Don't move in case you ever input such a file locally. Also don't move search-time field extractions.

0 Karma

rbacon
Path Finder

OK. So I should move the SEDCMD from the props.conf on the Indexer to forwarder, right? I suppose I could move the field extractions there as well.

0 Karma

wrangler2x
Motivator

SEDCMD happens at the parsing stage, so it applies in a heavy forwarder or in a indexer. If your forwarder is a heavy forwarder, the log data is already "cooked" when it arrives at the indexer, and the sedcmd in props.conf there will have no effect. If the forwarder is not a heavy forwarder, this should work on the indexer.

0 Karma

rbacon
Path Finder

Hi. We are not using a heavy forwarder.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Is this indexed using INDEXED_EXTRACTIONS=csv on the forwarder?

0 Karma

rbacon
Path Finder

Yes, in the props.conf file.

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...