Splunk Search

Delete/Ignore specified value -updated-

Toups
Explorer

I have searched the documentation and have not yet found how to omit or delete specific fields from an input.

The input in question is a ASCII string and we would like to remove the "irellivant" data prior to indexing and segmentation for a couple reasons. As always, thank you all in advance for your assistance.

  1. Few feilds indexing = optimized indexing and searching
  2. Minimize the volume of data (where practical) to maximize the storage capabilities.

Below is a sample log entry actual telephone numbers are masked here for secuirty. An example of the information we would like to omit would be (\x00\x00\x00) which is a null value indicating the end of the previous line.

"\x00\x00\x00160900429  C9E820     28155512127135552121                        20080310  #850 1      010"

ftk
Motivator

I've updated my answer based on your sample log

0 Karma

ftk
Motivator

You can do this at index time using SEDCMD and a regular expression designed to capture the text you want to remove.

Now you didn't post any sample data, but let's consider this simple scenario.

Sample Data:

10.0.0.2 WEBHOST1 Error:0xf0388d Level:Severe User:foo (c)1999 Prince productions ltd. All right reserved. The format of this log message is proprietary haha!
10.0.0.2 WEBHOST1 Error:0xf9990c Level:Info User:foo (c)1999 Prince productions ltd. All right reserved. The format of this log message is proprietary haha!

Now here the superfluous string is obvious, it's the silly copyright message. So in our props.conf we would use SEDCMD to strip the copyright notice for our monitor. In this example I assume that the sourcetype of these events is "silly_source".

Props.conf:

[silly_source]
SEDCMD-stripcopyright = s/\(c\)1999 Prince productions ltd. All right reserved. The format of this log message is proprietary haha!//g

Of course you can use any kind of regular expression in the sedcmd, so you could also use something like s/\(c\).*!$//g

I recommend that you read through the doc page on SEDCMD before you get started on this, it will help you make sense out of this.

[EDIT] RE: New sample data you can remove \x00\x00\x00 as with the following replacement regex:

s/(\\x00){3}//g

Toups
Explorer

I have updated the original post with a sample of the source log information.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...