Splunk Search

Filter several strings in transforms.conf

gozulin
Communicator

EDITED to add relevant info:

I'm trying to prevent indexing of entries containing certain strings (ACDB0000,ACM0033,W0032,L0041, \[DEBUG\])

This stanza worked fine when all I wanted to filter was debug entries:

#old transforms.conf
[setnull]
REGEX = \[DEBUG\])
DEST_KEY = queue
FORMAT = nullQueue 

But when I add a few more string to the REGEX, failure ensues. The log events containing ACM0033 and ACDB0000 aren't getting filtered out. They are still getting indexed. This is the new transforms file:

#new transforms.conf
[setnull]
REGEX = (W0032|L0041|ACM0033|ACDB0000|\[DEBUG\]) 
DEST_KEY = queue
FORMAT = nullQueue

This is my props.conf for both configs:

# same props.conf
[default]
TRANSFORMS-null = setnull
CHARSET = AUTO
NO_BINARY_CHECK = 1
pulldown_type = 1

[foo-prod]
TIME_FORMAT = %b %d %H:%M:%S
NO_BINARY_CHECK = 1
pulldown_type = 1

The log entries in question look something like this. A date, a level of severity in brackets and then a string of varying length. All generated in the standard Unix syslog format

2014-05-13 22:56:20,988 [INFO] ACDB0000: ACDB_LOG - IncomingRequest. guid=AN-ON method=register idx=0 <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Body><ns2:register 
0 Karma

gozulin
Communicator

What is the reason for changing the format from nullQueue to indexQueue?

Also, what is the function of:

[changeme_setnull]
REGEX =  .
DEST_KEY = queue
FORMAT = nullQueue

I would appreciate it if you would explain your reasoning to me so I could learn more.

0 Karma

gozulin
Communicator

I have edited my post to answer your question. Please let me know if you'd like more info.

0 Karma

alemarzu
Motivator

props.conf

[default]
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%9N
TRANSFORMS-changeme = changeme_setnull, changeme_setparsing
CHARSET = AUTO
NO_BINARY_CHECK = 1
pulldown_type = 1

transforms.conf

[changeme_setparsing]
REGEX = (W0032|L0041|ACM0033|ACDB0000|[DEBUG])
DEST_KEY = queue
FORMAT = indexQueue

[changeme_setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

What kind of log u have there ? I'm not sure but you can try this. (kinda new to splunk so bare with me)

0 Karma

gozulin
Communicator

What is the reason for changing the format from nullQueue to indexQueue?

Also, what is the function of:

[changeme_setnull]
REGEX =  .
DEST_KEY = queue
FORMAT = nullQueue

I would appreciate it if you would explain your reasoning to me so I could learn more.

0 Karma

gozulin
Communicator

I tried this search query:
W0032 OR L0041 OR ACM0033 OR ACDB0000 OR \[DEBUG\]
It finds ACM0033 and ACDB0000 entries but DEBUG, W0032 and L0041 appear to be filtered properly.

0 Karma

alemarzu
Motivator

Am I wrong or its a multiline log ? Did u try your regex in a search query ?

btw: I've edited the transforms.conf check that out if it works.

0 Karma

alemarzu
Motivator

As Rich said, can u provide some sample ? And what was the outcome ?

0 Karma

gozulin
Communicator

they are syslog generated on linux/solaris machines

0 Karma

richgalloway
SplunkTrust
SplunkTrust

How is it not working? Filtering too much or too little? Please provide some sample log events.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...