Getting Data In

Can you set up multiple filters for one log?

agentguerry
Path Finder

On your splunk indexers, can you set up multiple filters for one specific log?
I've been looking around, but maybe i'm not searching correctly.

ex.

/var/log/messages would have lines in it such as:

info: connected to remote host server.com
error: cannot ssh remote host server.com
critical: cannot ping remote host server.com

And say you want to filter out the lines that contain "info" and "error"

Can I have a "pre-indexing" filter to nullqueue any lines that have "info" or "error"?

Thanks.

0 Karma
1 Solution

rsennett_splunk
Splunk Employee
Splunk Employee

Yes, You can have multiple transforms, and the context of your question (can I have multiple) you want individual TRANSFORMS- directives

In your case, to keep everything except events for info: and error: you would use two directives in props and match them in transforms.conf

This scenario is basically what the doc example shows: http://docs.splunk.com/Documentation/Splunk/6.2.2/Forwarding/Routeandfilterdatad#Filter_event_data_a...

  1. In props.conf:

    [source::/var/log/messages]
    

    TRANSFORMS-nullcrit = setnull-crit
    TRANSFORMS-nullerror = setnull-error

  2. In transforms.conf:

    [setnull-error]
    REGEX = error:
    DEST_KEY = queue
    FORMAT = nullQueue

    [setnull-crit]
    REGEX = critical:
    DEST_KEY = queue
    FORMAT = nullQueue

With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!

View solution in original post

rsennett_splunk
Splunk Employee
Splunk Employee

Yes, You can have multiple transforms, and the context of your question (can I have multiple) you want individual TRANSFORMS- directives

In your case, to keep everything except events for info: and error: you would use two directives in props and match them in transforms.conf

This scenario is basically what the doc example shows: http://docs.splunk.com/Documentation/Splunk/6.2.2/Forwarding/Routeandfilterdatad#Filter_event_data_a...

  1. In props.conf:

    [source::/var/log/messages]
    

    TRANSFORMS-nullcrit = setnull-crit
    TRANSFORMS-nullerror = setnull-error

  2. In transforms.conf:

    [setnull-error]
    REGEX = error:
    DEST_KEY = queue
    FORMAT = nullQueue

    [setnull-crit]
    REGEX = critical:
    DEST_KEY = queue
    FORMAT = nullQueue

With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!

rsennett_splunk
Splunk Employee
Splunk Employee

If this works for you, please come back and accept my answer. Thanks!

With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!
0 Karma

agentguerry
Path Finder

I guess I should specify.

I'd want to keep any message in /var/log/messages, but then not index "nullQ" error and critical

Should something like this be possible on the indexer?

props.conf
[source::/var/log/messages]
TRANSFORMS-set= setnull-error,setnull-crit

transforms.conf
[setnull-error]
REGEX = error
DEST_KEY = queue
FORMAT = nullQueue

[setnull-crit]
REGEX = critical
DEST_KEY = queue
FORMAT = nullQueue

0 Karma

rsennett_splunk
Splunk Employee
Splunk Employee

I adjusted my answer to address the question as you show it. You need two separate TRANSFORMS directives on separate lines... and that'll do it.

With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!
0 Karma

agentguerry
Path Finder

I cannot wait to test this out! beautiful!

0 Karma

rsennett_splunk
Splunk Employee
Splunk Employee

awesome. 🙂
Sometimes when you first start messing with filtering... you might not see the results straight away for various reasons, all of them related to the act of testing. 🙂
A nice way to be sure you have no syntax errors, and see a nice satisfying result without having to restart the indexer, is to create a subset of sample data from your messages log.

add a stanza to props like this:
[nulltest]
TRANSFORMS-nullcrit = setnull-crit
TRANSFORMS-nullerror = setnullerror

and create the matching transforms.conf

Then, use the data onboarding wizard to just look at what the data will look like... if you pick your sourcetype (from the "custom" list) it will resolve it and you should see only the ones you are looking for (as long as the sample is deliberately short, of course) Super cool upgrade to the Onboarding wizard (formerly data previewer) - this way you don't have to pollute an index... since you don't have to really go all the way thru the wizard to see results.

Once you are sure... then put the props directives in the proper place and you'll have to restart the indexer to change the behavior on data that is already coming into Splunk...

With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!
0 Karma

agentguerry
Path Finder

I was under the assumption you have to restart splunk in order for the indexer to read its props/transforms files.

Is that not the case?

0 Karma

rsennett_splunk
Splunk Employee
Splunk Employee

only if you are making changes to an existing data source.

However, to test them... if you are going through the motions of on boarding something as "new", as with a sample file, the on boarding wizard will show you what it will look like if you did, go all the way to the index step. In fact, if you have a 'sandbox' index you could, go all the way and see the data indexed properly...
New data source, = no restart
Change to existing data source = restart

When you apply the changes to your REAL data, that is already being indexed... you have to restart for Splunk to apply the changes.

With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...