Need help with nullQueue (specifics included)

[echojacques]

Hi everyone,

For a few days now I've been tweaking my props.conf, transforms.conf, and rebooting Splunk trying to exclude certain events from being indexed (nullQueue). I have included the stanzas in my props.conf, transforms.conf, and an example of an event that I am trying to exclude below. I'm hoping that someone can save me another week of tweaking and rebooting Splunk trying to get this nullQueue to work 🙂

My app-local-props.conf file contains:

[source::*opsec*]
TRANSFORMS-null:setnull

Note: I'm not sure if I identifed the source correctly, you can see the full long path in the example event I included below.

My app-local-transforms.conf file contains:

[setnull]
REGEX=(?m)^service=(80)
DEST_KEY=queue
FORMAT=nullQueue

What I'm trying to exclude:

Here's an example of one of the multi-line events that contain "service=80" that I'm trying to send to the nullQueue. I modified the original event to shorten the length and also changed the IP addresses. You can see "service=80" near the middle of the second line:

loc=12345|time=19Aug2013 14:44:28|action=drop|orig=10.10.10.10|i/f_dir=inbound|i/f_name=eth1-01|has_accounting=0|src=10.10.10.10|s_port=49528|dst=10.10.10.10|service=80|proto=tcp|message_info=Address spoofinghost=SPLUNK-01 | sourcetype=opsec | source=/data/splunk/etc/apps/Splunk_TA_opseclea_linux22/bin/lea-loggrabber.sh --configentity SplunkLEA

I've tried many different variations (at least 20) of the REGEX but nothing has worked so far. Any advice or guidance is very appreciated!!

Thanks

[echojacques · Accepted solution]

Hi everyone,

Splunk support found the problem: the quotes around "source=80". I changed my transforms.conf REGEX to:

[setnullopsec]
REGEX=service=80
DEST_KEY=queue
FORMAT=nullQueue

... and it worked!!! I had tried service=80 (without the quotes) before but I must have had a problem with my props.conf source stanza when I had tried it. I guess my source and REGEX were both slightly off.

Thanks for all the help!

[Chubbybunny]

just to add some context, here is an example of both props and transforms settings to route the events to Null:

props.conf

[opsec]
TRANSFORMS = null

transforms.conf

[null]
REGEX=service=80
DEST_KEY = queue
FORMAT = nullQueue

[sowings]

Sourcetype definitions in props.conf don't use "sourcetype::" as a prefix. If your type is called "mytype" the stanza in props.conf would be [mytype].

Strictly speaking, I don't think you need the (?m) as events coming out of the LEA aren't multiline, as best as I can remember. Instead, they're simply long, and wrap in the UI.

I test my regexes for correctness in the UI before applying the transforms.conf rule to drop the events (making sure I got the right syntax before applying a potentially destructive rule). You could use regexr or other tools as Ayn suggests, but if you've already got similar events in Splunk and just want to validate the regex, try using the regex command:

sourcetype=opsec | regex _raw="service=80" This only returns rows where the whole line (_raw field) contains the service=80 string. If you get events back, then you know you've found the rule. But check the discovered service field, as this would match service=8000 or whatever as well. You might want to use the | as anchors in your regex, but remember that those would have to be escaped:

sourcetype=opsec | regex _raw="|service=80|". When you're satisfied that you have the regex right, cut and paste it into your transforms, restart your indexer, bingo.

[echojacques]

sowings: Yes, I restarted Splunk after the change. I have Splunk (enterprise) with Enterprise Security (this is the main Splunk indexer). It's Linux server and the Checkpoint (OPSEC LEA) is not being forwarded from another system. The Splunk Linux indexer has the OPSECLEA TA installed to received the Checkpoint data.

Thanks

[sowings]

Did you restart your indexer after making this change?

Can you tell me what kind of Splunk you have installed on the box that is collecting the checkpoint data? Is it the same as the indexer, or some other system?

[echojacques]

Sure, here are my current props and transforms stanzas that are now in the global home-etc-system-local directory:

PROPS.CONF

[opsec]
TRANSFORMS-null = setnullopsec

TRANSFORMS.CONF

[setnullopsec]
REGEX="service=80"
DEST_KEY=queue
FORMAT=nullQueue

Thanks again for your help.

[lukejadamec]

Can you repost your current props and transforms stanzas?

[echojacques]

Unfortunately it's still not working. I modified my props stanza to [opsec] and used the simple REGEX="service=80". I have also modified the props.conf and transforms.conf in the global directory instead of the app directory and not working either. I'm all out of ideas on this... might need to call for Splunk support.

[echojacques]

sowings: Thanks very much for all of this information. I have removed (?m) from my REGEX and I'm now testing this REGEX in my transforms:

[setnull]
REGEX="service=80"

I'll use the | anchor to limit to "80" once I get "service=80" to work (trying to keep it simple at first).

I did test my regex in regexpal.com and also in the search UI before modifying it in transforms. But I wasn't using the "regex_raw" command before (I was just typing "service=80" etc. into the UI search) so thanks for that tip.

I'll test again this morning and let you know if it works.
Thanks!

[Ayn]

I can see a couple of problems.

First and most importantly, your regex is looking for "service=80" at the start of the line. (That's the significance of ^). So your regex will not match your sample event. I would advise you to try your regexes in a tool like http://regexpal.com/ or http://www.gskinner.com/RegExr/ so you know that they match correctly.

Also, you should make sure that the source stanza ([source::*opsec*]) is being identified correctly. Wildcards will match anything but slashes so if you do have slashes in your source, chances are your settings for this source aren't being applied at all.

Finally you have "TRANSFORMS-null:setnull" - there should not be a colon there, it should be an equals sign, so "TRANSFORMS-null = setnull".

[echojacques]

Ok, I have "moved" the nullQueue from the APP-local directory to the home-etc-system-local directory and still not working...

[echojacques]

lukejadamec: thanks for the explanation, that makes sense. I'll test it at the global level today.

kristian: i have changed my props stanza to [opsec]. I'll let you know if it works after rebooting and testing.

I have another nullQueue (that works) setup for my Windows WMI data. It is also using: TRANSFORMS-null = setnull. Is it OK that it has the same name (setnull) or should I change this nullQueue name to setnull-opsec?

Thanks!

[kristian_kolb]

echojaques: if the sourcetype is opsec then the stanza in props.conf should be;

[opsec]
TRANSFORMS-null = setnull

Doing it in etc/system/local just ensures that the setting will have the highest priority and will always be active, regardelss of whether you enable/disable certain apps.

/K

[lukejadamec]

Global stanzas should go at the top (best practice), correct stanzas work where they belong.
Kristian gave you the quick answer to this document: http://docs.splunk.com/Documentation/Splunk/4.3.4/Admin/Wheretofindtheconfigurationfiles
Which basically says that index time activity happens first at the global level, which is controlled first by the etc/system/local configuration files.

[echojacques]

I had to disable the app since I was getting close to exceeding my license due to this problem.

Will try again tomorrow.

[echojacques]

Using sourcetype::opsec instead did not work...

[echojacques]

Also, not sure if it matters, but I have the stanzas at the top/beginning of my props.conf and transforms.conf files.

[echojacques]

kristian,

So I will modify my props.conf to use the sourcetype like this:

[sourcetype::opsec]
 TRANSFORMS-null=setnull

I will reboot and let you know if it works. Also, I thought I had to make the changes in the APP directory??

[kristian_kolb]

Use your sourcetype instead of the source in props.conf, if possible.

and yes, do it in local, not in default. Just to ensure that it is going to happen, make the changes to props/transforms in $SPLUNK_HOME/etc/system/local

[echojacques]

Oh, and the opsec source is actually the OPSEC LEA add-on.