About baf879

baf879 · ‎03-08-2018

The way mine works is actually like this: blacklist1 = EventCode="4688" Message="(A new process has been created)(?s).*(splunk-\w*mon|splunk-MonitorNoHandle)" So the blacklist feature in the case looks for Windows Security Event Log events with an EventCode of 4688, and then reads into the Message field and uses regex to match the message contents. For this one I believe was blacklisting EC 4688 when the splunk-*mon process was in the message or splunk-MonitorNoHandle was in the message. Another working example: blacklist3 = EventCode="4688" Message="(A new process has been created)(?s).*(GoogleUpdate\.exe|FlashPlayerUpdateService\.exe)" Again, looking at EventCode of 4688, and then checking for a match when the message contains "A new process has been created" and either GoogleUpdate.exe or FlashPlayerUpdateService.exe. If it matches, it gets dropped by the UF.

baf879 · ‎04-05-2017

Splunk support did contact me this morning. We weren't able to determine an exact cause of this behavior, but did find that stopping the Splunkd process caused it to stop hanging. Specifically, the command prompt window where I had run "Splunk offline" displayed a message indicating that primaries had been reassigned and it was complete. I set the splunk service to start manually, rebooted the server and then installed the Splunk 6.4.6 update. It appears to be working now - it rejoined the cluster and I have not seen any signs that there is a problem. *** I'll accept this as an answer with a caveat. I recommend contacting Splunk support in this situation, as they may identify something in the splunkd.log that points to a root cause, or may indicate that you should not terminate the process like I did ***

baf879 · ‎04-04-2017

Windows Server 2012 for the indexers. Windows Server 2012 R2 for the cluster master. Looking in splunkd.log on the CM, nothing that seems out of place (to me, at least). I see error messages about regex statements hitting a match limit (I use regex to blacklist some events), some warnings about cooked connections, and one of my search heads which is currently offline. Most of the log contains INFO events pertaining to CMBucket - event=isFixupComplete

baf879 · ‎04-04-2017

I've opened a support ticket but hoping someone may have seen this. I have an indexer cluster with two indexers and a cluster master and I'm upgrading all of them from 6.4.3 to 6.4.6. CM was upgraded and placed into maintenance mode. Indexer 1 was taken offline (by using "splunk offline"), upgraded and rebooted. On Indexer 2, issued a "splunk offline" command, and it's still running 5 hours later. The machine isn't locked - the status "dots" keep filling up the command window. Has anyone encountered this, or is anyone aware of a way to check the actual offline status and possibly close the window? I was following along with the upgrade procedure, but can't find any mention of this situation anywhere.

baf879 · ‎01-08-2016

Thanks! I actually got it working using this (right about the same time you posted): blacklist1 = EventCode="4688" Message="(A new process has been created)(?s).*(splunk-\w*mon)" Are there any advantages or disadvantages to doing it my way as opposed to your suggestion?

baf879 · ‎01-08-2016

Thank you for the input. I can see the advantage of "centralizing" the process of discarding events. We use a deployment server so while I may not "own" explicit administrative access to them, I can effect change on them via that method. Frankly, I'm not sure how much data is being discarded (or the best way to determine that). We had an explosion in our Windows security log events - to the tune of over 100 GB per day. My license is for 50GB so you can see how that is a problem 🙂 . At the indexer, I used props/transforms (with a regex provided from another group at my location) to drop some of our most frequently occurring events. Once that started, my licensing dropped down to just above 50GB per day. Around the same time, I started noticing the indexing latency, so I was trying to see if I could push that filtering capability out to the clients to effectively rule it out as the root cause. I haven't made changes in production yet, so it remains to be seen if the latency will be affected (if at all). But thanks to everyone's help, I now have a sort of template to follow if and when I need to drop other events.

baf879 · ‎01-08-2016

I've tried posting the solution twice, but it isn't showing up. I don't know if it's stuck awaiting moderation or what....

baf879 · ‎01-08-2016

All: Thanks again for all of the research and suggestions. I was able to get this resolved late yesterday - I posted my solution to this page but for whatever reason it isn't showing up so I'll try it again. Through a combination of reading (and re-reading, and staring, and re-reading) the inputs.conf.spec and flat out trying everything, this was the working solution: blacklist1 = EventCode="4688" Message="(A new process has been created)(?s).*(splunk-winprintmon)" blacklist2 = EventCode="4688" Message="(A new process has been created)(?s).*(splunk-regmon)" My next step is to consolidate these two lines into one, and include the other Splunk UF events I don't currently need (splunk-netmon, splunk-admon). The UF is limited to 10 blacklist items, so I'm thinking there is a way via regex to look for "splunk-*mon.exe". If anyone has a quick fix for that, I'd be glad to hear it. I appreciate everyone's suggestions and ideas, it was a big help in getting this implemented!

baf879 · ‎01-07-2016

No luck this time. I did comment out the entries you said probably aren't working, and bumped this up to #1. So my inputs.conf contains this line: blacklist1 = (?msi)EventCode=(4688).*^.*New Process Name:\s+.*(splunk-winprintmon.exe)

baf879 · ‎01-07-2016

It looks like your blacklist3 suggestion might have gotten messed up, as it's missing some asterisks that I had. Unless you purposely left them out...so should blacklist3 actually look like this: blacklist3 = (?msi)EventCode=(4688).*^.*New Process Name:\s+.*(splunk-winprintmon\.exe) This would omit the first carat, as well as the "trailing" .* . I also put the backslash between splunk-winprintmon and the .exe .

baf879 · ‎01-07-2016

OK great. Let me try removing that carat. My blacklist items are sequential (thanks for the reminder) - this one in particular was #3 on my list. The two preceding it were for Active Directory (4662 and 566); my Splunk PS engineer put that stanza there when she set up our Splunk cluster. The blacklist section actually looks like this: blacklist = 4656,5145,4985,4904,4905,4945,4957,5033,5024,5058,5440,5441,5442,5444,5632,6281,5031,5145 blacklist1 = EventCode="4662" Message="Object Type:\s+(?!groupPolicyContainer)" blacklist2 = EventCode="566" Message="Object Type:\s+(?!groupPolicyContainer)"

baf879 · ‎01-07-2016

Thanks. I am actually doing this at the indexer currently (for EventCode 4688 where my 3rd party inventory agent is being chatty). My license usage dropped significantly (by half), however, I started seeing some indexing latency. I was trying to eliminate this as a potential cause by pushing it out to the clients. Not sure if it really would cause the latency but figured it was worth investigating. Thank you for the tip regarding the .* at the end, I'll go in and remove that. There's one more thing I learned about regex today !

baf879 · ‎01-07-2016

Thanks. I am actually doing this at the indexer currently (for EventCode 4688 where my 3rd party inventory agent is being chatty). My license usage dropped significantly (by half), however, I started seeing some indexing latency. I was trying to eliminate this as a potential cause by pushing it out to the clients. Not sure if it really would cause the latency but figured it was worth investigating.

baf879 · ‎01-07-2016

Hi everyone, On my Universal Forwarder, I'm able to effectively blacklist Windows event codes when I do it based on the EventCode field. However, when I try to add regex to my blacklist entries it doesn't work. Essentially, I want to reduce the number of EventCode=4688 entries where the "New Process Name" field is coming from the Splunk client. So let's say, I want to blacklist events where the EventCode=4688 and the New Process Name contains "splunk-winprintmon.exe". Here's the contents of the actual event: 01/07/2016 12:38:09 PM LogName=Security SourceName=Microsoft Windows security auditing. EventCode=4688 EventType=0 Type=Information ComputerName=BBLAPTOP.local TaskCategory=Process Creation OpCode=Info RecordNumber=8541755 Keywords=Audit Success Message=A new process has been created. Subject: Security ID: NT AUTHORITY\SYSTEM Account Name: BBLAPTOP$ Account Domain: LOCAL Logon ID: 0x3e7 Process Information: New Process ID: 0x6c18 New Process Name: C:\Program Files\SplunkUniversalForwarder\bin\splunk-winprintmon.exe Token Elevation Type: TokenElevationTypeDefault (1) Creator Process ID: 0x2108 Process Command Line: I've been able to get "matching" regexes created when I try it out on a website like regexpal.com. The regex below matches the event text just fine on their site, but does not work with the Splunk forwarder: (?msi)^EventCode=(4688).*^.*New Process Name:\s+.*(splunk-winprintmon\.exe).* My inputs.conf contains: blacklist3 = (?msi)^EventCode=(4688).*^.*New Process Name:\s+.*(splunk-winprintmon\.exe).* I've also tried it without the leading (?msi)^ and was not successful. I really need to reduce my licensing volume as I'm frequently in violation, so if anyone has any ideas or solutions I would greatly appreciate it!

Posts	14
Solutions	2
Karma Given	0
Karma Received	6
Member Since	‎03-06-2014

Online Status	Offline
Date Last Visited	‎01-27-2022 01:36 PM

Splunk Offline command - running for hours

Universal Forwarder blacklist - Can anyone help wi...

Re: Universal Forwarder blacklist - Can anyone hel...

Re: Splunk Offline command - running for hours

Re: Splunk Offline command - running for hours

Splunk Offline command - running for hours

Re: Universal Forwarder blacklist - Can anyone hel...

Re: Universal Forwarder blacklist - Can anyone hel...

Re: Universal Forwarder blacklist - Can anyone hel...

Re: Universal Forwarder blacklist - Can anyone hel...

Re: Universal Forwarder blacklist - Can anyone hel...

Re: Universal Forwarder blacklist - Can anyone hel...

Re: Universal Forwarder blacklist - Can anyone hel...

Re: Universal Forwarder blacklist - Can anyone hel...

Re: Universal Forwarder blacklist - Can anyone hel...

Universal Forwarder blacklist - Can anyone help wi...