About oscarminassian

oscarminassian · ‎02-01-2018

Hey Splunkers, As @ebaileytu has suggested and also with help of David at Splunk support, it was found to be the Universal Forwarder version. We're in the process of rolling the 7.0.1 version out to PROD. This was limited to our Windows environment and the problem has completely disappeared in our DEV/SIT and UAT environments since the upgrade! Much winning!

oscarminassian · ‎12-05-2017

I bow to your supreme knowledge, Woodcock. I found some of the events that had been moved 11 hours into the next day! I've attempted to push the TIME_BEFORE_CLOSE out to 10 seconds. Let's see what happens overnight. 🙂

oscarminassian · ‎12-05-2017

Thanks for the insights, I did some back searching in our S3 archive. Looks like we've had this issue for a long long time, it's just never been reported.

oscarminassian · ‎12-05-2017

Hi Woodcock, 100% sure events are missing in Splunk from multiple servers. I'm able to access the servers and verify a with the raw files by searching for the missing event in Splunk. I've observed a gap of 30 seconds with missing events in our SIT environment, about 400 events before it picks back up again like nothing happened. Not sure if TIME_BEFORE_CLOSE fits into this, and indexing the whole file after a day, or even after 15mins is not an option in Production. Too much monitoring and alerting.

oscarminassian · ‎12-05-2017

No Parsing errors that I can find. Initially I was unable to find any events that had come in on the wrong date time, but I found some! It was hard to track down and I pretty much came across it by accident.

oscarminassian · ‎12-05-2017

This was one of my first thoughts, we had a puppet change a few months ago that removed the cookie from the IIS logs. Oh boy, the data didn't like that. It went away after the log file rotated. I've been able to verify that it's not the case and the logging is 100% uniform across our IIS fleet.

oscarminassian · ‎12-04-2017

@harsmarvania57, sure did and no luck! 😞 Yeah, these are all IIS sourcetype. I'm using the following search to separate the bad from the good and getting lots of results. index=web sc_status!=0 | regex sc_status!= ^\d{3}$ | regex sc_status!= ^\d{4}$ | regex _raw!=^\d{4}-\d{2}-\d{2} | stats count by sc_status host Also worth mentioning that we're on 6.6.1

oscarminassian · ‎12-04-2017

Hi all, Have you ever seen a UF missing events? I’ve observed some of our UF’s missing ~8 seconds of events and then picking up halfway through the event they reach. The gaps are creating some muddy data and it doesn’t seem to be limited to one server, I’ve got a list of 100 or so across all of our environments and corresponding Splunk clusters. Here's a 3 line example of what Splunk is seeing in the source(/app/search/show_source?blah). I've been able to manually confirm that there is a gap and plenty of logs between. 2017-12-03 22:25:37 GET /Something/Something/1 from=2017-12-02&to=2017-12-04 80 - 0.0.0.0 HTTP/1.1 - - Some.url.was.here.com.au 200 0 0 00000 000 00 - HasedKeyWasHere ServiceName - 0.0.0.0 HTTP/1.1 - - ome.url.was.here.com.au 200 0 0 000 000 0 - HasedKeyWasHere ServiceName - 202017-12-03 22:25:45 GET /Something/Something/1 from=2017-12-02&to=2017-12-04 80 - 0.0.0.0 HTTP/1.1 - - Some.url.was.here.com.au 200 0 0 00000 000 00 - HasedKeyWasHere ServiceName - I've tried this with and without line breaking logic to see if it would make any difference in the props.conf with no success. Which is not entirely surprising in hindsight. It should be worth mentioning that these are all IIS logs being forwarded to a 6 peer node cluster with no heavy forwarders inbetween.

oscarminassian · ‎11-29-2017

Thanks for the help guys and sorry for the delay in reply. Turns out we forgot to remove a few searchheads from the master node after a migration and upgrade so things were a little "muddled up". Once removed we did a rolling restart of the search head cluster and things seem to have returned to normal.

oscarminassian · ‎10-27-2017

Hi guys, The exact same problem over here on 6.6.1. We have a SH-Cluster setup and I've managed to clear the issue from one search head by restarting the Splunkd service, only to have it move over to another search head in the cluster! Rince repeat, the issue will only ever be on one search head at a time. I haven't observed any down the line impact, but the error is annoying! Oscar

oscarminassian · ‎10-09-2017

This is really great @ftk! I re-purposed it for a SQL replication alert that is often very very spiky (Values from 0 up 25 000 and back to 0 in 3-minute span). I changed the logic a little to ensure we had an actual problem for a set period of time. The following search is for 5 data points over a 5min time frame. index=sql source=blah sourcetype=sp_pendingcmds | where pendingcmdcount>= 10000 | stats range(pendingcmdcount) as difference list(pendingcmdcount) as list | streamstats latest(list) as maxSelect window=1, count(list) as listcount | where listcount>=5 | eval percent_difference=((difference/maxSelect)*100) I then set the alert to check percent_difference > 50 It's working a treat. I hope it helps someone else

oscarminassian · ‎09-25-2017

Hey Splunkers, Long time lurker first-time poster. I'm doing something similar but looking up IP addresses for brute force attacks. I'm sure I'm hitting a snag with my: |eval IP=mvindex(Source_address,0) The .csv is pretty straightforward: Source_address; count 10.0.0.5; 50 10.0.0.1; 2 10.0.0.4; 4 I get values back with my search but validating them, there are some discrepencies index=yes success=1 ipaddress=* | table ipaddress | dedup ipaddress | rename ipaddress as IP | dedup IP | eval From=1 | append [| inputlookup ohyes.csv | table Source_address | eval IP=mvindex(Source_address,0) | table * | eval From=2] | stats sum(From) as From by IP | eval status=case(From=3, "Present in both", From=1, "Only in search", 1=1, "Only in CSV") | where status="Present in both" | table IP status Any help would be greatly appreciated!

Posts	12
Solutions	1
Karma Given	13
Karma Received	1
Member Since	‎05-30-2017

Online Status	Offline
Date Last Visited	‎01-31-2021 11:42 PM

Why is Splunk universal forwarder missing events?

Re: Splunk Universal Forwarder missing events

Re: Splunk Universal Forwarder missing events

Re: Splunk Universal Forwarder missing events

Re: Splunk Universal Forwarder missing events

Re: Splunk Universal Forwarder missing events

Re: Splunk Universal Forwarder missing events

Re: Splunk Universal Forwarder missing events

Why is Splunk universal forwarder missing events?

Re: KV Store failing at SHC

Re: KV Store failing at SHC

Re: Rapid growth of values, then alerting on it.

Re: How to compare search and csv file?