About tccooper

tccooper · ‎01-15-2020

As of midnight, 1/15/2020, we have about 3.5 Indexes which are no longer indexing the entirety of the XML files being monitored on the UF. I have looked through the splunkd.log files and searched throughout the _internal indexes to see if anything obvious is sticking out, but that's not the case. The XML files are being parsed up through the header and sometimes to the first line, but the remainder does not show up in the system. Everything was working fine up until now. Any help would be appreciated!

tccooper · ‎08-31-2017

I have the following query index="XXXXXXXXXX" Device="*FPB*" OR Device="*VAV*" Point_Name="ActFlow" |bin span=15m _time |stats last(Value) as AirFlow by Device, _time |appendcols [|search index="XXXXXXXXXX" Device="*FPB*" OR Device="*VAV*" Point_Name="ActFlowSP" |bin span=15m _time |stats last(Value) as Setpoint by Device, _time] |eval difference = (1 - (Setpoint/AirFlow)) |where difference > .2 |fields _time, Device, difference The issue is the results of the subquery are not grouping based off the Device. Suggestions on how to fix?

tccooper · ‎08-09-2016

tccooper · ‎08-09-2016

This is an example of what the original query returns: Device Point_Name-D Point_Name-B Point_Name-C difference Dev_1 57.53 55 2.53 Dev_2 57.25 55 2.25 Dev_3 58.01 55 3.01 Dev_4 77.71 58 19.71 Dev_5 64.12 58 6.12 The format here is ideal for what we are trying to achieve, but we also need the "_time" field appended as well. I have tried replicating this in a timechart query with a span=15m variable, but that was not printing out what we needed. Since not all these records are coming in at the same moment, I thought the "timechart span=15m" would give us a nicely rounded _time field to work work for writing to the summary index.

tccooper · ‎08-09-2016

We are trying to chart multiple results with some success. I am able to have everything sorted based off the Device correctly. My issue is this is going in a summary index and I need to include the timestamp so we can have accurate results of when everything broke. Here is the existing query: index="XXXXXXXXXX" Device="*AAAAA*" Point_Name="BBBBBB" OR Point_Name="CCCCC" OR Point_Name="DDDDD"|chart limit=0 last(Value) over Device by Point_Name|eval difference = if(isnull(CCCCC),(AAAAA-DDDDD),(AAAAA-CCCCC))|where difference > 2 This gives me once record per Device where the difference is greater then 2 . However, I know there are multiple times throughout the day where this condition is broken. I need to sort out each record so we can count the number of consecutive 15 minute windows that this query returns results based on Device . Thanks in advance for help.

tccooper · ‎03-15-2016

No errors/warnings posted in the _internal index in the previous 5 minutes leading up to the OOM force close, just INFO level metrics logs. Here is the process usage printout from /var/log/messages when SplunkD invoked the OOM killer: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name [ 1130] 0 1130 2735 101 11 3 0 -1000 udevd [ 1231] 0 1231 2734 96 10 3 0 -1000 udevd [ 1233] 0 1233 2734 102 10 3 0 -1000 udevd [ 1567] 0 1567 2340 123 10 3 0 0 dhclient [ 1608] 0 1608 28024 114 24 3 0 -1000 auditd [ 1626] 0 1626 61894 634 24 4 0 0 rsyslogd [ 1637] 0 1637 3459 71 10 3 0 0 irqbalance [ 1648] 81 1648 5448 59 14 3 0 0 dbus-daemon [ 1732] 0 1732 19454 204 39 3 0 -1000 sshd [ 1749] 38 1749 7321 144 19 3 0 0 ntpd [ 1764] 0 1764 22240 461 45 3 0 0 sendmail [ 1772] 51 1772 20104 367 41 3 0 0 sendmail [ 2078] 0 2078 29879 158 15 3 0 0 crond [ 2088] 0 2088 4267 40 12 3 0 0 atd [ 2114] 0 2114 1615 31 9 3 0 0 agetty [ 2115] 0 2115 1078 24 8 3 0 0 mingetty [ 2119] 0 2119 1078 23 8 3 0 0 mingetty [ 2122] 0 2122 1078 23 8 4 0 0 mingetty [ 2124] 0 2124 1078 23 8 3 0 0 mingetty [ 2126] 0 2126 1078 24 8 3 0 0 mingetty [ 2128] 0 2128 1078 24 8 3 0 0 mingetty [20972] 0 20972 8025350 7442358 15342 34 0 0 splunkd [20973] 0 20973 14413 1135 25 3 0 -1000 splunkd [20987] 0 20987 63977 7339 58 3 0 0 mongod [21065] 0 21065 525407 43490 217 5 0 0 python [21088] 0 21088 24650 2038 48 3 0 0 splunkd [26700] 0 26700 42073 20408 76 3 0 0 splunkd [26703] 0 26703 14415 1082 21 3 0 -1000 splunkd Out of memory: Kill process 20972 (splunkd) score 938 or sacrifice child Killed process 20972 (splunkd) total-vm:32101400kB, anon-rss:29769432kB, file-rss:0kB

tccooper · ‎03-15-2016

I have a search head running splunk 6.2.3, in a non-clustered distributed environment, which is sporadically having the Linux OOM killer cause the splunkd process to crash. I have looked into the known THP issue and that isn't the cause (the particular type of VM it's on doesn't support THP). There is no rhyme or reason as to why it is forced to crash. Sometimes is takes 1 hour, sometimes it takes 3 hours. The VM is question is fairly robust memory wise (30 GB). I'm also kinda new to splunk so I'm hoping there is a simple fix that i haven't run across yet while crawling through here. Any and all help would be appreciated.

Posts	7
Solutions	0
Karma Given	0
Karma Received	3
Member Since	‎03-15-2016

Online Status	Offline
Date Last Visited	‎09-24-2020 04:06 PM

Data randomly stopped indexing full xml file

Can appendcols be used for grouping?

Chart multiple columns based on time and additiona...

SplunkD Causing Linux OOM Condition

Data randomly stopped indexing full xml file

Can appendcols be used for grouping?

Re: Chart multiple columns based on time and addit...

Re: Chart multiple columns based on time and addit...

Chart multiple columns based on time and additiona...

Re: SplunkD Causing Linux OOM Condition

SplunkD Causing Linux OOM Condition