Deployment Architecture

What does this search head cluster function alert WARN messages mean?

spectrum2035
Explorer

Every other day, we are getting following error on the internal index. Nearly 65,000 messages are generated for less than 15mins. What does this error actually mean?

_WARN  SHCFunctions - alert csv wrong action  csv = key,expire,ACTION,MD5,"__mv_key","__mv_expire","__mv_ACTION","__mv_MD5"\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n_

DavidHourani
Super Champion

Hi @spectrum2035,

Do you still have this issue ? Seems like a misconfigured lookup or alert action to generate a csv. can you try to link this to any newly added alert action ?

0 Karma

smitra_splunk
Splunk Employee
Splunk Employee

facing same problem ... no clues .... doesn't look like there is a correlation to errors reported by other splunkd logging components. just sudden spikes of SHCFunctions warnings.

0 Karma

amitm05
Builder

@spectrum2035
Only the error does not give much info. Can you try to add some more info about the error ?
I am guessing if SHC means Search Head Cluster. Please check if you are able to find any errors/warnings in Monitoring Console on your search head dashboards and any warnings on General Health checks

0 Karma

spectrum2035
Explorer

I did check the general health status of the SHC in DMC and couldnt find anything alarming...

Following are the 4 logs which was indexed just before the event happened....

I ACCESS [conn47] Successfully authenticated as principal __system on local
I NETWORK [thread1] connection accepted from 10.10.10.3:50374 #47 (23 connections now open)
127.0.0.1 - splunk-system-user [25/Jun/2019:16:16:04.090 +0100] "GET /services/data/inputs/threatlist?output_mode=json&search=disabled%3D%22false%22 HTTP/1.0" 200 41063 - - - 92ms
I ACCESS [conn20] Successfully authenticated as principal __system on local

If I look back to the earlier one's i have license usage events OR StatusMgr related events.. so there is no specific pattern..

0 Karma

skalliger
SplunkTrust
SplunkTrust

That's just a wild guess: Are you using Enterprise Security? And on Windows?

Skalli

0 Karma

spectrum2035
Explorer

Yes we are using ES but on RHEL

0 Karma

adonio
Ultra Champion

check ES version and Splunk version compatibility:
https://docs.splunk.com/Documentation/VersionCompatibility/current/Matrix/CompatMatrix
contact splunk support too

0 Karma

spectrum2035
Explorer

Thanks adonio, we have upgraded our servers nearly a year back and this started showing up for last 1 month only.

0 Karma

skalliger
SplunkTrust
SplunkTrust

I've never seen that logging category and I don't see SHCFunctions in the log.cfg either. Is that some custom app that logs into your _internal index?

Skalli

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...