As mentioned by @hrawat_splunk the workaround is to disable the splunk_internal_metrics on the forwarder: $SPLUNK_HOME/etc/apps/splunk_internal_metrics/local/app.conf   [install] state = disabled   The bug (SPL-251434) will be fixed in the following releases: 9.2.2/9.1.5/9.0.10 This app: Clones all metrics.log events into a metric_log sourcetype. The metrics_log sourcetype does the necessary conversion from log to metric event, and redirects the events to the _metrics index. ... View more

How to identify if you are hitting this bug: 1.) do you have persistentQueueSize set over 20MB in inputs.conf of the forwarder: Ie: etc/system/local/inputs.conf [splunktcp-ssl:9996] persistentQueueSize = 120GB   2.) is Splunk crashing after a restart where the crashing thread is Crashing thread: typing_0? 3.) you see in splunkd_stderr.log logs like: 2024-01-11 21:16:19.669 +0000 splunkd started (build 050c9bca8588) pid=2506904 Fatal thread error: pthread_mutex_lock: Invalid argument; 117 threads active, in typing_0 ... View more

the INDEXED_EXTRACTIONS configuration belongs in props.conf of the universal forwarder.   |tstats count where index=* sourcetype=my_json_data by host | stats values(host) The search above should tell you which hosts need to be looked at where you would remove INDEXED_EXTRACTIONS = json from the SHs and Indexers and move this configuration (INDEXED_EXTRACTIONS = json) to the forwarders props.conf. Make sure the forwarder inputs.conf for the json source you are ingesting is tagging the data with the appropriate sourcetype, then in props.conf reference that sourcetype stanza for your config: ie (UF): inputs.conf [monitor:///file] sourcetype=foo_json index=bar props.conf [foo_json] INDEXED_EXTRACTIONS = json     see:https://docs.splunk.com/Documentation/Splunk/6.5.2/Admin/Configurationparametersandt[…]A.&_ga=2.147263155.568450395.1710801981-1206481253.1693859797 INDEXED_EXTRACTIONS are unique in that they happen in the structured parsing queue of the universal forwarder where usually parsing happens at a HF or indexer if there is no HF. if you use a HF as the first point of ingest and no UF then you place it there on the HF. see: https://docs.splunk.com/Documentation/SplunkCloud/latest/Data/Extractfieldsfromfileswithstructureddata If you have Splunk Cloud Platform and want configure the extraction of fields from structured data, use the Splunk universal forwarder.   ... View more

@michaelissartel I haven't tested using an env variable as the field value before. If you do end up testing can you put your reply here for others? ... View more

Please see this knowledge article for details on the known issue and fix/workaround: https://splunk.my.site.com/customer/s/article/High-Skipped-Search-Ratio-after-upgrading-to-9-x ... View more

apply to data input instance where data is first read by Splunk props.conf [too_small] PREFIX_SOURCETYPE = false PREFIX_SOURCETYPE = <boolean> * NOTE: this setting is only relevant to the "[too_small]" sourcetype. * Determines the source types that are given to files smaller than 100 lines, and are therefore not classifiable. * PREFIX_SOURCETYPE = false sets the source type to "too_small." * PREFIX_SOURCETYPE = true sets the source type to "<sourcename>-too_small", where "<sourcename>" is a cleaned up version of the filename. * The advantage of PREFIX_SOURCETYPE = true is that not all small files are classified as the same source type, and wildcard searching is often effective. * For example, a Splunk search of "sourcetype=access*" retrieves "access" files as well as "access-too_small" files. * This setting applies at input time, when data is first read by Splunk software, such as on a forwarder that has configured inputs acquiring the data. * Default: true   ... View more

you will need to install the following packages for the above commands yum install iotop -y yum install sysstat -y ... View more

This message suggests there may be storage performance issues for the path mentioned on the Search Head. You may want to check the SH disk await times (ms) with the search below for the mount point where $SPLUNK_HOME  is mounted. Below 10ms is generally considered good performance. If you are seeing await times much higher you should confirm if the mount is NVME and if not, look to move the SH $SPLUNK_HOME mount to NVME for better performance (this is where bundles, search artifacts etc are stored).   | tstats max(data.avg_total_ms) as avg_total_ms where component::iostats sourcetype=splunk_resource_usage index=_introspection host IN ("*SH1*") by host data.mount_point _time span=60s | timechart span=60s max(avg_total_ms) by data.mount_point | eval ideal_latency=10   Some options for looking further into disk performance issues: The iostat command is used for monitoring system input/output device loading by observing the time the devices are active in relation to their average transfer rates.     iostat -t -x 2 1800 >> /tmp/iostats.txt   Iotop is an open source and free utility similar to top command, that provides an easy way to monitor Linux Disk I/O usage details and prints a table of existing I/O utilization by process or threads on the systems.     iotop -botqqq --iter=1800 >> /tmp/iotop.log   ... View more

to update an old post on lookup files in the knowledge:  - if the lookup file is used in an automatic lookup, it needs to be included in the knowledge bundle and sent to the indexers , so don't blacklist those. - if the lookup file is referenced by an accelerated datamodel , it needs to be included in the knowledge bundle and sent to the indexers , so don't blacklist those. ... View more

In python.log we are hitting some local size limit imposed by the email server: index=_internal source=*python.log* "Message exceeds local size limit" The 552 error was generated by the receiving SMTP email server complaining that the email hit the size limit on the server. Every mail server has custom settings for the size of mail users can send and receive through it. These limits can either be global or individual-account specific or both. In an Exchange server, there are mainly 4 settings for message size limitations: a. Global value for the mail server b. Receive and Send Connectors setting c. SMTP Virtual Server setting d. Individual user mailbox restriction If the message size in the email exceeds any of these limits that are allowed for a particular email user account, it will be rejected with this error 552 message. To resolve a valid email bouncing with error 552, these attachment limits have to be increased or adjusted appropriately. You could contact the mail system admin to better understand limits for email attachments. ... View more

You can run from the Monitoring Console (splunk_monitoring_console app context)  to check THP effective state: | rest /services/server/sysinfo splunk_server_group=dmc_group_indexer | table splunk_server version numberOfCores, numberOfVirtualCores physicalMemoryMB ulimits.open_files transparent_hugepages.effective_state | sort +splunk_server ... View more

note: if you are trying to set the output group to your token via splunk web UI you may notice that no output groups show up. Settings>Data>Inputs>HTTP Event Collect > New Token You must configure the groups for disabled = false in outputs.conf for them to appear in the UI.   ie: [tcpout] defaultGroup = clustered_indexers_with_useACK disabled = false [tcpout:clustered_indexers_with_useACK] server=idx1.splunk.com:9997,idx2.splunk.com:9997,idx3.splunk.com:9997 useACK = true disabled = false [tcpout:clustered_indexers_without_useACK] server = idx1.splunk.com:9997,idx2.splunk.com:9997,idx3.splunk.com:9997 useACK = false disabled = false   ... View more

deployment flow: client--->load balancer--->HFs--->Indexers ... View more

- The lookup command is a distributable streaming command when local=false, which is the default setting. This means the lookup will happen on the remote peers. There are benefits of running the lookup in parallel on each peer vs just on the SH. - It only becomes an orchestrating command when local=true. This forces the lookup command to run on the search head and not on any remote peers. - If you want to determine the difference in search performance compare the search run time using (local=false vs local=true) and check the job inspector. ... View more

Below are some guidelines to follow to know if it is safe to blacklist a lookup from the knowledge bundle: 1.) the knowledge bundles are created on the SH and stored in $SPLUNK_HOME/var/run/  In a search head cluster the knowledge bundle is sent from the shc captain to the indexers. The bundle on the peers (aka indexers) will be found in:  /opt/splunk/var/run/searchpeers we can use the following command to look inside (but not untar) the bundle to see what the largest lookup files are on the SHC captain node. Look at the most recent full bundle by date:  ls -lh  tar -tvf <name>.bundle | grep csv | sort -nk3 The largest files will show up at the bottom of the list.  2.) Take the top 5-10 largest lookup csv files and before blacklisting them: confirm the lookup file is not used in an automatic lookup, if it is, the indexers need the lookup file and it must be included in the bundle when replicated from SH-->indexer so we do not want to blacklist lookups that are automatic lookups.  This you can check with the following search:  lookup files which are configured for automatic lookups will be defined on the SH's transforms.conf  "filename" attribute (don't blacklist these from the knowledge bundle):  | rest /services/configs/conf-transforms splunk_server=<shc captain> | search filename=*| table splunk_server eai:acl.app title filename id | rename eai:acl.app as app title as stanza If they appear in the search below do not blacklist.  | rest splunk_server=<shc captain> /services/configs/conf-transforms | search filename=*| table splunk_server eai:acl.app title filename id | rename eai:acl.app as app title as stanza splunk_server as shc_captain | search filename=*file1.csv* OR filename=*file2.csv* If you are validating a standalone SH not in a SH cluster you can run this search on that SH and use splunk_server=local 3.) If the lookup file is referenced in a Data Model acceleration search it needs to be sent in the bundle to the indexers, therefore do not blacklist it in distsearch.conf on the SH. This would only apply to data models that are accelerated and have the lookup in the search the DM is running.  How to determine if a datamodel search is using a lookup: a.) on the SH where the DM is accelerated temporarily set the DMA permissions to global under > settings > datamodels > (switch app context to All) > for DMs with yellow lightning bolt >  edit permissions > global we do this so that we can see all accelerated DMs at this endpoint: /services/datamodel/acceleration b.) login to web UI of a SHC member and run this search: | rest splunk_server=local /services/datamodel/acceleration | fields title eai:acl.app search | rename eai:acl.app as app this will show any datamodels that are accelerated and the underlying search that is run when the DMA search runs in the background (every 5m by default) c.) we want to check if the search used by the DM has any lookups or has any macros which have lookups. copy the entire string of the 'search' field from the output of the search from item #3b above , and paste / run it as an ad-hoc search for the last -15m. Look in job inspector search properties at the job properties:   does the property value  for normalizedSearch contain the term *lookup* ? d.) if no, you are good to go and the accelerated datamodel isn't using any lookups e.) if yes, identify the lookup and make sure you don't blacklist the .csv from the knowledge bundle 4.)To blacklist the lookups from the knowledge bundle by applying the following to the search heads:  distsearch.conf  [replicationBlacklist] no_lookup1 = apps/<app1>/lookups/file1.csv no_lookup2 = apps/<app2>/lookups/file2.csv In a SHC this would be applied to the SHC deployer in the shc apps directory and pushed to the SHC members. Ie; /opt/splunk/etc/shcluster/apps/configuration_updates/local/ run from the deployer to push shcluster bundle to SHC members: ./splunk apply shcluster-bundle -target https://anySHCmember.com:8089 -preserve-lookups true -auth admin:pwd upon the next configuration change on the SH a new knowledge bundle will be created and sent to the indexers you can run the following search on the captain to see the bundles on the captain and their size: | rest splunk_server=local /services/search/distributed/bundle-replication-files | convert ctime(timestamp) as timestamp| eval sizeMB= round(size/1024/1024,2) | table splunk_server filename location size sizeMB timestamp title| rename splunk_server as "shc_captain" title as "bundle version" | join shc_captain [| rest splunk_server=local /services/configs/conf-distsearch | search maxBundleSize=* | table splunk_server maxBundleSize | rename splunk_server as shc_captain maxBundleSize as "maxBundleSize (distsearch.conf)"] All indexers initially need a full bundle. A full bundle replication event can be seen by looking in the indexer's splunkd_access.log file for the event: 10.10.10.10 - splunk-system-user [23/Jul/2020:13:44:31.950 -0400] "POST /services/receivers/bundle/rplinux04.sv.splunk.com HTTP/1.1" 200 19 - - - 3563ms subsequent changes at the SH which require a bundle replication will be a delta replication (only sending the changes) which can be seen on the indexer as: 10.10.10.10 - splunk-system-user [23/Jul/2020:13:33:22.251 -0400] "POST /services/receivers/bundle-delta/rplinux04.sv.splunk.com HTTP/1.1" 200 2132 - - - 436ms depending on how often changes are occuring at the SHs will determine how often bundles are replicated. You can see a timechart of this activity with the following search: note: insert a host filter that will match your indexers index=_internal host=<*idx*> source=*splunkd_access.log* *bundle* method=POST /services/receivers/bundle* | rex "\/services\/receivers\/(?<type>\S+)\/(?<SH_id>\S+)\s+HTTP" | eval type_SH=SH_id."::".type | timechart count by type_SH limit=0 in the chart above we can see that there were several delta replications as well as full bundle replications from various SHs which are peered to the indexers. The GUID id is the id of the search head cluster which the replication was sent from the SHC captain and is found in the SHC members server.conf [shclustering] stanza :  [root@rplinux08 bin]# ./splunk btool server list --debug shclustering | grep id /opt/splunk/etc/system/local/server.conf id = A4DC6068-1149-459E-87CD-34A834C275AC or a table view of the bundle replication events: index=_internal host=*idx* source=*splunkd_access.log* *bundle* method=POST /services/receivers/bundle* | rex "\/services\/receivers\/(?<type>\S+)\/(?<SH_id>\S+)\s+HTTP" | table _time host method type SH_id bytes spent clientip status This process of tuning your knowledge bundle needs to be repeated for each unique SHC and each unique standalone SH and may need to be revisited every so often as the environment changes. ... View more

or you could also run a search that is easier to digest: index=_internal source=*splunkd.log* log_level=ERROR OR log_level=DEBUG component=HttpInputDataHandler reply=* | eval response_to_client=case(reply=="0","success",reply=="1","Token disabled",reply=="2","Token is required",reply=="3","Invalid authorization",reply=="4","Invalid token",reply=="5","No data",reply=="6","Invalid data format",reply=="7","Incorrect index",reply=="9","Server is busy",reply=="10","Data channel is missing",reply=="11","Invalid data channel",reply=="12","Event field is required",reply=="13","Event field cannot be blank",reply=="14","ACK is disabled",reply=="15","Error in handling indexed fields",reply=="16","Query string authorization is not enabled",reply=="17","HEC is healthy",reply=="18","HEC is unhealthy, queues are full",reply=="19","HEC is unhealthy, ack service unavailable",reply=="20","Hec is unhealthy, queues are full, ack service unavailable") | stats count by host name channel source_IP response_to_client reply | rename host as "HEC Receiver" source_IP as "HEC client" ... View more