I made some changes to some properties files on my deployment server:
etc/system/local/serverclass.conf - added a new client
deployed. Seem to still work
then I added a blacklist
etc/apps/deployment-apps/splunk_ta_windows - added a blacklist
deployed, it and things seemd to stop working. Everything, couldn't search for current day
I had a space issue on my forwarder, which I resolved.
I see this error on my indexer now:
Truncating line because limit of 10000 bytes has been exceeded with a line length >= 11576 - data_source="/trvapps/splunk/var/log/splunk/remote_searches.log", data_host="tospkiu1", data_sourcetype="splunkd_remote_searches"
@pfabrizi the default setting for [splunkd_remote_searches] is 10000. You can increase this number via props.conf like the below stanza. In this example you would set the Truncate to 99999.
[splunkd_remote_searches]
TRUNCATE = 99999
The issue was my fault. I had made a backup copy of the original serverclass.conf, when I copied it back to restore it had root:root and splunk couldn't open the file. I fixed that and restarted the deployment server.
I get this error on my searchhead when starting it:
could not create path /oaisys_z843_splunk_file1/firedalerts/db appearing in indexes.conf: 13
I don't have the volume on my search head, just indexers ( san storage).