Cannot getting data into indexes

[gjhaaland]

Hi,

For some reason we cannot receive data to _interal or other indexes(all of them). Old indexes are still available through database. It looks like a generic problem, not related to any specific index. All I can see is _audit.

Maybe it's ok to backup $SPLUNK_HOME/etc, and then reinstall splunk sw? or if possible restart some processes, or modify config file. input, output.conf

Rgds

Geir

[gjhaaland]

No problems with permissions, diskusage ++. I think it's a global problems. I know that for some days ago I tried to setup pkcs12 certificate (estreamer)  on splunk server.   But can't remember where I did these settings. 

Out form commands: 

$ source /home/splunk/bin/setSplunkEnv && df -H $SPLUNK_HOME $splunk_db
Tab-completion of "splunk <verb> <object>" is available.
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos-home 886G 587G 300G 67% /home
$

$sudo /home/splunk/bin/splunk btool indexes list volume |egrep '(\[|path)'
[volume:_splunk_summaries]
path = $SPLUNK_DB

$df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/centos-root 52403200 11477568 40925632 22% /
devtmpfs 16312676 0 16312676 0% /dev
tmpfs 16329816 0 16329816 0% /dev/shm
tmpfs 16329816 10560 16319256 1% /run
tmpfs 16329816 0 16329816 0% /sys/fs/cgroup
/dev/sda3 1038336 173348 864988 17% /boot
/dev/mapper/centos-home 865131800 558906488 306225312 65% /home
tmpfs 3265964 12 3265952 1% /run/user/42
tmpfs 3265964 0 3265964 0% /run/user/1001
$

[gjhaaland]

Thanks.  No problems with persmissions.  It could be something wrong with with some confiles.  But since the proplems  involves all indexfiles it must be something global settings, or some services/program not running. 

Do you thinks it's best to backup $SPLUNK/etc, run installation/upgrade and next copy etc files into new installation. 

Geir

[isoutamo]

Hi

If you cannot get any new data then mos obvious reason is that you have that disk space full. Second one is that for some reason your permissions / ownerships have changed on disk.

Please try "source /opt/splunk/bin/setSplunkEnv && df -H $SPLUNK_HOME $SPLUNK_DB" as a root on cmd line. Also check if you have volumes in use and check that disk space also.

To find volumes you should login as splunk user and then use

splunk btool indexes list volume|egrep '(\[|path)'

Which show those physical disk areas what those are using.

If there are enough space left then you should check ownership of those directories / files and change those if needed.

Did I understand right that you get some new data into _audit index, but not anywhere else?

r. Ismo 

[richgalloway]

What error messages do you see?  Are the indexes or the disk they're  on full?

Restarting or re-installing Splunk may help correct some causes of the problem, but not the most likely ones.

[gjhaaland]

Thanks for the answer.  Everyting seems to be ok. 

disk not full, licenses ok, rebooted several times, restarted splunk several times. But still we don't receive  data into indexes.  To save time, I wondered if it's possible to backup some files $SPLUNK_HOME/etc, and then reinstall splunk sw +  copy files into new installation. 

Do you think it will work?

Rgds

Geir 

[SinghK]

Did you try btool to check your configs, indexes.conf , inputs etc. may be there is a overlapping setting routing data somewhere else.

[SinghK]

or you are getting any permissions issue on splunk.