You are low in disk space on partition "/opt/splunk/var/lib/splunk/audit/db". Indexing has paused. Will resume when free disk space rises above

[rameshlpatel]

I am getting this message on my indexer and search head.

First i set 5000Mb after getting this error i set this to 2000mb and some days same message seeing .

I want to know what is mean of this ? Where we are using this space in splunk ?

[lukejadamec]

Check the size of your indexes. If you are using the defaults then check the size of splunk\var\lib\splunk

The message you're getting is saying that the file system that holds splunk\var\lib\splunk is low on disk space.

As gfuente said, when the file system gets low on disk space then all indexing will stop until you correct the problem.

If the problem comes and goes by itself, then that means Splunk is deleting data based on the index retention policies, which temporarily frees up space on the file system.

[lukejadamec]

You should not post your comments as answers - it gets confusing.

[lukejadamec]

If you do not need the _audit index data, then you can delete the db folders found in splunk\var\lib\splunk\\audit\db
The modified date of the db folders is a rough approximation of the age of the data.

You should do a search on the _audit index to see what messages are causing the index to fill up so fast.

index=_audit earliest=-2d | stats count by action

[rameshlpatel]

Ok I am seeing that audit index reached to their maximum limit.

Can we clear this index logs ? Is that any harmful ?

[rameshlpatel]

First thing thanks for quick response.

But I am seeing that there is enough space avialable in FS upto 100gb.

And all indexed file goes on there specific folder structure. Then why it will affect on this folder ?

My questions is still uncleared : Why we are using this folder structure ? what actually we are storing ?

[gfuente]

Hello

This means that you are running out of free space on that FS. Splunk will stop all indexing until that problem is sorted. You should check your indexes size, and the free space available. One solution will be increase the FS size, or change the retention policies to delete old data.

Regards