Here is an example that shows how splunk monitors a file.
Suppose we monitor this file.
[monitor:///home/data/mylog.txt]
After all the file content is indexed, you can issue this command to check the monitoring status.
./splunk _internal call /services/admin/inputstatus/TailingProcessor:FileStatus -auth admin:changeme
You will got a result like this:
<s:key name="/home/data/mylog.txt">
<s:dict>
<s:key name="file position">1542</s:key>
<s:key name="file size">1542</s:key>
<s:key name="percent">100.00</s:key>
<s:key name="type">finished reading</s:key>
</s:dict>
</s:key>
which means 100% of the file is read and the pointer is at 1542 bytes.
So, where does Splunk store this information? It is in the fishbucket.
If you issue this command to check against the fishbucket:
./splunk cmd btprobe -d /opt/splunk613-b/var/lib/splunk/fishbucket/splunk_private_db --file /home/data/mylog.txt
You get this result:
key=0x4cb806285f84f0d9 scrc=0xdeb7db35f8958745 sptr=1542 fcrc=0x61910b3b6da87f06 flen=0 mdtm=1413872354 wrtm=1413872518
You will notice that the fishbucket doesn't have any filename information. Fishbucket doesn't store filename information. It only stores the crc of that file. The key to the monitoring file /home/mylog.txt is key=0x4cb806285f84f0d9
So, how is /home/data/mylog.txt and key=0x4cb806285f84f0d9 related?
Here is the command.
# ./splunk cmd btprobe --compute-crc /home/data/mylog.txt
Using logging configuration at /opt/splunk613-b/etc/log-cmdline.cfg.
crc=0x4cb806285f84f0d9 decimal=5528175313068290265
I tried this and it returned 'record not found'.
./splunk cmd btprobe -d /opt/splunk/var/lib/splunk/fishbucket/splunk_private_db --file /var/log/message
Using logging configuration at /opt/splunk/etc/log-cmdline.cfg.
record not found
And finally, I noticed that I added crcSalt = <SOURCE>
to the monitoring stanza and the command should be like this.
./splunk cmd btprobe -d /opt/splunk/var/lib/splunk/fishbucket/splunk_private_db --file /var/log/message --salt /var/log/message
Here is an example that shows how splunk monitors a file.
Suppose we monitor this file.
[monitor:///home/data/mylog.txt]
After all the file content is indexed, you can issue this command to check the monitoring status.
./splunk _internal call /services/admin/inputstatus/TailingProcessor:FileStatus -auth admin:changeme
You will got a result like this:
<s:key name="/home/data/mylog.txt">
<s:dict>
<s:key name="file position">1542</s:key>
<s:key name="file size">1542</s:key>
<s:key name="percent">100.00</s:key>
<s:key name="type">finished reading</s:key>
</s:dict>
</s:key>
which means 100% of the file is read and the pointer is at 1542 bytes.
So, where does Splunk store this information? It is in the fishbucket.
If you issue this command to check against the fishbucket:
./splunk cmd btprobe -d /opt/splunk613-b/var/lib/splunk/fishbucket/splunk_private_db --file /home/data/mylog.txt
You get this result:
key=0x4cb806285f84f0d9 scrc=0xdeb7db35f8958745 sptr=1542 fcrc=0x61910b3b6da87f06 flen=0 mdtm=1413872354 wrtm=1413872518
You will notice that the fishbucket doesn't have any filename information. Fishbucket doesn't store filename information. It only stores the crc of that file. The key to the monitoring file /home/mylog.txt is key=0x4cb806285f84f0d9
So, how is /home/data/mylog.txt and key=0x4cb806285f84f0d9 related?
Here is the command.
# ./splunk cmd btprobe --compute-crc /home/data/mylog.txt
Using logging configuration at /opt/splunk613-b/etc/log-cmdline.cfg.
crc=0x4cb806285f84f0d9 decimal=5528175313068290265