Getting Data In

How to troubleshoot why an indexer stopped receiving data on one index after I fixed a different index?

antessima
Explorer

We are working on configuring Splunk for the first time in advance of buying it, and I am having problems with the indexer. Or maybe the forwarder....not sure which is guilty here.

Here's what I'm doing: I have three indexes: one that monitors a specific file, one that monitors a different specific file, and one that monitors a group of files with similar names. Initially, index #1 was all I had set up, and it was working fine. I added the second and third indexes, and after setting them up in the inputs.conf and props.conf files on the forwarder, the second index started working just fine, but the first index stopped working. (The third one hasn't yet worked but that's not the question here.)

So, the question is, why did the first index stop working? These are the entries in the two files for the three indices and their associated sourcetypes:

inputs.conf

[monitor:///usr/local/ourstuff/logs]    <== This is index #3
disabled = false
index = transactions
sourcetype = translog
crcSalt = <SOURCE>
whitelist = *_transLog

[monitor:///usr/local/ourstuff/logs]     <== This is index #1
disabled = false
index = server_all
sourcetype = server_dblog
crcSalt = <SOURCE>
whitelist = yellowboxSync

[monitor:///usr/local/ourstuff/logs]    <== This is index #2
disabled = false
index = performance
sourcetype = httplog
crcSalt = <SOURCE>
whitelist = HttpLog

props.conf

[default]
maxDist = 500

[translog]
PREFIX_SOURCETYPE = True
SHOULD_LINEMERGE = False
is_valid = True
maxDist = 9999
MAX_TIMESTAMP_LOOKAHEAD = 28
BREAK_ONLY_BEFORE = "xmlInquery"

[server_dblog]
PREFIX_SOURCETYPE = True
SHOULD_LINEMERGE = False
is_valid = True
maxDist = 9999
MAX_TIMESTAMP_LOOKAHEAD = 28

[httplog]
PREFIX_SOURCETYPE = True
SHOULD_LINEMERGE = False
is_valid = True
maxDist = 9999
MAX_TIMESTAMP_LOOKAHEAD = 28

Any ideas would be appreciated.

0 Karma
1 Solution

somesoni2
Revered Legend

First of all, these are data monitors, not Index (index is where data is stored like transactions in this case).
Second, you say you're monitoring a single file but if you see the monitoring stanza, you're monitoring the same directory, thus only one will work.

This is how I'll write inputs.conf. Please update the full file names as per your situation.

 [monitor:///usr/local/ourstuff/logs/*_transLog]    <== This is monitoring #3
 disabled = false
 index = transactions
 sourcetype = translog
 crcSalt = <SOURCE>

 [monitor:///usr/local/ourstuff/logs/yellowboxSync]     <== This is monitoring #1
 disabled = false
 index = server_all
 sourcetype = server_dblog
 crcSalt = <SOURCE>

 [monitor:///usr/local/ourstuff/logs/HttpLog]    <== This is monitoring #2
 disabled = false
 index = performance
 sourcetype = httplog
 crcSalt = <SOURCE>

View solution in original post

somesoni2
Revered Legend

First of all, these are data monitors, not Index (index is where data is stored like transactions in this case).
Second, you say you're monitoring a single file but if you see the monitoring stanza, you're monitoring the same directory, thus only one will work.

This is how I'll write inputs.conf. Please update the full file names as per your situation.

 [monitor:///usr/local/ourstuff/logs/*_transLog]    <== This is monitoring #3
 disabled = false
 index = transactions
 sourcetype = translog
 crcSalt = <SOURCE>

 [monitor:///usr/local/ourstuff/logs/yellowboxSync]     <== This is monitoring #1
 disabled = false
 index = server_all
 sourcetype = server_dblog
 crcSalt = <SOURCE>

 [monitor:///usr/local/ourstuff/logs/HttpLog]    <== This is monitoring #2
 disabled = false
 index = performance
 sourcetype = httplog
 crcSalt = <SOURCE>

antessima
Explorer

I thought I had tried that previously, but I did as you indicated and now both #1 and #2 work simultaneously. Thank you for your help!

0 Karma
Get Updates on the Splunk Community!

More Ways To Control Your Costs With Archived Metrics | Register for Tech Talk

Tuesday, May 14, 2024  |  11AM PT / 2PM ET Register to Attend Join us for this Tech Talk and learn how to ...

.conf24 | Personalize your .conf experience with Learning Paths!

Personalize your .conf24 Experience Learning paths allow you to level up your skill sets and dive deeper ...

Threat Hunting Unlocked: How to Uplevel Your Threat Hunting With the PEAK Framework ...

WATCH NOWAs AI starts tackling low level alerts, it's more critical than ever to uplevel your threat hunting ...