In the past, we must have specified some paths in the inputs.conf file that didn't have sourcetypes. We have since removed those paths but the forwarder is still trying to monitor those paths. I see that the apps/learned/metadata/local.meta has those paths in it. Can I just clear out that entire file, or should I pick out only the lines with the relevant path?
Additionally, does this mean that any time we remove a monitored path we need to verify that it's not in the metadata file?
Nah, you gotta start at the source.
First, find the forwarders that are defined without sourcetypes. Update those stanzas to have a sourcetype defined (and make sure they are using the deployment server to make your life easier!). That'll be in the inputs.conf file.
Then you could do a search to see what 'learned' items still are coming in. Once everything is assigned a sourcetype, you shouldn't have to worry about much in terms of cleanup. There might be some metadata stuff but I wouldn't get bothered by that as it doesn't have any real functional impact.
Nah, you gotta start at the source.
First, find the forwarders that are defined without sourcetypes. Update those stanzas to have a sourcetype defined (and make sure they are using the deployment server to make your life easier!). That'll be in the inputs.conf file.
Then you could do a search to see what 'learned' items still are coming in. Once everything is assigned a sourcetype, you shouldn't have to worry about much in terms of cleanup. There might be some metadata stuff but I wouldn't get bothered by that as it doesn't have any real functional impact.
But we no longer want these paths to be monitored. Therefore, we removed them from the inputs.conf, but they are still getting monitored by Splunk.
"But we no longer want these paths to be monitored. " - Good! That's how I interpreted things
"but they are still getting monitored by Splunk." - sounds like we need to do a little splunk ninja work to find out what monitor stanza still has them. This is going to be a great learning because we're probably going to end up using btool (my favorite command).
Let's start with this: Show me what symptoms you have observed that lead you to the conclusion that such items are still monitored by Splunk?
Regardless...I'm deleting it. Looks like it was created back when we were first implementing Splunk.
Sounds like, as you surmised, someone did a command line add of that one particular item before you kicked off formal use of the Deployment Server.
If you think this all worked out, be sure to "accept" the answer (should be a link above at the first answer) so others can use our collaboration if they get into the same snafu.
ok Burch...in trying to prove myself right, I found the issue - and yes, I used the btool command like you predicted. There is a monitor stanza in the etc/apps/search/local/inputs.conf file that has this path (/apps/tomcatlogs) specified. It's the only thing in that file. I have no idea why that is there, or how it got there. Would that "search" app be deployed from our deployment server, or would that have been created manually on the host server? I see no "search" app that is in our "deployment-apps" folder on the deploy server.