Why running "splunk enable boot-start" did not start the indexing of my log data?

[tony_luu]

Splunk was installed and run as root.
I did a "splunk enable boot-start" which created a /etc/init.d/splunk script.
Upon system reboot, a "ps" shows that splunkd is running.
However, my logs are not indexed (per Settings/Indexes page).
After i did a manual "./splunk restart" then it started to index data.
Q: what am I missing?
Thanks,

[lguinn2]

I would take a look at the internal logs. There are two ways to do that:

1) Run this search

index=_internal sourcetype=splunkd

There will be tons of events, you may want to filter further

2) or take a look at $SPLUNK_HOME/var/log/splunk/splunkd.log

I am not sure what you will find, but I expect that the reason will be in there somewhere. Please update with what you discover.

[tony_luu]

index=_internal sourcetype=splunkd log_level=ERROR

... ERROR TailingProcessor - Input stanza path, '$MY_DATA_PATH/' is not absolute ...

So, splunk did not recognize $MY_DATA_PATH which is defined in profile.d

I move the definition to $SPLUNK_HOME/etc/splunk-launch.conf and splunk can see it. Problem solved for me.
Still not sure if that is the preferred solution though, i.e. put it in splunk-launch.conf

[rsennett_splunk]

it sounds like you have more than one instance of Splunk installed and that the init.d is pointing at the wrong one... "start" is "start" so... it seems like when you thought you'd started the right instance, you probably hadn't.

[tony_luu]

Thanks rsennett for a quick response.
I'm pretty sure there's only one splunk instance installed, and the init.d/splunk script did spell out the path "/opt/splunk/bin/splunk" correctly.
Is there anything in the logs that i can check?
splunkweb works file but splunkd seems to have problems
The TCP data is indexed but flat file logs are not.

Could this be the issue?
my inputs.conf uses env variable $MY_DATA_PATH, e.g.
[monitor://$MY_DATA_PATH/*]

and $MY_DATA_PATH is defined in /etc/profile.d/MySplunkApps.sh
export MY_DATA_PATH=/my/data/path

Maybe when splunk started, that env variable was not yet defined.

Thanks for helping.

[lguinn2]

If you want that environment variable, put it in the user profile for whatever linux user is running splunk. Even better, make it an absolute path or use a path that is relative to the APP that contains the inputs.conf file. See my answer below - you should see the error in the splunkd.log if this is the problem.