How to config inputs.conf to use host name of server its deployed on?

[joesrepsol]

HostName:
XXXXXXXX*p528*

File Path:
/dsto/sw/prod/webapps/jbossEAP6.1/servers/appname1/log/p520/server.log <-- not this one
/dsto/sw/prod/webapps/jbossEAP6.3/servers/appname1/log/p528/server.log <-- Ingest this log!
/dsto/sw/prod/webapps/jbossEAP6.3/servers/appname2/log/p528/server.log <-- Ingest this log!
/dsto/sw/prod/webapps/jbossEAP7.0/servers/appname1/log/p540/server.log <-- not this one

Looking to build a "dynamic" monitor app to collect some JBoss logs from our prod servers. The path that I want to use for the file to ingest matches the last 4 digits of the hostname (see example). There will be many mounts on a single server but I only want the server.log files who's path includes the matching last 4 digits of the hostname.

Can I do a lookup/query for the hostname, then use that parameter in the monitor line?

So what will my monitor line in the inputs.conf have to look like to make this happen?

Example inputs.conf pushed out to host XXXXXXXXp528

[monitor:///dsto/sw/prod/webapps/jbossEAP*/servers/appname*/log/?dynamic lookup to get p528?/server.log]
disabled = false
sourcetype = jboss:server:log
index = jboss

Thanks in advance for help on this subject!!!

Splunk Enterprise v6.5.1

Joe

[FrankVl]

You can use environment variables in your inputs.conf stanzas.

I got the following working:

defined environment variable in /opt/splunk/etc/splunk-launch.conf:

HOST_FILTER=splunkvm1

inputs.conf:

[monitor:///opt/test/$HOST_FILTER/*.log]
index=test
sourcetype=syslog

Result as seen in splunkd.log:

TailingProcessor - Adding watch on path: /opt/test/splunkvm1

So all you need to figure out is a convenient way for you to set that environment variable for each of your servers.

[joesrepsol]

ahhhhh.. this looks promising! Digging in to setup this variable and try this out. Fun! Thank you everyone... good discussion.

[joesrepsol]

I already have a server class setup just for these jboss servers... so adding a custom $HOST_FILTER$ like variable with the last 4 digits of the server name should be fairly easy.

[FrankVl]

Only thing is that you can't configure splunk-launch.conf through an app. It really needs to be in /etc/splunk-launch.conf.

So you'll need to find a convenient way to add a server specific line to that file on each separate server. Or find another way to set the relevant environment variable, I believe that can also be done from the command line somehow.

Anyway: that would be a one time thing on each server and no ugly scripts or symlinks or so, so hope it helps 🙂

[FrankVl]

I'm not aware of any way to parameterize your input stanzas like that, to only ingest files from folders that match the server's host name.

Apart from manually creating all the inputs, I guess your best bet is to create some kind of script that generates the relevant input configs (and matching serverclass.conf entries if you use a DS to deploy).

An alternative take could be to create a small script that finds the relevant folders on the server and creates symlinks for them in a generic place where you then point the Splunk input.

[joesrepsol]

I'm at that same point. Was hoping not to have to make 200+ unique apps to monitor this stuff, and it doesn't help if (or "when") they add more jboss servers... or more apps on the existing servers. Hence my desire for this dynamic natured monitor inputs.. Hmmm??? Still thinking of how else to do this.

[FrankVl]

Well, perhaps you can deploy a Splunk scripted input that periodically updates the set of folders that gets symlinked into the path that Splunk monitors. That way you can deploy and manage both the symlinking script and the splunk inputs through central Splunk tools, and deal with dynamic situation on the server as well as dynamic set of servers.

Alternative could of course be to set up a central server that mounts all the log folders from all the servers and then put a forwarder there. But then you still have the burden of maintaining the mounts on that server and it may be too much to handle for 1 forwarder. Plus it doesn't exactly improve the data distribution (assuming you have multiple indexers).

[joesrepsol]

Thanks for the ideas... I guess Im still surprised that I cannot lookup the name of the host that the forwarder is running on (i.e. this forwarder) and use that in the inputs.conf. Still working through ideas that can make that work.

I wish the jboss admin's had not setup the servers this way, but this is the way it is, so I'm trying to work with what I got.

[FrankVl]

Now that I'm thinking about it: I have used the $SPLUNK_HOME environment variable in input.conf monitor stanzas. Which kind of suggests that there may actually be an opportunity to fix this with a parametrized stanza somehow (unless that $SPLUNK_HOME is the only thing you can refer to like that)....

[FrankVl]

Got it working, see my new answer added below 🙂

Feeling silly that I didn't think of that use of $SPLUNK_HOME before and realize that should offer some possibilities.

[livehybrid]

Hi there,
You can use wildcards in your monitor stanza as adonio suggested, you'll end up with something like this:

[monitor:///dsto/sw/prod/webapps/jbossEAP*/servers/appname*/log/*/server.log]
disabled = false
sourcetype = jboss:server:log
index = jboss

Regards
Will

[livehybrid]

Ah I may have misunderstood your question...
Is there a reason you cannot use the wildcard approach? If it is so you can specify a specific index or sourcetype then this can always be overwritten with a transform based on the hostname..

[joesrepsol]

The reason the wildcard technique doesn't solve the issue is that there are 20+ mounts on each jboss server, some that link to the other servers log files (suppsoed to be easy, so if you log into 1 box, you can get to the logs on the other 20+). Unfortunately that complicates what I need to do with splunk.

I wanted to pull the logs for this_server_only, so I need to find the matching server name in the path. So the records that get ingested into Splunk show the correct host (i.e. logs from p523 show the host XXXXXXXp523).

[livehybrid]

Re-reading it again, I see what you mean now - as you will have many mounts..

[adonio]

hello there,
please take a look at this link:
https://docs.splunk.com/Documentation/Splunk/7.0.2/Data/Specifyinputpathswithwildcards
hope it solves all the challenges.

[joesrepsol]

Thanks. I did look at this page for wildcards, etc... but was not clear to me if I can query/use the host name in my inputs.conf as a filter to what I want to monitor. Hence posting my question to the forum.