What could be causing Splunk to log my data under the wrong hostname?

[lacrosse1991]

Hello,

I recently added my meraki appliance as a datasource on my Splunk instance (reading from a file that syslog-ng is logging to). Everything was working fine for a few days, but I noticed last night that the hostname for my data changed from my statically set hostname to the IP address of my meraki device instead.

How can I can I revert the hostname back to my statically set one? Please see the screenshots below for my configuration:

[Richfez · Accepted solution]

The way to find this is to use btool. You can run it by $SPLUNK_HOME/bin/splunk btool.

NOTES:
1) btool in debug mode (which we'll need) can be a bit verbose. Make your screen big, and we'll pipe it to a pager. In all cases of running btool I will pipe to more because that lives on both windows and *nix, but feel free to use | less on *nix and/or | clip (and then paste into notepad) on Windows. Maybe even better is to redirect the output to a file, then edit that file in any way you are comfortable with. For that, instead of | more you would > filename.txt each time, then edit whatever filename you picked.

2) Also, I'll provide both the *nix then the Windows commands, just use the right one, K?

3) On *nix, change to your splunk bin directory, usually cd /opt/splunk/bin if you installed it in the default location. For Windows, cd Program files\splunk\bin.

So, first is to see the input.

./splunk btool inputs list --debug | more
splunk btool inputs list --debug | more

That will pipe it the pager: scroll down until you find the section for Meraki (and/or 10.0.3.1?) and take a look. For instance, here's a completely unrelated input from my own system:

/opt/splunk/etc/apps/fw/local/inputs.conf                              [monitor:///var/log/remote/192.168.0.1/*]
/opt/splunk/etc/system/default/inputs.conf                            _rcvbuf = 1572864
/opt/splunk/etc/apps/fw/local/inputs.conf                              disabled = false
/opt/splunk/etc/system/local/inputs.conf                                host = maxwell
/opt/splunk/etc/apps/fw/local/inputs.conf                              index = fw
/opt/splunk/etc/apps/fw/local/inputs.conf                              sourcetype = fw

Much can be ignored, the parts we want are "host", "sourcetype" and "index". Well, you are stuffing it into main, so index doesn't count for much. 🙂

What this says is that the "host" as this input sees it is being supplied by /opt/splunk/etc/system/local/inputs.conf. The sourcetype and index are both being set in /opt/splunk/etc/apps/fw/local/inputs.conf. I'll bet if you looked at your inputs like this, you'll see what file is making this be wrong.

I think this is what's wrong. Note I haven't told you how to fix it - it might be pretty apparent how to fix it when you see it, or maybe not. Either way, just post back with what you found specifically and we can help with that.

---

There's a slim chance it's not the problem, though and you'll have to check props and transforms.

./splunk btool props list --debug | more
splunk btool props list --debug | more

This is where things get awesomely useful, but also a bit more complex. I did the above then scrolled around until I found my [fw] settings. (Because from inputs.conf we learned the sourcetype was set to "fw", right?) Here's a snippet of them:

/opt/splunk/etc/apps/fw/local/props.conf                          [fw]
/opt/splunk/etc/system/default/props.conf                         ANNOTATE_PUNCT = True
/opt/splunk/etc/system/default/props.conf                         AUTO_KV_JSON = true
/opt/splunk/etc/system/default/props.conf                         BREAK_ONLY_BEFORE =
...
/opt/splunk/etc/system/default/props.conf                         SHOULD_LINEMERGE = True
/opt/splunk/etc/system/default/props.conf                         TRANSFORMS =
/opt/splunk/etc/apps/fw/local/props.conf                           TRANSFORMS-rem-tr-111 = remove-tr-111-lines
/opt/splunk/etc/system/default/props.conf                         TRUNCATE = 10000
...

What we're looking for here is the "TRANSFORMS" stuff. I am not quite positive (foggy-brained at the moment) but I'm pretty sure it'll be a transform that resets this some other way, if this is what's happening. So, we see I have two Transforms. A blank one from default/props and a fw/local/props.conf that has a TRANSFORM labeled "remove-tr-111-lines". Now, that sounds innocuous (with respect to changing a host), but let's take a look.

./splunk btool transforms list --debug | more

Output for that section:

/opt/splunk/etc/apps/fw/local/transforms.conf                          [remove-tr-111-lines]
/opt/splunk/etc/system/default/transforms.conf                         CAN_OPTIMIZE = True
/opt/splunk/etc/system/default/transforms.conf                         CLEAN_KEYS = True
/opt/splunk/etc/system/default/transforms.conf                         DEFAULT_VALUE =
/opt/splunk/etc/apps/fw/local/transforms.conf                            DEST_KEY = queue
/opt/splunk/etc/apps/fw/local/transforms.conf                            FORMAT = nullQueue
/opt/splunk/etc/system/default/transforms.conf                         KEEP_EMPTY_VALS = False
/opt/splunk/etc/system/default/transforms.conf                         LOOKAHEAD = 4096
/opt/splunk/etc/system/default/transforms.conf                         MATCH_LIMIT = 100000
/opt/splunk/etc/system/default/transforms.conf                         MV_ADD = False
/opt/splunk/etc/apps/fw/local/transforms.conf                          REGEX = dnsmasq\[\d+\]: TR-111
/opt/splunk/etc/system/default/transforms.conf                         SOURCE_KEY = _raw
/opt/splunk/etc/system/default/transforms.conf                         WRITE_META = False

Yeah, there's nothing there rewriting host. It would probably be obvious if there were. It'll say something like

DEST_KEY = MetaData:Host
FORMAT = host::HostA

Or one of those. Or both (going from memory here). In any case, if something like that exists, it'll ALSO tell you which file is controlling this.

Anyway, I think that's enough to get started with. I hope it's in the inputs, that'll be easier. 🙂

-Rich

[lacrosse1991]

thanks! the first recommendation did the trick. I had set a source type when I initially added a datasource, but it looks like it got removed somehow (not sure how I did that, but that's what labbing is for), so I just ended up setting it manually again which did the trick.

[gschreiner_splu]

There's a possibility the auto-generated sourcetype is doing that. I notice there are three "sourcetypes" in the picture you have, what are those two source types?

It seems to me, from the info you provided, it seems like the data was fine when it was coming in under one sourcetype, then maybe switched to another sourcetype, which the host is set by a default sourcetype transform.

[DalJeanis]

1) I'd probably send it to a separate index also anyway, since it doesn't cost anything. 2) There has to be a transform running afterward, sending it elsewhere 3) If the IP address of the Meraki is static, then you could always specify another transform to change the host from the IP back to "Meraki"... but you shouldn't have to.

[lacrosse1991]

I was able to get things working (see answer), but what is the advantage of making additional indexes? I'll have to look into setting that up