Getting Data In

Syslog data to new Splunk instance

bobmc859
New Member

I've recently inherited an old Splunk installation, and I'm in the process of migrating it over to a new updated instance. One of my issues is they have a syslog server that is collecting logs from the firewall and other devices and forwarding that to the old Splunk instance, which appears to be working as expected.

I want to forward that data to the new instance, however, I'm not seeing where they are monitoring those specific log files at, which the Syslog server is dumping that data into specific text files on a different partition. I've reviewed the server .conf file, and it shows the old Splunk instance IP, however, the inputs config file does not have any references to monitoring anything other than Windows event logs. I've also reviewed the default inputs .conf file, and again, nothing specific regarding the universal forwarder monitoring and forwarding the data.

If I change the IP in the server .conf file to the new instance, the data is forwarded, but I'm not sure how it's actually doing that.

Any suggestion on where I should be looking at or how this is being forwarded?

Thanks,

Bob

0 Karma
1 Solution

ragedsparrow
SplunkTrust
SplunkTrust

Hey @bobmc859 !

One of the most useful tools you will have is btool. To find where a file is being monitored try using this:

$SPLUNK_HOME$/bin/splunk btool inputs list --debug

It will give you the location for your inputs locations for any file monitors you are using. You can also do this for outputs.conf:

$SPLUNK_HOME$/bin/splunk btool outputs list --debug

This is where I would start.

Here are some good links as well:
https://www.splunk.com/blog/2017/05/02/inheriting-a-splunk-enterprise-deployment.html
https://docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/Usebtooltotroubleshootconfigurat...

View solution in original post

0 Karma

ragedsparrow
SplunkTrust
SplunkTrust

Hey @bobmc859 !

One of the most useful tools you will have is btool. To find where a file is being monitored try using this:

$SPLUNK_HOME$/bin/splunk btool inputs list --debug

It will give you the location for your inputs locations for any file monitors you are using. You can also do this for outputs.conf:

$SPLUNK_HOME$/bin/splunk btool outputs list --debug

This is where I would start.

Here are some good links as well:
https://www.splunk.com/blog/2017/05/02/inheriting-a-splunk-enterprise-deployment.html
https://docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/Usebtooltotroubleshootconfigurat...

0 Karma

bobmc859
New Member

Thanks @ragedsparrow ! Unfortunately when I attempt to run the btool I get the error "SPLUNK_HOME must be set. Stopping." I understand I need to set an environment variable, which I tried but still didn't doesn't seem to work. This server is W2K16 and I'm only running the Universal Forwarder so I attempted both setx SPLUNK_HOME "C:\Program Files\SpunkUniversalForwarder\bin" and setx $SPLUNK_HOME "C:\Program Files\SpunkUniversalForwarder\bin" command but nether seem to do the trick. Any advice on what I'm missing here?

Thanks,
Bob

0 Karma

ragedsparrow
SplunkTrust
SplunkTrust

Hey @bobmc859 Bobm

$SPLUNK_HOME$ needs to be your base install path. In your case it would be:

C:\Program Files\SpunkUniversalForwarder\

Try that. If setting that doesn't work, just substitute it in the command:

C:\Program Files\SpunkUniversalForwarder\bin\splunk btool list inputs --debug

C:\Program Files\SpunkUniversalForwarder\bin\splunk btool list outputs --debug
0 Karma

bobmc859
New Member

Thanks again @ragedsparrow , unfortunately that still didn't work as I expected, though I was able to get the command to run using ./splunk cmd btool inputs list --debug in powershell

However reviewing the results of the inputs or outputs config files I do not see how the firewall and other device data in Syslog is being sent to Splunk. Any other suggestions?

0 Karma

ddrillic
Ultra Champion

So, nothing shows up when running? -

 ./splunk btool inputs list  --debug | grep syslog
0 Karma

bobmc859
New Member

@ddrillic I'm running windows so I ran this instead ./splunk btool inputs list --debug | select-string syslog but received zero results from it.

0 Karma

ragedsparrow
SplunkTrust
SplunkTrust

Try this:

./splunk btool inputs list --debug | findstr syslog

That might work a little better than Select-String

0 Karma

bobmc859
New Member

Unfortunately that returned zero results as well

0 Karma

ragedsparrow
SplunkTrust
SplunkTrust

Is there any way you could share the full output of the btool (with anything sensitive obfuscated)? It's not typical for Splunk to be forwarding a file input and it not show up in the btool output.

Something like the below example should work in Powershell:
./splunk btool inputs list --debug | Out-File -FilePath C:\temp\inputs.log

Usually there are 3 locations that Splunk monitors files:

$SPLUNK_HOME$/etc/system/local/inputs.conf
$SPLUNK_HOME$/etc/apps/APPNAME/default/inputs.conf
$SPLUNK_HOME$/etc/apps/APPNAME/local/inputs.conf

Depending on how well the previous admin followed best practice, they may have also modified: $SPLUNK_HOME$/etc/system/default/inputs.conf ... However, this scenario is not typical is is highly advised against. Using the btool is a way to find out, in an aggregated fashion, where the conf file is in the Splunk system that you may be looking for.

0 Karma

bobmc859
New Member

@ragedsparrow I think I found it. Going through the output pulling out sensitive info I found the following:

C:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\local\inputs.conf [monitor://E:\logging]

Which I believe means that one of the app he installed is what's controlling that data being pulled in, right? Does that mean he installed the original UF via an app?

0 Karma

ragedsparrow
SplunkTrust
SplunkTrust

So, the SplunkUniversalForwarder app holds the configurations that normally define default inputs and settings for a Universal Forwarder. It can be utilized like any other app, though, as you can see here. It doesn't mean that the UF was installed via an app, only that the app named SplunkUniversalForwarder is where the previous person stored their custom configurations. They may also have put their outputs.conf configuration in there as well, so to add the new indexer, you would just need to modify the outputs.conf to include it.

0 Karma

ragedsparrow
SplunkTrust
SplunkTrust

You could do a search in Splunk to view the source of the logs. This should tell you where the log files are written.

| metadata type=sources index=<insert Index Here> | table source

You can also try this:

| tstats count where index=<insert index here> by index sourcetype source

Either of these should tell you how the data is coming in. If it is being written to a log file by your syslog server and then being picked up by a Splunk forwarder, you can use those btool commands to find out which configuration is being used to read the log files.

0 Karma

bobmc859
New Member

@ragedsparrow using the first option I can see source is pointing to the E partition and specific text files. Though again the input conf file does not show anything in it to specifically monitor for those text files or even the partition in general.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...