Getting Data In

Where do you configure the location of log files? Is it inputs.conf on the forwarder?

euphvx
Explorer

Hi, I am brand new to Splunk. I've read up on what I can in the past few days and need some help clarifying some things. Our old splunk admin left the company and I've been asked to help with Splunk while we are replacing her. Where do I configure log source? My unix admin tells me they installed the forwarders correctly - which is fine since I can see the syslogs from the server but they want extra application logs to mimic the setup of another server (I didn't set that up).

It sounds like the templates and everything else is there. I just don't know where to configure the log locations on this new server. I think it has to be configured in inputs.conf from the unix server's splunkforwarder/etc/system/local directory. Is this correct? They are pushing back this issue on me telling me that I configure this on Splunk server itself. Please help clarify!

Tags (2)
0 Karma
1 Solution

chanfoli
Builder

Inputs can have configurations in the location you specified on the forwarder as well as on the indexer itself for parsing, sourcetyping, transformations and other index-time functions. Also some distributed deployments make use of the forwarder-management/deployment server functionality where a central server pushes out configs in the form of apps.

Most commonly you would have a monitor stanza defined in an inputs.conf, the location of which could be as you said (splunkforwarder/etc/system/local) or it could also be under $SPLUNK_HOME/etc/apps/SOMEAPPNAME/local if the input was configured as part of an app and pushed out the the forwarder. You might be able to poke around and figure out how your environment is configured, but you will need to learn where to look on the various systems, or you will need some actual support/consulting help.

Hopefully this helps with your specific question.

View solution in original post

chanfoli
Builder

Inputs can have configurations in the location you specified on the forwarder as well as on the indexer itself for parsing, sourcetyping, transformations and other index-time functions. Also some distributed deployments make use of the forwarder-management/deployment server functionality where a central server pushes out configs in the form of apps.

Most commonly you would have a monitor stanza defined in an inputs.conf, the location of which could be as you said (splunkforwarder/etc/system/local) or it could also be under $SPLUNK_HOME/etc/apps/SOMEAPPNAME/local if the input was configured as part of an app and pushed out the the forwarder. You might be able to poke around and figure out how your environment is configured, but you will need to learn where to look on the various systems, or you will need some actual support/consulting help.

Hopefully this helps with your specific question.

euphvx
Explorer

Thanks for the clarification. I checked my $SPLUNK_HOME/etc/apps/SOMEAPPNAME/local and found the app that was applied to the server in question. In the local directory there is only 1) props.conf and 2) transforms.conf.

Based on that, would it be safe to say then that this particular deployment I will need my Unix administrator to look at his forwarder inputs.conf on the target server's splunkforwarder directory to configure the location of the logs on that same box?

We've been going back and forth so I want to be able to give them something to look at and do their due diligence. They have another *nix server that has been setup and sending logs already so I may ask for them to see how the inputs.conf file looks like on that server. Thanks very much for the help!

0 Karma

chanfoli
Builder

If you found an app on your forwarder with the monitor input in question, there is a possibility that this app was pushed out to the forwarder by a deployment server, possibly your "splunk server" serving as an indexer/search head/ deployment server.

If possible I would run this command on your forwarder:

$SPLUNK_HOME/bin/splunk cmd btool --debug deploymentclient list
0 Karma

euphvx
Explorer

I worked with our unix admin and found the inputs.conf file under the app directory. Sure enough it was full of log source paths (for a different server). The unix admins copied that file from one server to another and expected it to work. I hope this is it.

I am asking the developers to check the paths and make corrections. After a new inputs.conf is created/modified, I will have the admin replace the file and then restart splunkd.

0 Karma

euphvx
Explorer

We got it working! We found the inputs.conf file located at the forwarder/etc/apps/name_of_app/local directory. It had numerous lines and we can see where the missing log sources are and added them. Now we have the logs we need. Yay!

Chanfoli, we did run that command and the ouputs were:

/opt/splunkforwarder/etc/system/local/deploymentclient.conf [deployment-client]
/opt/splunkforwarder/etc/system/local/deploymentclient.conf [target-broker:deploymentServer]
/opt/splunkforwarder/etc/system/local/deploymentclient.conf targetUri = oursplunkname.com:ourport#

0 Karma

chanfoli
Builder

I if that oursplunkname.com:ourport# item is an actual running deployment server, keep in mind that the app could have originally been pushed from that server and if you made local changes the app, they could get overwritten by the copy of the app on the deployment server.

In any case you will want to check for this app on that server under $SPLUNK_HOME/etc/deployment-aps/, then have a look at $SPLUNK_HOME/etc/system/local/serverclass.conf to see if you can see a class which references this app, there might be a whitelist and/or blacklist which tells the server which forwarders get this app.

0 Karma

euphvx
Explorer

Yup you are right. I found the inputs.conf in the deployment-apps/nameofourapp/local directory. I made sure the updates we made are reflected there. Thanks for pointing it out!

The serverclass.conf was the first thing I did before we ran into the issue. I whitelisted the server in the appropriate section for the app.

I think we are good now! I learned so much in a span of a few hours. Thanks for your help!

0 Karma

jayannah
Builder

Yes, it's in universal forwarder inputs.conf

You can put inputs.conf file in ..etc/system/local/ or ..etc/app//local/ directory. Remember that ..etc/system/local configuration has the highest precedence.

If you are log source in say system-1 and the log file to be monitored in /log/file1, then you can install the Universal forwarder on system-1 and configure in inputs.conf to read the log file path /log/file1 either in ..etc/system/local/ or ..etc/app//local/ directory.

Please find many sample example at http://docs.splunk.com/Documentation/Splunk/latest/Admin/Inputsconf

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...