Getting Data In

Deployment Server & Univeral Forwarder

Kate_Lawrence-G
Contributor

Ok...

I have a new installation of Splunk (v-4.2.4) up and running and am starting to use the deployment server.
I have one test client and have installed the univeral forwarder in /opt/splunkforwarder
Connectivity is up and running and the client is showing up in the deployment monitor as active.

I've created a new deployment-app (linux-base) and have successfully deployed it to the aforementioned client. It shows up in /opt/splunkforwarder/etc/apps/linux-base and consists of an inputs.conf that is fairly basic as I just want to capture information from /var/log

Here is inputs.conf

[monitor:///var/log]
disabled=false
followTail = 1
recursive = true
index = amz
blacklist = (\.gz$|\.tgz$|faillog|wtmp|lastlog|core|audit|[0-9]$|store.log$)

Now here is the problem: I never see any info from /var/log. It doesn't appear to be processed at all?

I verified this by running ./splunk btool inputs list and have confirmed that I'm only seeing the $Splunk_Home data. Which is what is defined by /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/inputs.conf.

What I'm not sure about is why it doesn't see the configuration I've defined in the app I deployed? Shouldn't it pick up the alterate inputs.conf and forward the info to the indexer as defined by the /opt/splunkforwarder/etc/system/local/outputs.conf?

Any help is greatly appreciated!

Thanks

Kate

jsb22
Path Finder

Did you configure the forwarder to send the data to the Indexer with the outputs.conf? Did you enable data recieving on the indexer? I'm no expert, but you may need to define a whitelist for what logs you do want. Maybe a "whitelist=*" since you appear to blacklist everything you don't want already? The following is an exerpt for a forwarder I use:

[monitor:///var/log]
whitelist=(\.log|log$|messages|secure|auth|mesg$|cron$|acpid$|\.out)
blacklist=(lastlog)
index=linuxserver
disabled = 0
0 Karma

dwaddle
SplunkTrust
SplunkTrust

You may want to ship a couple of additional files in your app, and double-check your serverclass.conf settings.

In my inputs-only apps, I always ship a default/app.conf with these minimum settings:

[install]
state=enabled
is_configured=true

[ui]
is_visible=false
is_manageable=false

I ship an (empty) local/app.conf as well. (Which is probably not necessary).

And, back on the deployment server, make sure these two settings are set for the app in question:

restartSplunkd=true
stateOnClient=enabled

UPDATE:

Also, make sure that when assembling your app, you follow the Splunk pattern for these apps. All *.conf files need to be in either default/ or local/ -- otherwise Splunk won't parse them into the configuration.

0 Karma

dwaddle
SplunkTrust
SplunkTrust

Cool. I updated the actual answer with that tidbit of advice. If you don't mind, please click the "accept answer" checkbox to the left so the answers system will treat this as "answered". Thanks.

0 Karma

Kate_Lawrence-G
Contributor

This was it!! somehow I totally missed that I didn't have the inputs.conf file in the correct directory...I just knew it had to be something like that. Thanks for help!

0 Karma

dwaddle
SplunkTrust
SplunkTrust

Kate - so, btool still doesn't show your input as being part of the overall splunk configuration? Is your inputs.conf in either the default or local directory of your deployed app?

RicoSuave
Builder

check splunkd.log for any errors connecting back to the indexer. or maybe post what you find in that log file on here.

0 Karma

Kate_Lawrence-G
Contributor

Thanks dwaddle! I made those changes and sent out the updated deployment and everything is updated. However I still don't see any of the /var/log data being processed just the $Splunk_Home data under /opt/splunk....

0 Karma
Get Updates on the Splunk Community!

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...