Getting Data In

rsyslog server with UF not sending events to Splunk

Branden
Builder

Hi. At Splunk's recommendation, I have a centralized syslog server (using rsyslog) that writes to /logs/hostname/year/month/day/file.log
This works fine.

However, I cannot get the Universal Forwarder to send the events to the Splunk Indexer. I added my stanza to /opt/splunkforwarder/etc/system/local/inputs.conf. When that didn't work, I created an app and put the stanza into /opt/splunkforwarder/etc/apps/syslog/local/inputs.conf
Didn't work.
Here is my stanza:

[monitor:///logs/*]
disabled = false
host_segment = 2
index = main
sourcetype = syslog

That looks straightforward to me.
I checked the Splunk logs on the Indexer and there's no sign that it's ever receiving these events.
In the UF logs I see that it has added a watch to /logs:

INFO  TailingProcessor - Parsing configuration stanza: monitor:///logs/*.
INFO  TailingProcessor - Adding watch on path: /logs.

I have verified that the port is open between the UF and the Indexer.

Indexer is running 7.2.4 and UF is running 7.1.2.
Am I missing something?

Thank you in advance!

0 Karma
1 Solution

solarboyz1
Builder

Have you verified the user account running Splunk can access the syslog files?

By default, I believe only root can read them. You may need to change the permission, and update your rsyslog configuration to write the files so that Splunk can read them.

View solution in original post

solarboyz1
Builder

I think I see the issue:
https://docs.splunk.com/Documentation/Splunk/latest/Admin/Inputsconf

Note concerning wildcards and monitor:
You can use wildcards to specify your input path for monitored inputs. Use
"..." for recursive directory matching and "" for wildcard matching in a
single directory segment.

Try the following for you input:

 [monitor:///logs/.../*]

ddrillic
Ultra Champion

Absolutely @solarboyz1 and @Branden can see it via the message you posted earlier -

INFO  TailingProcessor - Adding watch on path: /logs.
0 Karma

solarboyz1
Builder

Posted it as a separate answer for points and other users who might find it useful...but might not drill into the notes of the first answer.

0 Karma

solarboyz1
Builder

Have you verified the user account running Splunk can access the syslog files?

By default, I believe only root can read them. You may need to change the permission, and update your rsyslog configuration to write the files so that Splunk can read them.

Branden
Builder

Thank you for your response. I thought of that and, yes, I have verified that the splunk user can access the files. The dirs are 755 and the files are 644. I also changed to the splunk user and manually verified.

0 Karma

solarboyz1
Builder

I think I see the issue:
https://docs.splunk.com/Documentation/Splunk/latest/Admin/Inputsconf

Note concerning wildcards and monitor:
* You can use wildcards to specify your input path for monitored inputs. Use
"..." for recursive directory matching and "*" for wildcard matching in a
single directory segment.

Try the following for you input:
[monitor:///logs/.../*]

Branden
Builder

You are amazing, my friend. That solved this issue. Thanks!

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...