Getting Data In

Why is one device out of others are receiving logs by syslog-ng but not by splunk cloud?

cbwillh
Path Finder

We have an Ubuntu syslog-ng server with the splunk forwarder.

It is configured with 18 device types (over 100 devices in those device type classes) sending logs to the syslog-ng server and then those logs are forwarded to splunk cloud by the universal forwarder.

Everything has been working perfectly for months and all data from all logs have been confirmed as arriving in splunk cloud through daily searches for ALL syslog indexes and sourcetypes configured.

As of yesterday, 1 of 18 devices is returning "0" results found when running a search for its index and sourcetype even though the syslog-ng server contains current up to date logs for the device in the location specified in the syslog-ng.conf configuration file.

So the issue is that the logs are making it to the syslog-ng server and being processed into the folder specified in the syslog-ng.conf file, but they are not making it from that location to the splunk cloud servers.

Nothing has changed on the syslog-ng server and I have already restarted BOTH the syslog-ng and the splunk services and even tried a reboot of the server with no change to the issue.

Anyone else ever seen this?

0 Karma
1 Solution

cbwillh
Path Finder

thanks for the help nittala_surya, I appreciate the time you took to respond.
I believe I nutted this out on my own though and will post what I found for anyone else who might run across this.

I compared the last entry for the sourcetype in splunk cloud with that sourcetypes physical log on the syslog-ng server

I noted that the hour was different but the minutes and seconds matched perfectly. this led me to suspect the timezone was off somewhere.

So I checked the splunk cloud -> Settings -> Source Types settings for the sourcetype I was having the issues with...

and I found that the timezone set for the Source Type that was configured in Splunk Clouds Settings had the incorrect Time Zone set. I corrected this and I also added the following to my syslog-ng.conf files line for the udp connection settings:
time_zone(PST8PDT) keep_timestamp(no)

I then saved my changes and restarted the syslog-ng service and did a service syslog-ng status to confirm it came good and was started.
I monitored over the next 15 minutes and noted that my logs were starting to show with the correct times in splunk cloud again.

I am guessing that somehow this misconfigured Source Type was there since it was created and perhaps a dip in traffic to that log yesterday helped it to become obvious for the first time.
at any rate she seems to be working again now.

I noted that the timezone was incorrect.

View solution in original post

0 Karma

cbwillh
Path Finder

thanks for the help nittala_surya, I appreciate the time you took to respond.
I believe I nutted this out on my own though and will post what I found for anyone else who might run across this.

I compared the last entry for the sourcetype in splunk cloud with that sourcetypes physical log on the syslog-ng server

I noted that the hour was different but the minutes and seconds matched perfectly. this led me to suspect the timezone was off somewhere.

So I checked the splunk cloud -> Settings -> Source Types settings for the sourcetype I was having the issues with...

and I found that the timezone set for the Source Type that was configured in Splunk Clouds Settings had the incorrect Time Zone set. I corrected this and I also added the following to my syslog-ng.conf files line for the udp connection settings:
time_zone(PST8PDT) keep_timestamp(no)

I then saved my changes and restarted the syslog-ng service and did a service syslog-ng status to confirm it came good and was started.
I monitored over the next 15 minutes and noted that my logs were starting to show with the correct times in splunk cloud again.

I am guessing that somehow this misconfigured Source Type was there since it was created and perhaps a dip in traffic to that log yesterday helped it to become obvious for the first time.
at any rate she seems to be working again now.

I noted that the timezone was incorrect.

0 Karma

sudosplunk
Motivator

Glad you were able to resolve this. Happy Splunking!

PS: You can accept your answer to mark this question complete.

0 Karma

sudosplunk
Motivator

I've seen this in the past but a restart would fix it. If 1 of 18 devices is having this issue, then there is a chance of latency or corrupted files. See how your thruput looks like for that sourcetype.

kbps by sourcetype: index="_internal" source=*metrics.log group="per_sourcetype_thruput" | timechart avg(kbps) by series

eps by sourcetype: index="_internal" source=*metrics.log group="per_sourcetype_thruput" | timechart avg(eps) by series

If you wanted host or source, use group="per_source_thruput" or group="per_host_thruput" instead.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...