Getting Data In

UDP droppage because of ext4 filesystem:

balbano
Contributor

I have been having an issue where one of my 2 log servers have dropping a tremendous amount of UDP packet data (from syslog-ng/rsyslog based traffic).

One of 2 log servers has been dropping UDP packets like crazy. However the other one was fine.

While the changes that were mentioned in here did improve the situation, the drop rate was at a significant level where it was ridiculous.

After banging my head over it for why one was dropping and the other one wasnt dropping, I realized a key difference in the log servers: The working server was having the logs write to an ext3 partition and the server dropping logs was writing to an ext4 filesystem.

As a test, I moved the log destination to an ext3 filesystem with default settings and now its working fine.

Now the question, what are the appropriate ext4 settings for receiving syslog-ng / rsyslog data?

This is what I currently have setup (which is causing the UDP Droppage):

/dev/$my_device /$my_log_dest ext4 noatime,data=writeback,defaults,acl 1 2

I suspect its possibly my journaling option "data=writeback" but I'm not for certain.

Can someone give some insight on this?

Thanks.

Brian

Tags (3)

Wilcooley
Path Finder

Sorry for bringing up an old question but I happened upon this after some recent IRC discussion.

I am curious about the size of the journal in your ext4 file system. With ext3 (and presumably ext4 by extension), having too small of a journal was a source of stalls or hangs when writing. This could happen if you initially created a small file system and then grew it significantly. You can find out with the dumpe2fs command (sub '4' for '2' if on EL5):

dumpe2fs -h /dev/XXX |grep Journal

It would also be interesting to know what features are enabled; you can get that with either the dumpe2fs or tune2fs -l command.

Also, what kernel version & distro are you using?

I am assuming that you're using Splunk as the UDP listener and not feeding via an intermediary syslog server? (My Splunk never sees UDP traffic because I feed it via rsyslog.)

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...