Getting Data In

Did the forwarder stopped forwarding, possibly due to ulimit?

ramprakash
Explorer

Hi Splunkers,

I have the forwarder installed on nix machine. It was working perfectly until today when I made some changes in Inputs.conf to add more log files. When I restarted the forwarder again, it came up and Splunk is restarted successfully but no logs were forwarding.

However, I did face some warning. "Set the Ulimit, Splunk may not work"
Is Ulimit the issue?
If it is, then suddenly why did it stopped working?

Thanks for the help.

0 Karma

woodcock
Esteemed Legend

Yes, this absolutely could, and based on the warning, probably has caused the problem. You have run out of address space to store open file descriptors for splunk:
http://www.georgestarcher.com/splunk-ulimits-and-you/

ddrillic
Ultra Champion

On the forwarder, for the proper id, what does the ulimit command show - ulimit -n?

0 Karma

ramprakash
Explorer

This is the present setting

time(seconds) unlimited
file(blocks) 2097151
data(kbytes) unlimited
stack(kbytes) 32768
memory(kbytes) unlimited
coredump(blocks) 2097151
nofiles(descriptors) 2000
threads(per process) unlimited
processes(per user) unlimited

0 Karma

ddrillic
Ultra Champion

nofiles(descriptors) 2000 is almost the minimum - it should be higher.

0 Karma

amitm05
Builder

The below post has helped many on this issue. You'd want to check this -
https://answers.splunk.com/answers/13313/how-to-tune-ulimit-on-my-server.html

0 Karma

ramprakash
Explorer

Hi.So ulimit may be the reason it suddenly stopped forwarding when i restarted ??
It was working fine previously

0 Karma

amitm05
Builder

yes, this can be the reason as your errors are directly pointing to that. Can you make sure, the additional monitoring that you added, how many files and what size are they ? you can check for the resources usage by splunkd on you m/c to see for the performance.
If the additional monitoring requires splunk to open too many file descriptors but the defined ulimit is not sufficient, you'd face this problem

0 Karma

bryhenderson
Explorer

Maybe Splunk is monitoring too many files on your forwarder for the OS to handle. You could try increasing the ulimits:

https://docs.splunk.com/Documentation/Splunk/7.3.0/Troubleshooting/ulimitErrors

I would also ensure you didn't accidentally add a directory with a huge volume of files. I'd double check your inputs.conf.

ramprakash
Explorer

Thanks ..Can i change the ulimits to unlimited...will it not impact OS performance

0 Karma

bryhenderson
Explorer

I'd take into account what else the server is doing and how many files you are monitoring, as well as the type of hardware your server is using. Bumping up the ulimits will allow the OS to monitor more files but at a cost of performance.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...