Getting Data In

Splunkd tainted with universal forwarder 7.1.2 on Linux kernel 4.9

sakti
Engager

My Splunk Universal forwarder crashes with following logs as soon as started . I don't see same crash on a different kernel,

Splunk universal forwarder version: 7.1.2-a0c72a66db66.i386
splunkd.log has,

02-01-2019 10:46:23.792 -0800 ERROR ProcessRunner - Error from ProcessRunner helper process: ERROR - Failed opening "": No such file or directory
02-01-2019 10:46:23.792 -0800 ERROR ProcessRunner - Error from ProcessRunner helper process: terminate called after throwing an instance of 'EventLoopException'
02-01-2019 10:46:23.792 -0800 ERROR ProcessRunner - Error from ProcessRunner helper process: what(): Main Thread: about to throw an EventLoopException: error from EventLoop poll: No such file or directory
02-01-2019 10:46:23.959 -0800 FATAL ProcessRunner - Unexpected EOF from process runner child!
02-01-2019 10:46:23.959 -0800 ERROR ProcessRunner - helper process seems to have died (child killed by signal 6: Aborted)!

dmesg has,

Feb 1 10:13:37 co169 kernel: [65023.192026] CPU: 0 PID: 13240 Comm: splunkd Tainted: P O 4.9.108.Ar-10738448.4213F #1
Feb 1 10:13:37 co169 kernel: [65023.192029] task: ffff880128fb8bc0 task.stack: ffffc90005518000
Feb 1 10:13:37 co169 kernel: [65023.192031] RIP: 0023:[<00000000f77d6c09>] [<00000000f77d6c09>] 0xf77d6c09
Feb 1 10:13:37 co169 kernel: [65023.192038] RSP: 002b:00000000ffb93760 EFLAGS: 00200206
Feb 1 10:13:37 co169 kernel: [65023.192040] RAX: 0000000000000000 RBX: 00000000000033b8 RCX: 00000000000033b8
Feb 1 10:13:37 co169 kernel: [65023.192041] RDX: 0000000000000006 RSI: 00000000ffb93828 RDI: 00000000f70f8000
Feb 1 10:13:37 co169 kernel: [65023.192043] RBP: 00000000ffb93778 R08: 0000000000000000 R09: 0000000000000000
Feb 1 10:13:37 co169 kernel: [65023.192044] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Feb 1 10:13:37 co169 kernel: [65023.192046] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Feb 1 10:13:37 co169 kernel: [65023.192048] FS: 0000000000000000(0000) GS:ffff88013fa00000(0063) knlGS:00000000f6dae700

Older version of Universal forwarder (6.6) works on the same kernel(4.9). From kernel message with flag P, is this crash related to the way splunk licensing works(assuming this changed between 6.6 and 7.1) ?

0 Karma

sakti
Engager

Version 7.0 crashes as well. Downgrading to 6.6 worked, but would like to get support for 7+ versions.

My system information is:

bash-4.3# uname -a
Linux co169 4.9.108.Ar-10738448.4213F #1 SMP PREEMPT Sat Dec 15 12:30:10 PST 2018 x86_64 x86_64 x86_64 GNU/Linux

0 Karma
Get Updates on the Splunk Community!

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics GA in US-AWS!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...