Monitoring Splunk

24284 bytes long Event log message causes splunkd to crash - workaround

FloydATC
Explorer

One of my Exchange 2010 hubcas servers is logging monstrous messages into the Event Log about conflicting updates from two ActiveSync devices accessing the same mailbox. (Each event Message is 24284 bytes and contains several debug traces and contains pretty much everything except what that particular Microsoft employee had for breakfast that day)

Anyway, when I try to select that Message into a table, the splunk server craps itself and must be manually restarted:
... | table _time, host, LogName, Message

Tags (4)
0 Karma

ahall_splunk
Splunk Employee
Splunk Employee

Also, I suspect that the Message field has a certain format. View the event in the event viewer and then extract just the information you need and display that. I suspect the Device ID, Device Type and username are logged in there. The actual processing doesn't care what size the event is - only the display.

0 Karma

FloydATC
Explorer

Yes, the problem only seems to affect 'table' view of the 'Message' field, showing the '_raw' event works as expected.

0 Karma

FloydATC
Explorer

Workaround: Limit the potentially huge string to a more sane length, say 1024 characters

... | eval msg=substr(Message,0,1024) | table _time, host, LogName, msg

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...