Monitoring Splunk

24284 bytes long Event log message causes splunkd to crash - workaround

FloydATC
Explorer

One of my Exchange 2010 hubcas servers is logging monstrous messages into the Event Log about conflicting updates from two ActiveSync devices accessing the same mailbox. (Each event Message is 24284 bytes and contains several debug traces and contains pretty much everything except what that particular Microsoft employee had for breakfast that day)

Anyway, when I try to select that Message into a table, the splunk server craps itself and must be manually restarted:
... | table _time, host, LogName, Message

Tags (4)
0 Karma

ahall_splunk
Splunk Employee
Splunk Employee

Also, I suspect that the Message field has a certain format. View the event in the event viewer and then extract just the information you need and display that. I suspect the Device ID, Device Type and username are logged in there. The actual processing doesn't care what size the event is - only the display.

0 Karma

FloydATC
Explorer

Yes, the problem only seems to affect 'table' view of the 'Message' field, showing the '_raw' event works as expected.

0 Karma

FloydATC
Explorer

Workaround: Limit the potentially huge string to a more sane length, say 1024 characters

... | eval msg=substr(Message,0,1024) | table _time, host, LogName, msg

0 Karma
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...