Dashboards & Visualizations

How can I forward data from Splunk Heavy Forwarder to an external LogRhythm installation?

Phentermime
Explorer

Hi,

We are currently in a situation where we need to forward all kinds of events from a customers Splunk installation to a LogRhythm solution. For some reason, this forwarding needs to be done by Syslog - which is fine for all log data sourced from system messages over syslog - but is bad for all log data sourced from Windows event logs.

Why do I think the latter is bad?

Well, we're having problems with getting it right:

  • LogRhythm claims that they cannot receive syslog in CEF; so the Splunk app for CEF is not an option (even though it rid us from the CR/LF/NL problem of forwarding by syslog!).
  • Forwarding Windows event logs to a syslog server introduces us to the problem with // inside an event is being interpreted as a split (line breaker) resulting in one actual event ending up being as many one lined events as there are 's in the message section of the event being forwarded. Doing this when the external part also use Splunk in not a big problem since you can set SHOULD_LINEMERGE to true in props.conf on the receiving side.
  • An option for removing the issue is of course to use a SED-script in props.conf to replace the with a space when indexing the data (unfortunately not available at a later stage as for instance when forwarding 🙂 )

Then it is time to introduce the fact that LogRhythm also would like the Windows events to be in XML format. And it introduces some additional fun to the situation:

  • Setting renderXml=1 (or TRUE) is not an issue - works smoothly.
  • Setting renderXml=1 reduces the data amount with quite a few bytes per event.
  • renderXml=1 seems to give us exactly the data like you see if you in windows event viewer go to details pane and hit the radio button for "XML view". For instance, Level text is replaced by the corresponding Level number (0 for info, 3 for warning, 2 for error), as well are failure reason, and not all event text are found as event data.
  • LogRhythm are still having trouble parsing the data even though syslog forwarding are of raw XML data for Windows events.
  • I guess it should be solvable on the LR side because as I see it Splunk is only forwarding the XML formatted events as Raw to the LR syslog reception.

But how is this solvable? What am I missing on the Splunk side? Are there more to be done on the Windows side to get more or better data into the details and XML? Using XML formatted events must mean that the receiving side can resolve numbered levels, reason codes and whatnots to make it human readable at least at the alerting level.

So, in essence, the real question here is: Have anyone out there any experience with forwarding data to LogRhythm from Splunk?

Any help is deeply appreciated.

Kind regards,
John

0 Karma

jkat54
SplunkTrust
SplunkTrust

Are you sending “cooked” data to LR?

Cooked data has the splunk header with something like this:

*** SPLUNK INDEX=... SOURCETYPE=... SOURCE=... _raw=...

The xml wouldn’t start until after the _raw= which would confuse a schema on ingest app like LR.

There are options on outputs to not send cooked data. The article rich777 mentioned covers the configuration I believe..

0 Karma

Richfez
SplunkTrust
SplunkTrust

There are a variety of examples on forwarding data to third party systems. Would one of the techniques listed in the link before help?

0 Karma
Get Updates on the Splunk Community!

Introducing Splunk Enterprise 9.2

WATCH HERE! Watch this Tech Talk to learn about the latest features and enhancements shipped in the new Splunk ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...