Sending rsyslog JSON format

[shaharl]

Hello,

I have tried today to integrate Splunk with Rsyslog that Contains JSON.
The issue is that rsyslog is sending the information with timestamp, hostname, and application name before the JSON message that my application made.

[stmyers7941]

I did this with a similar situation where a syslog header was added to the front of XML events. In props.conf, make a LINE_BREAKER setting for the sourcetype (or whatever) and add the regex for the syslog header, and end with the opening of your JSON (XML in my example):

`[sam:xml]
# Strips syslog header and makes events pure XML
LINE_BREAKER = (\d\s\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}-\d{2}:\d{2}\s[\w_-]+\s[\w-_]+\s\d+\s-\s-\s)
SHOULD_LINEMERGE = false`

Obviously, your regex will vary. Post a sample event if you need help with the regex.

[shaharl · Accepted solution]

my application is using rsyslog to write logs, so anyway i need to find way to change the log format because it's not standard JSON.

rsyslog default fromat is TIMESTAMP HOSTNAME APP MSG

so we solved this issue by creating new template at syslog without timestamp,hostname,application (in other words - just JSON messages)

By adding at /etc/rsyslog.d/mysqpplication.conf:
$template MyTemplate,"%msg%\n"
:programname, isequal, "MYSQPP" @10.0.100.220:555;MyTemplate

And add to /opt/splunk/etc/system/local/props.conf
[MyApp]
pulldown_type = true
INDEXED_EXTRACTIONS = json
KV_MODE = JSON
category = Structured
description = MyApp

[esix_splunk]

JSON and Syslog are different formats and dont particularly play well together due to the nature of the format of syslog. There are some custom rules you can compile for Rsyslog that will help with JSON.

BUT.. since you're using Splunk, why are you trying to ingest these with rsyslog? Why not use a Splunk Universal Forwarder and ingest the JSON files directly into Splunk. Splunk understands JSON format with no problems..

[somesoni2]

Can you post some sample log entries?