Getting Data In

Using the Splunk HTTP Event Collector with log4j2, why are some message field keys not being parsed?

AlGon
Engager

Hello,

I use Splunk HTTP Event Collector (splunk-library-javalogging-1.5.1.jar) with log4j2. Here is my (simplified) log4 configuration file:

 <?xml version="1.0" encoding="UTF-8"?>
    <Configuration status="DEBUG" monitorInterval="30">
        <Appenders>
            <!-- Splunk logger -->
            <Http name="http" url="http://%host%:8088" token="%splunk-token%" >
                <PatternLayout pattern="%m" />
            </Http>
        </Appenders>
        <Loggers>
            <Root level="info">
            </Root>
            <Logger name="eventLogger" level="info" additivity="false">
                <AppenderRef ref="http" />
            </Logger>
        </Loggers>
    </Configuration>

I use SplunkCimLogEvent class to format the logging message as explained in the class javadoc:

import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
...
private static final Logger EVENT_LOGGER = LogManager.getLogger("eventLogger");
...
SplunkCimLogEvent splunkCimLogEvent = new SplunkCimLogEvent("eventName", "eventId");
splunkCimLogEvent.addField("field1", field1Value);
splunkCimLogEvent.addField("field2", field2Value);
splunkCimLogEvent.setAuthApp("appName");
splunkCimLogEvent.setAuthUser("userName");

EVENT_LOGGER.info(splunkCimLogEvent.toString());

In Splunk, I get event formatted like following (raw text) :

{"message": "\"name=eventName\" \"event_id=eventId\" \"field1=field1\" \"field2=field2\" \"app=appName\" \"user=userName\"","severity":"INFO"} 

I get only "message" and "severity" as JSON title (see attached image)
alt text
whereas I was expected every message field key as JSON title.
Is it possible to do this with log4j?

Thank you.

1 Solution

gblock_splunk
Splunk Employee
Splunk Employee

Hi AlGon

I just looked at this again, my previous answer was incorrect which I deleted.

If you set CIM fields in the logging library, today, they are appended as key-value-pairs in the message. The logging libraries (log4j etc) are not built around objects. That message is then forwarded on within the JSON payload which HEC requires. Splunk does not support extracting kvp from a value of a field in a JSON object today. If it did, this would not be a problem and the fields would be properly extracted. We have a filed a bug for this and are exploring potentially fixing this in the future. We're also adding support in HEC for sending raw payloads (we have this already in cloud), so we may be able to add support in our logging libraries, which would also solve the problem.

In the interim, one option is to explore is using regex extractions to extract the fields. I've done some spiking on this which I posted here. Basically it introduces a sourcetype which extracts any kvp it finds in the "event" value. The same approach may work for your need.

View solution in original post

obesechicken13
Explorer

I don't think the appenders can log events with specific fields. They put everything in the message field. It's then difficult to extract these fields.

0 Karma

gblock_splunk
Splunk Employee
Splunk Employee

Hi AlGon

I just looked at this again, my previous answer was incorrect which I deleted.

If you set CIM fields in the logging library, today, they are appended as key-value-pairs in the message. The logging libraries (log4j etc) are not built around objects. That message is then forwarded on within the JSON payload which HEC requires. Splunk does not support extracting kvp from a value of a field in a JSON object today. If it did, this would not be a problem and the fields would be properly extracted. We have a filed a bug for this and are exploring potentially fixing this in the future. We're also adding support in HEC for sending raw payloads (we have this already in cloud), so we may be able to add support in our logging libraries, which would also solve the problem.

In the interim, one option is to explore is using regex extractions to extract the fields. I've done some spiking on this which I posted here. Basically it introduces a sourcetype which extracts any kvp it finds in the "event" value. The same approach may work for your need.

AlGon
Engager

Thanks for you answer.

I chose another solution which is to build manually the HTTP request:

    InetAddress ip = InetAddress.getLocalHost();

    JSONObject bodyRequestJson = new JSONObject();
    bodyRequestJson.put("host", ip.getHostName());
    bodyRequestJson.put("event", event.toJson());
    StringEntity httpBody = new StringEntity(bodyRequestJson.toJSONString());

    HttpClient httpclient = HttpClientBuilder.create().build();
    HttpPost httpPost = new HttpPost("http://hostname:8088/services/collector/event");
    httpPost.setHeader("Content-type", "application/json");
    httpPost.setHeader("Authorization", "Splunk token");
    httpPost.setEntity(httpBody);
    HttpResponse response = httpclient.execute(httpPost);
    LOGGER.info("http event collector response: " + response);

I obtain what I want.
I will follow the changes on Splunk HEC and log4j.

0 Karma

viliam
New Member

Can you please advise what is the progress with fixing or expected ETA?
I'm using Splunk enterprise and it would help me heaps if we can get all fields parsed.

Thanks

viliam

0 Karma

gblock_splunk
Splunk Employee
Splunk Employee

Hi villam

Did you check the link I sent. That's the only mechanism in the short term. No ETA yet on the fix in Splunk.

0 Karma

gblock_splunk
Splunk Employee
Splunk Employee
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...