Add data programatically

[aburks18]

How do I add data programatically to splunk using the java sdk but none of the Input Kinds? Is there a way to add a java object as an input and inside splunk tell it how to interpret the object?

[Damien_Dallimor]

As okrabbe states , you can't send a Java Object (binary) direct to Splunk, it must be deserialized into text.

Have a look at https://github.com/damiendallimore/SplunkJavaLogging

As long as your Object has a toString() method implemented, you could use a SplunkJavaLogging appender(log4j, logback) to send the object directly to Splunk.Your toString() method is handling the deserialization.

Some examples here : https://gist.github.com/3263731

[okrabbe_splunk]

Damien, great answer. I saved the github link for future reference.

[okrabbe_splunk · Accepted solution]

Are you looking to just directly add data to an index from a java program?

For example, here is an event getting submitted over HTTP from the docs:

// Retrieve the index for the data
Index myIndex = service.getIndexes().get("test_index");

// Specify  values to apply to the event
Args eventArgs = new Args();
eventArgs.put("sourcetype", "access_combined.log");
eventArgs.put("host", "local");

// Submit an event over HTTP
myIndex.submit(eventArgs, "This is my event");

Java SDK docs

Splunk cannot just "index a java object". It needs to be converted into an ascii format at some point and it needs to make sense with some sort of time stamp.

[Neeraj_Luthra]

Did you read the code samples that Damien pointed out?

If you still need help with the 404 error, could you provide more information? Keep in mind that the code snippet expects that you have legit service object and you will also need to replace the index name to a valid one from your instance.

[aburks18]

Ok, thanks for all the help and bearing with me but I am receiving a HTTP 404 error when executing the pasted code above. Should I ask a new question for this?

[okrabbe_splunk]

Ok. The example does not actually index a log file. They are just using "access_combined.log" as the sourcetype. Your sourcetype would be "myjava_app" or whatever you want to call it.

The actual text of the event is the last call which is "This is my event".

[aburks18]

Yes I understand that, but it is not pertinent at this point. The example you pasted shows the event being indexed from a log file where you specify the sourcetype property. Does the data that splunk indexes have to exist in a file of some sort or can it be in memory as a program is running?

[okrabbe_splunk]

The examples in the link I gave show adding events directly to splunk using HTTP or an open socket. That would work fine but you just need to take into consideration reliability aspects if that is important to you. For example, how to handle the splunk indexer being down?

A forwarder would just remember its state and pick back up when the indexer comes back online but you would want code to handle that exception.

[aburks18]

What would be the alternative to writing it to a file?

[okrabbe_splunk]

It depends on the use case but the reasons it is optimal to write it to a file is that forwarders can do compression, acknowledgements, you would have the data written somewhere in case something goes wrong, and you could modify it using configuration files rather than code.

[okrabbe_splunk]

An index is where the data is stored in splunk. Later on when you access the data you would specify an index. Generally you want to group similar data in the same index. The main reasons to split off into a new index is if the data has different access requirements or if it has a different retention period.

The code I pasted was a way to programmatically push the event to Splunk using the Java SDK.

I would say though that the best practice would likely be to write an ascii representation of your object out to a file and use the splunk forwarder to send it on.

[aburks18]

Just so I'm clear, what is the difference between an index and an input?

I have this java class (let's say all it holds are strings) and I would like to take each intance of the class and add the data into splunk. To do this, you are suggesting that I first convert my object into ascii format and I shoul be able to go from there?