How to edit my configurations so that my Kubernetes host log will be reformatted entering Splunk?

[svemurilv]

Hi ,
In my kubernetes host generating logfile for the docker container, the logs are coming to Splunk in the following notation.

{   [-] 
     log:    2017-01-06 19:37:54,851 - application-library-rest-api - DEBUG - 140176312547072 - app - bundle_ids - http://172.16.88.14:5000/bundleids

     stream:     stderr 
     time:   2017-01-06T19:37:54.852272882Z 
}

But my Splunk search log should be like following format

2017-01-06 19:27:30,846 - application-library-rest-api - DEBUG - 140252463982336 - app - bundle_ids - http://172.16.88.13:5000/bundleids?bid=com.jhml2.ringtone

This is my conf files
inputs.conf

[monitor:///var/log/containers/*.log]
disabled = 0
sourcetype = kubernetes

cat transforms.conf

[kubernetes-log-filename]
REGEX = /var/log/containers/(?<podname>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<containername>(POD|[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*))-(?<containerid>[0-9a-f]+).log
SOURCE_KEY = source

[kubernetes-log-row]
REGEX = [0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3} - (?<component>(\w|-)+) - (?<level>(\w|-)+) - (?<thread>\d+) - (?<module>(\w|-)+) - (?<function>(\w|-)+) - (?<message>.*)

[kubernetes-log-row2]
REGEX = \[?[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2} \+[0-9]{4}\]?( \[[0-9]+\] )?\[(?<level>\w+)\](?<message>.*)

cat props.conf

[kubernetes]
#INDEXED_EXTRACTIONS = JSON
#FIELDALIAS-kubelog = log as _raw
TRANSFORMS-kube = kubernetes-log-filename, kubernetes-log-row, kubernetes-log-row2

please advise me how does it work, and what's the wrong in my code?

-thanks

[mattymo]

I have also managed to create a kubernetes sourcetype what will strip json logs and support sourcetype renaming, multiline logs, and multiple line breakers.

https://github.com/splunk/docker-itmonitoring/blob/7.0.0-k8s/app-k8s/default/props.conf

# Experimental props to remove json wrapper, escaped quotes, encoding and other cruft from docker logs. 
#
[kubernetes]
CHARSET=UTF-8
SHOULD_LINEMERGE=false
NO_BINARY_CHECK = true
# remove docker json wrapper, then remove escapes from the quotes in the log message. 
SEDCMD-1_unjsonify = s/{"log":"(?:\\u[0-9]+)?(.*?)\\n","stream.*/\1/g
SEDCMD-2_unescapequotes = s/\\"/"/g
# another exprimental version of the sed.
#SEDCMD-1_unjsonify = s/{"log":"(?:\\u[0-9]+)?(.*)\\n","stream.*?([\n\r])/\1\2/g
category = Custom
disabled = false
pulldown_type = true
TRUNCATE=150000
TZ=UTC

Check it out and feel free to contribute any known logs patterns and other things that need to be stripped out of those nasty docker json logs 🙂

Repo also includes a prototype app and metdata collection. Come contribute as we shape how to officially support docker/kubernetes etc!

[aaraneta_splunk]

@svemurilv - Did the answer provided by jenthomas help provide a working solution to your question? If yes, please don't forget to resolve this post by clicking "Accept". If no, please leave a comment with more feedback. Thanks!

[jenthomas · Accepted solution]

Hi,
I'm using SEDCMD in my props.conf file to achieve this:

[kubernetes]
INDEXED_EXTRACTIONS = json
SEDCMD-removelog = s/{"log":"{(.*)}"/{\1/

[tomxor]

Hi, it doesn't work for me. I've set it on the search head (attached to sourcetype).
Is this the right place?

thanks

[lguinn2]

The SEDCMD needs to go in a props.conf on the indexers (or wherever the data is being parsed, but definitely not the search head).