Hi Folks, I am trying to extract fields from a text string that is included in a JSON event, submitted to Splunk via an HEC. (Additionally, modifying its sourcetype and index based on a regex. An example of the event: {"log":"2019-01-31 06:09:07:382+0000 pid=10 method=GET path=/ugc_service/ugcs/search format=json controller=ugcs action=search status=200 duration=124.48 view=15.78 db=3.25 es=100.61 time=20190131060907 service=rosi_ugc_service request_id=d79a3fd0175237c5d7d1 event=process_action.action_controller platform=keep params={\"page\":\"1\",\"per_page\":\"4\",\"query\":\"1380\",\"search_field\":\"%22\",\"sort\":\"most_recent\",\"_\":\"1482163239481\"} locale=en-US store=us x_forwarded_for=1.1.1.1, 2.2.2.2\n", "stream":"stdout", "time":"2019-01-31T06:09:07.382816774Z", "kubernetes":{"pod_name":"ugc-ugc-master-cc5b0195-unicorn-746bcc4765-sr697", "namespace_name":"rosi", "pod_id":"a134b3d9-1af0-11e9-bf52-005056b56c98", "labels":{"app":"ugc-unicorn", "pod-template-hash":"3026770321", "release":"ugc", "version":"ugc_master_cc5b0195"}, "annotations":{"cni.projectcalico.org/podIP":"10.52.4.36/32", "sidecar.istio.io/status":"{\\\"version\\\":\\\"89e068fb406338c02d8272a6459d5d4d7565ef2b228c8ad6778daffe24016ff0\\\",\\\"initContainers\\\":[\\\"istio-init\\\"],\\\"containers\\\":[\\\"istio-proxy\\\"],\\\"volumes\\\":[\\\"istio-envoy\\\",\\\"istio-certs\\\"],\\\"imagePullSecrets\\\":null}"}, "host":"us-perf-kubewrk-003.atl01.example.com", "container_name":"k8s-ugc", "docker_id":"727ca8a5aefb82385222900ef822875f7983b23550f34816e916b9836648030f"}} When initially indexed it creates all of the fields present in the JSON, both top level and nested, however I would like to "re-parse" the "log" field so that the key=value pairs are available. I guess I could parse it all with a REGEX , however that seems a little fragile, in the event that the developers add / reorder the fields in the log. I have tried using REPORT-xxx in transforms.conf without any success. As mentioned I have been updating the source type based on a regex match against the log like this: props.conf ----------------- [source::http:perform-k8s] TRANSFORMS-changesourcetype = kubernetes_change_sourcetype_access_combined, kubernetes_change_sourcetype_rosi_application TRANSFORMS-changeindex = kubernetes_change_index_access_combined, kubernetes_change_index_rosi_application transform.conf ---------------------- [kubernetes_change_index_access_combined] DEST_KEY = _MetaData:Index REGEX=^{"log":"\S+ \S+ \S+ \S* ?\[[^\]]+\] \\"[^"]*\\" \S+ \S+(?: \S+)? FORMAT = apache [kubernetes_change_index_rosi_application] DEST_KEY = _MetaData:Index REGEX=^{"log":"\S+\s+\S+\s+pid=\d+\s+method=\S+\s+path=\S+ FORMAT = rosi [kubernetes_change_sourcetype_access_combined] DEST_KEY = MetaData:Sourcetype REGEX=^{"log":"\S+ \S+ \S+ \S* ?\[[^\]]+\] \\"[^"]*\\" \S+ \S+(?: \S+)? FORMAT = sourcetype::k8s_access_combined [kubernetes_change_sourcetype_rosi_application] DEST_KEY = MetaData:Sourcetype REGEX=^{"log":"\S+\s+\S+\s+pid=\d+\s+method=\S+\s+path=\S+ FORMAT = sourcetype::k8s_rosi_application This is using Splunk release 6.6.8, with the log events being submitted via fluent-bit, I am open to trying splunk-connect-for-kubernetes (https://github.com/splunk/splunk-connect-for-kubernetes) however it feels as though I will need to deal wth the same data format either way. If you got this far, thanks for reading 🙂 mgh ... View more

Hi, I am trying to configure some index-time field extractions on a SplunkForwarder, so that I can tag all events from the Forwarder with a couple of additional fields. It is impractical to try to do this at search-time. I have tried adding the following files to the forwarder, in an attempt to follow the documentation: /opt/splunk/etc/system/local/props.conf [default] TRANSFORMS-role = xxx-role TRANSFORMS-environment = xxx-environment /opt/splunk/etc/system/local/transforms.conf [xxx-environment] DEFAULT_VALUE = common FORMAT = environment::"common" WRITE_META = true [xxx-role] DEFAULT_VALUE = client FORMAT = role::"client" WRITE_META = true /opt/splunk/etc/system/local/fields.conf [role] INDEXED=true INDEXED_VALUE=false [environment] INDEXED=true INDEXED_VALUE=false N.B. The config files look "normal" however I can't manage to format the question. The only change made to the indexing server has been to add the same fields.conf file. Having restarted both the forwarder and indexer, I am still unable to search on either "role" or "environment". (I am trying to search with environment="common" for example.) Any advice/solution would be greatly appreciated. Thanks, mgh ... View more

According to the documentation for Splunk version 3.x there is the ability to alias a sourcetype, however it does not appear to exist under version 4.x. I find myself in the position where I have many applications all logging via log4j and would like to be able to filter my searches on application type. I was hoping to be able to setup the forwarders via the CLI, adding the monitor statements with an explicit -sourcetype. The only other option I can see is to setup TAGs on each of the source statements based on filename (Can tags be managed automatically for certain sources, perhaps based on a regex?) Any suggestions or clarifications would be greatly appreciated. Regards, mgh P.S. In case it was not immediately obvious, yes I am very new to splunk. ... View more