Getting Data In

After deploying KV_MODE = auto_escaped in props.conf to my search head cluster, why are we seeing unexpected search results?

dankulinski
New Member

I am trying to set up KV_MODE = auto_escaped for a particular source. The stanza looks like the following:

[source:///var/log/test.log]
KV_MODE = auto_escaped

I used the test data directly from the Splunk documentation:
field = "value with \"nested\" quotes."

The resulting search shows the field, field with a value with \.

I have set this in the props.conf on the deployer in the following areas:

$SPLUNKHOME/etc/master-apps/_cluster/local/props.conf

$SPLUNKHOME/etc/shcluster/apps/search/props.conf

Neither of these produce the correct results.

0 Karma

lguinn2
Legend

The KV_MODE in props.conf only applies to search-time field extractions. This means that automatic field extractions occur only for data that has already been indexed. So in the indexed data, the backslashes must appear. This is actually fairly rare in text files (or data streams).

Can you show an example of some data that results from a search and has escaped characters?

0 Karma

dankulinski
New Member

Here is the data example:

Oct 10 22:53:38 server.com com.MetricsForwarder message_type=raw, TEAM=INGEST, version=2.0, environment=xxx, message="{\"header\":
{\"namespace\":\"xxxx\",\"route-definition-id\":\"mobile-device-id\",\"job-instance-id\":\"123456789\",\"step-name\":\"\",\"step-instance-id\":\"123456789\",\"job-status\":\"SUCCESS\",\"step-status\":\"SUCCESS\",\"message-timestamp\":123456789}

,\"payload\":\"{
n \\\"namespace\\\" : \\\"xx-xxxxx\\\",
n \\\"name\\\" : \\\"xxxx\\\",
n \\\"internal-name\\\" : \\\"xxxx\\\",
n \\\"version\\\" : \\\"1.0\\\",
n \\\"state\\\" : \\\"published\\\",
n \\\"category-internal-name\\\" : \\\"id-mapping\\\",
n \\\"provider-id\\\" : \\\"1234\\\",
n \\\"provider-name\\\" : \\\"XXXX\\\",
n \\\"input\\\" :
{\\n \\\"category-internal-name\\\" : \\\"input\\\",\\n \\\"input-uris\\\" : [ \\\"xx:\/\/xxxxxxxxxx-xxxxx\/prod\/data\/data\/input\/4312\/20161005\/mobile.123456789.DL.send\\\" ]\\n }

,
n \\\"output\\\" : [
{\\n \\\"category-internal-name\\\" : \\\"id-mapping\\\",\\n \\\"output-uris\\\" : [ \\\"nnn:\/\/xxxxx-xxxxx\/prod\/etl\/xxx\/xxxx\/device_processing\/data_canonical\/4312\/xx_xxxxx\/123456789\/123456789\\\" ]\\n }

],
n \\\"update-timestamp\\\" : 123456789,
n \\\"status\\\" : \\\"success\\\",
n \\\"instance-id\\\" : \\\"application_123456789\\\",
n \\\"input-validation\\\" : {
n \\\"threshold\\\" : {
n \\\"exceeds-threshold\\\" : false,
n \\\"exceeds-drop-threshold\\\" : false,
n \\\"exceeds-warn-threshold\\\" : false,
n \\\"percent\\\" :
{\\n \\\"drop-percent\\\" : 100,\\n \\\"warn-percent\\\" : 100\\n }


n },
n \\\"summary\\\" :
{\\n \\\"total-processed\\\" : 95,\\n \\\"total-pass\\\" : 95,\\n \\\"total-drop\\\" : 0,\\n \\\"total-warn\\\" : 0,\\n \\\"total-filter\\\" : 0\\n }

,
n \\\"failed-validations\\\" : [ ]
n },
n \\\"output-validation\\\" : {
n \\\"threshold\\\" : {
n \\\"exceeds-threshold\\\" : false,
n \\\"exceeds-drop-threshold\\\" : false,
n \\\"exceeds-warn-threshold\\\" : false,
n \\\"percent\\\" :
{\\n \\\"drop-percent\\\" : 5,\\n \\\"warn-percent\\\" : 5\\n }


n },
n \\\"summary\\\" :
{\\n \\\"total-processed\\\" : 95,\\n \\\"total-pass\\\" : 95,\\n \\\"total-drop\\\" : 0,\\n \\\"total-warn\\\" : 0,\\n \\\"total-filter\\\" : 0\\n }

,
n \\\"failed-validations\\\" : [ ]
n },
n \\\"output-profile\\\" : {
n \\\"id\\\" : 1,
n \\\"internalName\\\" : \\\"xxxxxxxxxx\\\",
n \\\"name\\\" : \\\"xxxx XXXX xxxx xxxxx\\\",
n \\\"profileDefintionUri\\\" : null,
n \\\"dataSetUri\\\" : null,
n \\\"schema\\\" : null,
n \\\"variables\\\" : null,
n \\\"metrics\\\" : [ {
n \\\"id\\\" : 2,
n \\\"displaySequence\\\" : 2,
n \\\"internalName\\\" : \\\"xxxx_x_x_xxxx_xx_xxxx\\\",
n \\\"name\\\" : \\\"Generic comment\\\",
n \\\"metrics\\\" : [ {
n \\\"dimensions\\\" :
{\\n \\\"device_id_type\\\" : \\\"1\\\"\\n }

,
n \\\"value\\\" : 92
n } ]
n }, {
n \\\"id\\\" : 1,
n \\\"displaySequence\\\" : 1,
n \\\"internalName\\\" : \\\"xxxx_x_xx_xxxx\\\",
n \\\"name\\\" : \\\"Generic field\\\",
n \\\"metrics\\\" : [
{\\n \\\"dimensions\\\" : null,\\n \\\"value\\\" : 92\\n }

]
n }, {
n \\\"id\\\" : 1,
n \\\"displaySequence\\\" : 0,
n \\\"internalName\\\" : \\\"xxxx_x_xx_xxxx\\\",
n \\\"name\\\" : \\\"Generic Field 2\\\",
n \\\"metrics\\\" : [
{\\n \\\"dimensions\\\" : null,\\n \\\"value\\\" : 95\\n }

]
n } ]
n },
n \\\"process-counts\\\" :
{\\n \\\"input\\\" : 95,\\n \\\"input-validation-input\\\" : 95,\\n \\\"input-validation-output\\\" : 95,\\n \\\"transform-input\\\" : 95,\\n \\\"transform-output\\\" : 95,\\n \\\"output-validation-input\\\" : 95,\\n \\\"output-validation-output\\\" : 95,\\n \\\"output-id-mapping\\\" : 95\\n }

\\n}\"}"
0 Karma

dankulinski
New Member

Wanted to ask one last time, did the above example help to show the data we are collecting and if it will be split right with the auto kv_mode?

0 Karma

dankulinski
New Member

Did the above example help to show the issue we are having? Hoping to get more clarification on this issue.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...