Getting Data In

log with key value pair or transforms.conf performance diffrence?

jazzythemartian
New Member

Hi,

to gain index size I made the log format as below. I didn't use key value pair.

20121101095842|192.168.1.2|KRQQQShcnQdRK8pLKTXC|20138494756382|I|PLAY|this the detailed info|1

And in transforms.conf I defined the fields.
DELIMS="|"
FIELDS=time,sourceip,session_id,customer_id,channel,op_type,detail,result_code

What if I made the log format like;

time=20121101095842,sourceip=192.168.1.2,sessiın_id=KRQQQShcnQdRK8pLKTXC,customer_id=20138494756382,channel=I, op_type=PLAY, detail=this the detailed info|result_code=1

Is there any performance diffrence between these two? a big diffrence in speed?

thanks,

a.

0 Karma

kristian_kolb
Ultra Champion

Well, as you've probably calculated, you'll save some license space - in this case like 40%. I cannot see any immediate downside to the approach - as long as you keep the number and order of fields constant. With key=value pairs, that is not relevant, as the extraction takes place automatically.

You should probably set KV_MODE=none for this sourcetype in props.conf.

http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf

Whether a REPORT is faster than KV_MODE=auto... I don't know - perhaps a little.

/K

0 Karma

sowings
Splunk Employee
Splunk Employee

I agree with your gut.

0 Karma

kristian_kolb
Ultra Champion

Naturally - having both is the worst 🙂

Gut feeling says that REPORT + KV_MODE=none should be faster than KV_MODE=auto. Should be fewer, less complicated steps. Though for some searches the difference might not be even noticeable.

sowings
Splunk Employee
Splunk Employee

REPORT with DELIMS is definitely faster if you turn off KV_MODE=auto for that type. 🙂 I'm not sure if "properly configured" REPORT with DELIMS alone is faster than key=value pairs, however.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...