Splunk Search

Pre-processing / Post-processing while loading data

mohitab
Path Finder

Sorry if this question lacks objectivity.

Basically, in my current SPA webapp, I am making three queries to Splunk that has CSV data loaded. The queries calculate a lot time differences and do string operations on the fields. Unfortunately, a big part of these three queries involves exact repeated sections involving these operations.

I want to improve the query time by avoiding these repetitive calculations. I was wondering if there are ways in which during loading the data:
- Either I can pre-process some of the field to add more fields .
- OR After getting data loaded, post-process the fields to add more fields.

I am also trying to get an exact breakup of query time in terms of actual query processing time and network transfer time to fetch the data.

Any suggestions !

Tags (1)
0 Karma
1 Solution

thomrs
Communicator

I use a lookup table to stash results from an expensive to enrich another query.

http://docs.splunk.com/Documentation/Splunk/6.2.1/SearchReference/Lookup

If that does not meet my need I speed things up with a time series index.

http://docs.splunk.com/Splexicon:Tsidxfile

The job inspector has all kinds of data about searches.

http://docs.splunk.com/Documentation/Splunk/6.2.1/Knowledge/ViewsearchjobpropertieswiththeJobInspect...

View solution in original post

thomrs
Communicator

I use a lookup table to stash results from an expensive to enrich another query.

http://docs.splunk.com/Documentation/Splunk/6.2.1/SearchReference/Lookup

If that does not meet my need I speed things up with a time series index.

http://docs.splunk.com/Splexicon:Tsidxfile

The job inspector has all kinds of data about searches.

http://docs.splunk.com/Documentation/Splunk/6.2.1/Knowledge/ViewsearchjobpropertieswiththeJobInspect...

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...