Splunk Search

How to accelerate searches in dynamic dashboard?

stwong
Communicator

Hi all,

We created a dashboard where $d_name$ in following search is user input:

  <search>
      <query>
        <![CDATA[index=application host="landing.itsc.cuhk.edu.hk" POST OR GET status<400 $d_name$ | rex field=uri "\/(?&lt;deptcode&gt;[^\/]+)\/" | fields deptcode useragent| search deptcode=$d_name$ | timechart count
        ]]>
        </query>
  </search>
  <fieldForLabel>deptcode</fieldForLabel>
  <fieldForValue>deptcode</fieldForValue>
</input>

Seems using post-process search is slower in our environment as there are more than 1 search peer. Is that correct?
Then I try to convert the inline searches into reports and thus accelerate wherever possible. However, can I do that with input field to be passed to the saved search/report, while base search is not used?

Thanks a lot.
Regards,
/ST Wong

0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi stwong,
as first time you could extract deptcode field before, in this way you can insert the search for this field in the primary search: this accelerate your search!
see http://docs.splunk.com/Documentation/Splunk/6.5.1/Knowledge/Aboutsummaryindexing and http://docs.splunk.com/Documentation/Splunk/6.5.1/Report/Acceleratereports

You could create a summary index (e.g. using tscollect command) and then use tstats command)

index=application host="landing.itsc.cuhk.edu.hk" POST OR GET status<400 
| rex field=uri "\/(?<deptcode>[^\/]+)\/" 
| table _time deptcode useragent other_fields 
| tscollect namespace=mynamespace

| tstats count AS cnt FROM mynamespace WHERE deptcode=$d_name$ GROUPBY _time deptcode useragent other_fields
| timechart sum(cnt)

Bye.
Giuseppe

View solution in original post

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi stwong,
as first time you could extract deptcode field before, in this way you can insert the search for this field in the primary search: this accelerate your search!
see http://docs.splunk.com/Documentation/Splunk/6.5.1/Knowledge/Aboutsummaryindexing and http://docs.splunk.com/Documentation/Splunk/6.5.1/Report/Acceleratereports

You could create a summary index (e.g. using tscollect command) and then use tstats command)

index=application host="landing.itsc.cuhk.edu.hk" POST OR GET status<400 
| rex field=uri "\/(?<deptcode>[^\/]+)\/" 
| table _time deptcode useragent other_fields 
| tscollect namespace=mynamespace

| tstats count AS cnt FROM mynamespace WHERE deptcode=$d_name$ GROUPBY _time deptcode useragent other_fields
| timechart sum(cnt)

Bye.
Giuseppe

0 Karma

stwong
Communicator

Thanks. Shall I put the tscollect statement as a base search in the dashboard, and use tstats in panel searches? Seems it's beneficial when there are lot of panels that need tstats as the tscollect command takes some time to build. The dropdown list built from search is the first one in the panel, and takes some time before user can get the list (tscollect + tstats).

Besides, chances are user will select the "All" option which is "deptcode=*". Seems tstats can't do with wildcard.

Thanks again.
/ST

0 Karma

stwong
Communicator

Hi, I run tscollect multiple times for some testing. Seems data will be counted multiple times if I run tscollect on the same time range repeatedly. Is that true?

How can I delete the 'namespace' (or tsindx file) created by tscollect?

Thanks a lot.
/ST

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi stwong,
tscollect is a stupid command: it doesn't check if the events was already ingested, so you have to configure you time periods without overlap otherwise you'll have duplicate events.
If you cannot do this, you have to insert in your tscollect search also a condition on indextime, discarding the ones with indextime < the time of last schedule (e.g. if scheduling is every hour, -h@h).
I asked to splunk to insert this check in the future developments and I hope!
In addition it isn't possible to delete some events from a tsidx index, you can only delete the entire index.
You can delete the tsidx index phisically deleting files that you can find by default in $SPLUNK_HOME/var/lib/splunk/tsidxstats/
Bye.
Giuseppe

0 Karma

stwong
Communicator

Hi Giuseppe,

Thanks a lot for your help. Will try to make the time periods correct to avoid duplicated events. However, seems it's a bit difficult for us to use tscollect as we're using search head cluster, without any shared storage.
Anyway, will keep trying in order to learn more.
Thanks again and bye.

/ST

0 Karma

gcusello
SplunkTrust
SplunkTrust

No you have to insert tscollect in a scheduled search and use tstats in your panels.
Tscollect creates a parallel index with less fields, all indexed and more quicker.

0 Karma

stwong
Communicator

If making tscollect in a scheduled search, shall I specify time range based on the scheduled, say running everyday, with time range of -1d@d so that all data will be tscollect'd?

Thanks again.
/ST

0 Karma

gcusello
SplunkTrust
SplunkTrust

Yes, tscollect schedule is related to your needs about near real time monitoring: so you can schedule one time a day using -d@d - @d time period or less time.
Bye.
Giuseppe

0 Karma

stwong
Communicator

Thanks a lot.

Bye
/ST

0 Karma

niketn
Legend

I am assuming you have multiple Search Heads and Multiple Indexers. Splunk documentation mentions that Post-Processing is not useful in case of Multiple Indexers, it also suggests to reuse the same search in suce case (refer to Post-process searches) : http://docs.splunk.com/Documentation/Splunk/latest/Viz/Savedsearches

A post-process is effective for conserving resources if the base search is on a single indexer. If you are in an environment with a search head using multiple indexers, post process might not be effective for conserving resources. In this scenario, it might be more effective to use the same search multiple times within a dashboard.
____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

stwong
Communicator

Right, that's what I read from manual. Thanks a lot.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...