Splunk Search

Question about timemodifier

yuwtennis
Communicator

alt textHi!

I would like to ask about the timemodifier.

I have a following search including subsearch,

index=hoge
[ search index=hoge _index_earliesst=-1d@d _index_latest=@d
| stats
earliest(start) as earliest
latest(stop) as latest by field
| eval earliest=substr(earliest,5,2) . "/" . substr(earliest,7,2) . "/" . substr(earliest,1,4) . ":" . substr(earliest,9,2) . ":" . substr(earliest,11,2) . ":" . substr(earliest,13,2)
| search conditionA
| eval latest=substr(latest,5,2) . "/" . substr(latest,7,2) . "/" . substr(latest,1,4) . ":" . substr(latest,9,2) . ":" . substr(latest,11,2) . ":" . substr(latest,13,2)
| fields field earliest latest
| format "(" "(" "" ")" "OR" ")"
]

My purpose is to search the events that meets the conditionA that were indexed the previous day
and pass the earliest and latest time of each field to the main search.

However, when the number of events should the main search returns are 5000 , it scans more number of events.

For example,
field earliest latest
fieldA 1/25/2014 00:00 1/25/2014 01:00 3 records exists
fieldB 1/25/2014 02:00 1/25/2014 02:00 5 recoreds exists
fieldC 1/26/2014 00:00 1/26/2014 01:00

  • my latest event in this record is 1/25/2014 01:50:00

if I expect the subsearch to return (fields="fieldA" earliest="1/25/2014:00:00" latest="1/25/2014:01:00") OR ( field="fieldB" earliest="1/25/2014:02:00" latest="1/25/2014 02:00") , I expect the main search to scan only 8 records, But it seems that it scans the event more than I expect.

Is the timemodifier not working corrctly if you concatenate with OR's?

I have added a screen shot where the scanned events are increasing although the mathing events are finished.

Thanks,
Yu

Tags (2)
0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Right - based on your comments, you're probably looking for the map command. Consider these two simple searches:

index=_internal (earliest=-30d@d latest=-29d@d) OR (earliest=@d latest=now) | bin _time span=1d | stats count by _time

That's roughly what you're doing now, tell the search to OR two timeranges... and as you observed, this scans the events in between rather than skipping to the past instantaneously. On my PC, this takes about four seconds.

| stats count | eval times="-30d@d|-29d@d,@d|now" | makemv delim="," times | mvexpand times | makemv delim="|" times | eval starttime=mvindex(times,0) | eval endtime=mvindex(times,1) | map search="search index=_internal earliest=$starttime$ latest=$endtime$ | bin _time span=1d | stats count by _time"

This achieves the same thing, but in a different way. The first bit before map creates two events with fields starttime and endtime, which are used by map to run two searches based on those timeranges. The result is the same as in the earlier search, but my PC only takes about one second.

This should be applicable to your problem, run the current subsearch first and pass into map the earliest/latest times as well as the condition.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Not quite sure about a thousand... map will run a search for each event it's given, so it would run a thousand searches. By default it stops after ten, so running a thousand may or may not break things. Just give it a shot...

While you're studying map, take a look at localize - maybe that could simplify your subsearch for building the timeranges.

0 Karma

yuwtennis
Communicator

Thanks martin!

I have to do some studying with this though. By the way would this work with about thousand set of following conditions( all concatenated with OR)?

(field=hoge1 earliest=xxxx latest=xxxx)

0 Karma

yuwtennis
Communicator

Yes.

So it will be like,

(field=hoge1 earliest=xxxx latest=xxxx) OR (field=hoge2 earliest=xxxx latest=xxxx) OR ..... and so on.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Good, then I don't need to think about that 🙂

Are you basically trying to do this: "Run a subsearch, get a bunch of results, use each result to build a timerange and possibly further filters, and run a search off that"?

0 Karma

yuwtennis
Communicator

Hello martin.

Yes. The subsearch returns the set of
field
earliest time in relative time format
latest time in relative time format

format command is working fine.

It seems that the time modifier is not working as expected in each AND boolean expression. Maybe putting to much time modifier confuses the search...

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Does the subsearch return what you expected? (Check search inspector)

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...