Splunk Search

How can I create a subsearch with multiple time ranges?

aguthrie1190
Path Finder

I'm trying to write a subsearch that searches multiple sections of time. What I have works until I wrap it in brackets for the subsearch. Here is what I have so far:

| gentimes start=-3
| eval earliest=starttime
| eval latest=starttime+20
| eval search="My search here"
| fields search earliest latest
| format "index=notAnIndex earliest=-1m | append [search" "" "" "" "] | append [search" "]"
| rex field=search mode=sed "s/\"//g"

This returns this search string:

 index=notAnIndex earliest=-1m 
| append [search earliest=1521504000 latest=1521504020 My search here ] 
| append [search earliest=1521590400 latest=1521590420 My search here ] 
| append [search earliest=1521676800 latest=1521676820 My search here ]

I have to use append instead of OR's because using OR will search the whole time frame between the earliest and latest time specified, which defeats the purpose of this. If I copy and paste the returned search string into a new search, it works great. If I wrap the original search in brackets, I get this error:

Error in 'append' command: The last argument must be a subsearch.

Looking through the job inspector, I can see that my earliest and latest times are being stripped out of the search. And then the error is probably from that _time that is tacked on to the end of the subsearch

litsearch index=notAnIndex 
| append [search My search here ] 
| append [search My search here ] 
| append [search My search here ] _time>=1521676800.000 _time<1521676820.000 
| fields keepcolorder=t "*" "_bkt" "_cd" "_si" "host" "index" "linecount" "source" "sourcetype" "splunk_server" 
| remotetl nb=300 et=1521676800.000000 lt=1521676820.000000 remove=true max_count=1000 max_prefetch=100

This seems like a bug, but maybe I just don't understand how the subsearch is supposed to work here. Has anyone found a way to make something like this work?

0 Karma

dlee360
Explorer

I’m going to be honest, i didn’t read your question throughly and I’m on mobile, but it seems like you could do this via multisearch instead of all the sub searches.

| multisearch [search index=test earliest=-3d latest=-2d] [search index=test earliest=-2d latest=-1d] | stats blah

0 Karma

aguthrie1190
Path Finder

Unfortunately that doesn't do the trick. But I'm definitely going to start using multisearch, that seems like a lot simpler solution than append.

Still, when I format that string into a subsearch, it pulls the times outside of each of the individual searches.

[| gentimes start=-3
| eval earliest=starttime
| eval latest=starttime+30
| eval search="*"
| fields search earliest latest
| format "| multisearch [search" "" "" "" "] [search" "]"
| rex field=search mode=sed "s/\"//g"
| rex field=search mode=sed "s/'/\"/g"]

leads to this error:
Error in 'multisearch' command: Invalid argument: '_time>=1522022400.000'

And this is from logs for what it's actually searching in the subsearch:

litsearch | multisearch [search ] [search ] [search ] _time>=1522022400.000 _time<1522022430.000 
| fields keepcolorder=t "*" "_bkt" "_cd" "_si" "host" "index" "linecount" "source" "sourcetype" "splunk_server" 
| remotetl nb=300 et=1522022400.000000 lt=1522022430.000000 remove=true max_count=1000 max_prefetch=100

So that _time>=1522022400.000 _time<1522022430.000 bit is messing up the subsearch.

0 Karma

p_gurav
Champion

Hi,

Can you try:

| append [search index=anyindex earliest=1521504000 latest=1521504020 | remaining search ] 
0 Karma

aguthrie1190
Path Finder

Can you expand a little? I don't know where you meant for me to put that. I added it to the beginning of the format section, so that the subsearch would output the append at the beginning, but unfortunately that did not work.

0 Karma

p_gurav
Champion

Can you tell e what search you are running in append command, so i can explain it better.

0 Karma

Richfez
SplunkTrust
SplunkTrust

Agreed, I think the actual search you are running would be useful - there may be something in there that's "upsetting" the closing bracket or causing this issue.

Could you post the actual search please?

0 Karma

aguthrie1190
Path Finder

[ | gentimes start=-3
| eval earliest=starttime
| eval latest=starttime+20
| eval search="*"
| fields search earliest latest
| format "index=notAnIndex earliest=-1m | append [search" "" "" "" "] | append [search" "]"
| rex field=search mode=sed "s/\"//g"]

0 Karma

aguthrie1190
Path Finder

That should get you every log for the 20 seconds between midnight and 00:00:20 for the last three days. If you want to limit that to just a specific string:

[ | gentimes start=-3
| eval earliest=starttime
| eval latest=starttime+20
| eval search="call_start"
| fields search earliest latest
| format "index=notAnIndex earliest=-1m | append [search" "" "" "" "] | append [search" "]"
| rex field=search mode=sed "s/\"//g"]

If you remove the brackets, you can see that the search is formatted properly. If you look at the job inspector it appears that the earliest/latest times are stripped from the subsearch

0 Karma

aguthrie1190
Path Finder

This seems like it is a bug. I've made a simpler search to show the behavior.

This works:

[| gentimes start=-1
| eval search="* earliest=-5m"]

This doesn't work:

[| gentimes start=-1
| eval search="* earliest=-5m | append [ search * earliest=-20m latest=-15m ]"]
0 Karma

aguthrie1190
Path Finder

You can replace "My search here" with the search "*" to see the behavior. I'm using "call_start" as that's a frequently seen log in my system.

The idea is to take the output of gentimes, get a lot of 30 second slices going back many days, and then append each of those 30 second slices together to get a picture of what happens historically.

I'm using Splunk 6.4.0 in production, I tried this in a lab in 7, and it didn't error out with the same error message, but the same behavior was seen in the job inspector.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...