How do I add the results from Search2 (last 5 minutes) to Search1 (last 30 days) with respect to differences in time ranges?

[x2xj]

Hi there,

I have two searches that work great independently, however, I now have a need to combine them both. The first search looks over a 30 day time frame which is what I want. The second search only looks at the last 5 mins which is also what I want. Specifically, I need the results from the second search added to the first search with respect to the differences in time frames.

The problem I seem to be running into is that the stats dc(pid) within the first search is pulling the dc of PIDs over the last 30 days which included PIDs that are not currently running and throws our numbers off. What we care about is the dc of PIDs currently running which is where the idea of putting in a hardcoded value of last 5min specifically for the PIDs came from.

Search 1 (last 30 days):

index=unix_metrics (host=y0123t322 OR host=y0123t323 OR host=y0123t324 OR host=y0123t325 OR host=y0123t326)  (sourcetype=vmstat OR sourcetype=CPU OR sourcetype=ps) (CPU=all OR process="bwengine*" OR memUsedPct>=0) | eval cpu_percentage=100-pctIdle | stats avg(cpu_percentage) as "cpu_perct_used" avg(memUsedPct) as "memory_perct_used" dc(pid) as "CountofBWEngines" by host | eval Deployable=case((cpu_perct_used<=80 AND memory_perct_used <=80), "Yes",(cpu_perct_used>=80 OR memory_perct_used>=80), "No") | eval Summary="CPU: ".round(cpu_perct_used, 0)."% Memory: ".round(memory_perct_used, 0)."% (".CountofBWEngines." engines)" | table host,Deployable,Summary | sort - Deployable

Search 2 (last 5 mins):

index=unix_metrics (host=y0123t322 OR host=y0123t323 OR host=y0123t324 OR host=y0123t325 OR host=y0123t326)  (sourcetype=ps) (process="bwengine*")| stats dc(pid) as "CountofBWEngines" by host

[Richfez]

That's a bit to untangle, but I think your issue is that you have a 5 minute search that defines the pids on which hosts that you want to collect more information from a longer search about? Boy, even that summary sounds confusing. Anyway. I can't tell from what you wrote if you need to join on PID or on host but I'll (probably incorrectly) assume PID.

First step, get that list of pids from the hosts in the last 5 minutes.

index=unix_metrics earliest=-5m (sourcetype=ps) (process="bwengine*") | fields pid

Test that - it should return a list of pids and that's all. If you are trying to search on host, just fields host at the end instead of fields pid.

Next, use that as a subsearch. A subsearch will get executed before the "main" search and is a secondary search you can use as arguments to another command. (That text gleefully stolen almost verbatim from one of the subsearch docs page.)

index=unix_metrics [index=unix_metrics earliest=-5m (sourcetype=ps) (process="bwengine*") | fields pid]  (sourcetype=vmstat OR sourcetype=CPU OR sourcetype=ps) (CPU=all OR process="bwengine*" OR memUsedPct>=0) | eval cpu_percentage=100-pctIdle | ...

The subsearch is the part in the [...] brackets. Suppose your subsearch returns 55, 66 and 77, that will substitute into you search.

For example, a really simple one:

* [search earliest=-5m host="myhostname" | top 5 dest_ip | fields dest_ip]

The subsearch returns the top 5 dest_ip fields. If I set the search as a whole to be the last 1 day (or whatever), then the * runs over that time frame (being a wide-open, very bad search) and includes all results where those top 5 ip addresses exist in some form in the event over the last day. Tada!