How to search data which is not present in the last 60 days

[james_n]

Hi experts,

I have a multiple errors like "***error occured" , "failed error **** " and etc, I need to check what are errors occurred in the last 24 hours, whether the same errors occurred in the last 60 days or not, if not occurred in the past then trigger a alert which means these are the new errors we considering. Please help on this. Thanks in advance.

[gcusello]

Hi @james_n,
try something like this:

index=your_index earliest=-60d latest=now
| eval new=if(now()-_time<86400,"yes","not")
| stats dc(new) AS count values(new) AS new
| where count=1 AND new="yes"

Ciao.
Giuseppe

[james_n]

Hi @gcusello , Thanks for the quick response, I want to show the list of errors which are matched like i want to show which error occurred in last 24 hours but not in last 60days.

all the errors which are present in the last 24 hrs need to compare with last 60days, if not matched, we need to show those, else no need to show. Thanks for the help

[gcusello]

Hi @james_n,
in this search, you have all the errors in last 60 days and errors in last 24 hours are labelled new="yes".
So if there are errors with new="yes" and new="not", they are present in the last 24 hours AND in the previous 60 days, so they are excluded from the results.
If instead you have ONLY new="yes", this means that you have these errors only in the last 24 hours but not in the previous period, the result I understood you want.
If you want a different result, please explain it.

Ciao.
Giuseppe

[james_n]

Hi @gcusello , thanks for the info, Let me explain what I required. I have a field called cc_errors which contains multiple exceptions. Now I am counting the exceptions like base search | stats count by cc_errors
So, I want a list of exceptions in the last 24hrs which are not present in earlier like 60 days.
i.we need to compare last 24 hours exceptions with last 60 days data.
ii. If any exceptions are not matched, we need to show those, and considering as new exceptions.
Thanks in advance.

[gcusello]

Hi @james_n,
I could hint to use a search for the last 24 hours compared with the same search in the last 60 days, but this solution can run only if you have few events, because there's the limit of 50,000 results in subsearch, so, if you have (and I think that's possible) too many results, the subsearch doesn't run, so the best approach is the one I hinted:

 index=your_index earliest=-60d latest=now
 | eval new=if(now()-_time<86400,"yes","not")
 | stats dc(new) AS count values(new) AS new BY cc_errors
 | where count=1 AND new="yes"

did you tried something like this or did you find some problems with this approach?

Ciao.
Giuseppe

[james_n]

Hi @gcusello Thanks for the replay and patience, I have tried the below approach as you suggested, but I could see the errors which are present in both last 24 hours and last 60 days data. Please check once and let me know if i missed anything. Thanks in advance.

expected output: need to compare last 24 hours errors with last 60 days errors and then display not matched errors.

earliest=-60d latest=now
 | eval new=if(now()-_time<86400,"yes","not")
 | stats dc(new) AS count values(new) AS new BY cc_errors
 | where count=1 AND new="yes"

[gcusello]

hi @james_n;

yes it's correct!

only one little variation: probably you need only the cc_errors list and not also new and count fields, so add at the end of your search 

| sort cc_errors

| table cc_errors

Ciao.

Giuseppe

[james_n]

Hi @gcusello , Thanks for the reply. It's not working for me as excepted, I need to display the error list which are present only in last 24hrs but not in last 60 days. I am considering those exceptions as new exceptions.

i. need to compare last 24 hours data exceptions with last 60 days data. if any 24hrs data exceptions are not there in last 60 days data then we have to display those.

note: Expected results below 5k only so no problem with limit of 50,000 results in subsearch

thanks in advance.

[gcusello]

Hi @james_n,
I hint to spend some additional time on this approach because I used it for a Use Case like the your!
Anyway, if you're sure to have less than 50,000 results in 60 days, you can use something like this:

index=your_index earliest=-24h latest=now NOT [ search index=your_index earliest=-60d latest=-24h | dedup cc_errors | fields cc_errors ]
| ...

Ciao.
Giuseppe