Splunk Search

If two events occur X seconds apart ignore the first one

dbcase
Motivator

Hi,

I have a log file that reports an event twice. It is the exact same event except it is repeated 1 or 2 or 3 or up to 5 seconds apart. It is only repeated twice. What I'd like to do is ignore the first event and report the second event.

Tags (1)
0 Karma

niketn
Legend

Can you add your field names and some sample data?

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

dbcase
Motivator

Sure!

Here is the query that I have

index=cox stuck OR unstuck  | rex "GET\s(?<URL>\S+)"  | rex "(?<threadStatus>(STUCK|unstuck))"| rex "(?:.*?ExecuteThread:\s'){2}(?<threadID>\S+)[']"  | eval timestamp=strftime(_time,"%x %X")| sort _time| dedup threadID host _time| stats list(URL) as URL list(timestamp) as Time list(threadStatus) as "Thread Status" by host threadID|sort host threadID

It generates a report that looks like this

host .               threadID           URL                                                                Time                     Thread Status

host_portal1    7                  /rest/icontrol/sites/72178/rules                01/24/17 03:01:03            STUCK
                                                  /rest/icontrol/sites/72178/rules                  01/24/17 03:02:03           STUCK
                                                                                                                            01/24/17 03:02:10           unstuck

If you notice the first two lines they are identical except for the 1 second differential in time. I'd like to eliminate one of the two lines (doesn't matter which one) so the report looks like this

host . threadID URL Time Thread Status

host_portal1    7                  /rest/icontrol/sites/72178/rules                01/24/17 03:01:03            STUCK
                                                                                                                           01/24/17 03:02:10           unstuck

Does that help?

0 Karma

dbcase
Motivator

Ugh well the formatting kinda sucks but hopefully you can get the idea.....

0 Karma

hunters_splunk
Splunk Employee
Splunk Employee

Hi dbcase,

I think you can use the dedup command to remove deplicate events that contain identical combination of values for the fields that you specify. You can specify the number of events with duplicate values, or value combinations, to keep. You can sort the fields. When you sort, the dedup command deduplicates the results based on the specified sort-by fields.
For example, assuming you use clientip and action to identify events, you can use the following search:

... | dedup clientip action sortby +_time

For detailed information about the dedup command, please refer to documentation:
http://docs.splunk.com/Documentation/Splunk/6.5.1/SearchReference/Dedup
Hope this helps. Thanks!
Hunter

0 Karma

dbcase
Motivator

Hi Hunters,

I thought about that but the challenge is the event is identical except for the time. So if I dedup and exclude the time I remove other events that I'm interested in. If I dedup and include the time it doesn't do anything because the time is unique.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...