Hi,
I have a log file that reports an event twice. It is the exact same event except it is repeated 1 or 2 or 3 or up to 5 seconds apart. It is only repeated twice. What I'd like to do is ignore the first event and report the second event.
Can you add your field names and some sample data?
Sure!
Here is the query that I have
index=cox stuck OR unstuck | rex "GET\s(?<URL>\S+)" | rex "(?<threadStatus>(STUCK|unstuck))"| rex "(?:.*?ExecuteThread:\s'){2}(?<threadID>\S+)[']" | eval timestamp=strftime(_time,"%x %X")| sort _time| dedup threadID host _time| stats list(URL) as URL list(timestamp) as Time list(threadStatus) as "Thread Status" by host threadID|sort host threadID
It generates a report that looks like this
host . threadID URL Time Thread Status
host_portal1 7 /rest/icontrol/sites/72178/rules 01/24/17 03:01:03 STUCK
/rest/icontrol/sites/72178/rules 01/24/17 03:02:03 STUCK
01/24/17 03:02:10 unstuck
If you notice the first two lines they are identical except for the 1 second differential in time. I'd like to eliminate one of the two lines (doesn't matter which one) so the report looks like this
host . threadID URL Time Thread Status
host_portal1 7 /rest/icontrol/sites/72178/rules 01/24/17 03:01:03 STUCK
01/24/17 03:02:10 unstuck
Does that help?
Ugh well the formatting kinda sucks but hopefully you can get the idea.....
Hi dbcase,
I think you can use the dedup command to remove deplicate events that contain identical combination of values for the fields that you specify. You can specify the number of events with duplicate values, or value combinations, to keep. You can sort the fields. When you sort, the dedup command deduplicates the results based on the specified sort-by fields.
For example, assuming you use clientip and action to identify events, you can use the following search:
... | dedup clientip action sortby +_time
For detailed information about the dedup command, please refer to documentation:
http://docs.splunk.com/Documentation/Splunk/6.5.1/SearchReference/Dedup
Hope this helps. Thanks!
Hunter
Hi Hunters,
I thought about that but the challenge is the event is identical except for the time. So if I dedup and exclude the time I remove other events that I'm interested in. If I dedup and include the time it doesn't do anything because the time is unique.