I have logs in which some events occured twice in same timestamp.......so i need to identify and eliminate those repeated timestamp alone.... I found that with dedup it provide once occurence of timestamp... but i need to eliminated repeated timestamps alone .....
For example .,,,.. these are the fields obtained by using regex ...Fields--> Date & CountryCode
Date CountryCode
Mar 28, 2013 5:42:30 AM AUS
Mar 28, 2013 5:42:30 AM AUS
Mar 28, 2013 5:50:21 AM PAK
Mar 28, 2013 5:57:14 AM USA
Mar 28, 2013 5:59:45 AM SGP
. .
. .
. .
So here i need to eliminate repeated timestamp... here the first two field values are repeated....
So it must be eliminated......
The result should be like this......
Date CountryCode
Mar 28, 2013 5:50:21 AM PAK
Mar 28, 2013 5:57:14 AM USA
Mar 28, 2013 5:59:45 AM SGP
.
So the repeated timestamp should be avoided .... Can u guide me .....plz...............
So... you want to drop any row where the timestamp occurs more than once? Try this:
... | eventstats count by _time | where count=1
Splunk has a dedup command to remove duplicates. You just mention the fields to compare and it will keep only the first it detects.
... | dedup Date, CountryCode
Bob
So you want to remove all of the results for the duplicated events, not just the extras, right?
Assuming there are fields called Date
and CountryCode
;
...| stats c(CountryCode) as count first(CountryCode) as CountryCode by Date | where count < 2 | fields - count
/K
So... you want to drop any row where the timestamp occurs more than once? Try this:
... | eventstats count by _time | where count=1
you are really rocking man !!! . It worked finally ...thank U
If that's what he wants he can do an eventstats count by _time CountryCode to eliminate that risk.
there's a risk that two events with different CountryCodes may happen in the same second..