Hey folks,
I'm looking at a summary index that's being generated through the Splunk Web (e.g. the source is being set to the savedsearch's name, into an index selected right at the bottom of Splunk Web for the saved search).
We're plugging in plenty of data, and it's ticking along nicely. The search command is basically this:
index=raw_index_with_stuff source=specific_alpha
| stats avg(field_a) AS field_a avg(field_b) AS field_b avg(field_c) AS field_c by host
| rename host AS hostname
Nothing too complicated.
The events in the summary index similarly look right. E.g.:
01/09/2017 11:55:00 -0500, search_name="Summary_Stuff_5m", search_now=1483981500.000, info_min_time=1483980900.000, info_max_time=1483981200.000, info_search_time=1483981502.975, field_a=1.000 field_b=1.921 field_c=1.114 hostname=laptop1701a
Now, my problem is looking at the fields. If we do a simple query like:
index=summary_index_with_stuff source=Summary_Stuff_5m
| timechart avg(field_a) by hostname
It doesn't generate results. (It's also averaging averages, which is somewhat dubious, I know.)
Clicking on the events that feed into that (e.g. into the events tab), we see all the individual events properly summarised, as above, which leaves me scratching my head as to why that's the case.
When I click on an individual event (the > under the i column), we get closer to the heart of the issue:
A colleague of mine suggests this could be the result of the summarising search running twice - but that would surely generate two identical events, not an event with identical values in what should be single valued fields?
My suspicion is that it could be interplay with a field extraction (or similar)? (As it looks, to me, like certain fields have been extracted twice), but beyond 'a field extraction' (there are many), I'm currently stumped as to what I'm looking for.
... View more