Metadata showing wrong last Indexed time

[moesaidi]

I have a query that runs once a day to tell me if certain source types have no data coming in after X time.
The query has been working fine for some time; but recently has started alerting me that a single source type hasn't been updated in over a week.
When I check the source type, I do see logs being ingested, updates happening as expected, etc..

Any ideas why the metadata isn't showing the right information? or perhaps I'm looking at this the wrong way:

| metadata type=sourcetypes
| where recentTime < now() - 86400
| eval LastIndex = strftime(recentTime, "%F %T") 
| rename sourcetype AS "Source Type" LastIndex AS "Data Last Ingested"
| table "Source Type" "Data Last Ingested"

[woodcock · Accepted solution]

You are looking at it wrong. See here:
https://docs.splunk.com/Documentation/SplunkCloud/latest/SearchReference/Metadata

It says (in part) this:

The "firstTime" field is the timestamp for the first time that the indexer saw an event from this host.
The "lastTime" field is the timestamp for the last time that the indexer saw an event from this host.
The "recentTime" field is the indextime for the most recent time that the index saw an event from this host. In other words, this is the time of the last update.

The metadata command is essentially a macro around tstats. For the clueful, I will translate:

The "firstTime" field is "min(_time)".
The "lastTime" field is "max(_time)".
The "recentTime" field is "max(_indextime)".

These can be broken out BY host/sourcetype/index as well as Indexer ( splunk_server ).

[moesaidi]

Thank you!

But with what you've explained, and what that document references, it means recentTime should show me the indextime of the most recent time that the index saw an event from a host.

The results of my query show that the last indexed event, then, for sourcetype XYZ is Aug 12, but when I look for data from that sourcetype, I see it indexing data on average every minute 24/7, latest being the time of the search.

Would we not expect recentTime to update and reflect the most recent time the indexer saw data for that sourcetype?

If I'm still misunderstanding I do apologize..

[woodcock]

You are assuming that the data is in Splunk correctly; that is a HUGE assumption. The problem is probably that you are timestamping events wrong. It is not only perfectly possible, but HIGHLY common for Splunk to be too-lightly configured such that the indexers have to guess at timestamps and timezones (instead of being told everything exactly) such that events are mis-timestamped, sometimes many hours into the past or even into the future. A first pass at examining this problem is to get the Meta woot! app. But it is a huuuuuuuuuuuuge undertaking to get all of this fixed if you have been Splunking a long time and are just now peeking into the kitchen to watch the sausage maker (we can definitely help you there so PM me).

[moesaidi]

Thanks again!
I'm still not 100% convinced though. We've had this query running daily for months, if not years, and has always accurately shown the recentTime as we would expect it.
The data being ingested does appear to be logging timezones and timestamps properly (and confirmed configured timezones properly as well), in relation to the indexed time VS event time.
This leads me to believe the metadata is not being updated perhaps? What is odd is that it's happening to only one sourcetype out of 30+ source types.

I'll check out the app and see if it can help examine the problem and I appreciate your help! It gives me some good direction I think on where to start!

[woodcock]

Just take the bad time that it is showing you and do

index=* OR index=_* sourcetype=BadSourceType earliest=<recentTimeHere> latest=<recentTimeHere> and take a GOOOOOOOD hard look at that event.  I suspect that you will find that it is broken.

[moesaidi]

We just had maintenance done on our indexer this past weekend, and after a reboot - it appears to have resolved this.
The most recent indexed time is now accurate.

Still no idea what caused it as the event from that time is /was right so it is odd!