Getting Data In

Very strange issue with timestamp's from a scripted input

joshd
Builder

Hoping someone can shed some light on this issue...

I just upgraded my lightweight forwarders to the new universal forwarder and across four of the forwarders I have two scripted inputs that produce the same thing on each forwarder. However I noticed in my logs that one of the scripted inputs on a specific forwarder is creating log entries in splunkd.log like below every time it runs:

07-19-2011 11:13:34.292 -0400 WARN  DateParserVerbose - A possible timestamp match (Fri Sep  2 03:32:16 2005) is outside of the acceptable time window. If this timestamp is correct, consider adjusting MAX_DAYS_AGO and MAX_DAYS_HENCE.  Context="source::/opt/splunkforwarder/etc/system/bin/logger.sh fas_batch_app 8180|host::db1|stats|remoteport::54811"
07-19-2011 11:14:34.295 -0400 WARN  DateParserVerbose - A possible timestamp match (Fri Sep  2 03:32:16 2005) is outside of the acceptable time window. If this timestamp is correct, consider adjusting MAX_DAYS_AGO and MAX_DAYS_HENCE.  Context="source::/opt/splunkforwarder/etc/system/bin/logger.sh fas_batch_app 8180|host::db1|stats|remoteport::54811"
07-19-2011 11:15:34.294 -0400 WARN  DateParserVerbose - A possible timestamp match (Sun Jun 19 19:30:08 2005) is outside of the acceptable time window. If this timestamp is correct, consider adjusting MAX_DAYS_AGO and MAX_DAYS_HENCE.  Context="source::/opt/splunkforwarder/etc/system/bin/logger.sh fas_batch_app 8180|host::db1|stats|remoteport::54811"
07-19-2011 11:16:34.295 -0400 WARN  DateParserVerbose - A possible timestamp match (Sun Jun 19 19:30:08 2005) is outside of the acceptable time window. If this timestamp is correct, consider adjusting MAX_DAYS_AGO and MAX_DAYS_HENCE.  Context="source::/opt/splunkforwarder/etc/system/bin/logger.sh fas_batch_app 8180|host::db1|stats|remoteport::54811"
07-19-2011 11:17:34.296 -0400 WARN  DateParserVerbose - A possible timestamp match (Tue Aug 30 02:43:12 2005) is outside of the acceptable time window. If this timestamp is correct, consider adjusting MAX_DAYS_AGO and MAX_DAYS_HENCE.  Context="source::/opt/splunkforwarder/etc/system/bin/logger.sh fas_batch_app 8180|host::db1|stats|remoteport::54811"

...Now this scripted input is set to run every 60 seconds and as mentioned every time it runs it generates these messages. The time on the forwarder and the receiver are sync'd to the same NTP server, so no issues there. It is not reading from any file, the scripted input only generates a single line to standard out with no date stamp like so:

[dir:splunkforwarder]# /opt/splunkforwarder/etc/system/bin/logger.sh fas_batch_app 8180
fas_batch_app : ActiveThreadCount=208 MaxMemory=1908932608 TotalMemory=1130823680 ActiveThreadGroupCount=10 FreeMemory=568403320

So now I cannot figure out where it is getting this way back timestamp from, and as you can see with each logged message the timestamp is totally out of whack, one logged message it says "Sep 2, 2005" then the next time it runs and logs a message the timestamp is "Jun 19, 2005". I would think the forwarder/receiver is somehow tagging the message incorrectly, but I've looked at other events from this same forwarder with another scripted input that generates a single line to stdout and it has no issues and logs no messages.

Has anyone seen anything like this? Where should I start?

Thanks in advance!

mslvrstn
Communicator

This answer is probably no longer of relevance to you, but perhaps some other reader will stumble upon it as I did.

The problem is that you don't have a timestamp in your scripted output, so Splunk goes looking for one.
Both values for MaxMemory and TotalMemory appear to Splunk to be valid epoch-relative dates, but I'm guessing that 1908932608, which maps to 29 Jun 2030 03:03:28 GMT is too far in the future.

Then it sees 1130823680, which maps to 01 Nov 2005 05:41:20 GMT
Every time your script runs, there is likely a slightly different value for TotalMemory so that computed time is going to jump around, as you've experienced.

One way around it would be to add an explicit timestamp at the beginning of the line of output.
Another would be to disable the timestamp processor for this input so that it would get a timestamp that matched the time the input was received

DATETIME_CONFIG=NONE

See Disable timestamp processor for more info on the latter.

MuS
SplunkTrust
SplunkTrust

thanks for this!
just solved my problem with scripted inputs result are being indexed with a wrong timestamp. added current time stamp at first output line and everything is good now 😃

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...