Getting Data In

Any idea why a particular sourcetype would stop showing data after 10/31/16 23:59:59?

tbourne
Engager

Here are some pieces of info that may be relevant:

  • The sourcetype in question shows no data after midnight on October 31st when searching
  • Setup: 1 Splunk server, no replication or anything, 40 nodes (4 in question) reporting to splunk
  • Splunk server version: 6.5.0 (installed on 10/25/16)
  • Splunk server has more than 20% free disk on all drives
  • All other sourcetypes on the splunk server are working, even sourcetypes that dump into the same index as the broken sourcetype
  • No changes have been made on the splunk server or the nodes
  • The metrics.log on the splunk server shows data coming in for that sourcetype from the 4 nodes in question
  • The splunk forwarder logs on each of our nodes shows that it is sending data into the splunk server using the sourcetype
  • Each node in question is reporting data to the splunk server on additional sourcetypes and they are searchable in splunk

Any ideas would be greatly appreciated. I've restarted the splunk server (full windows reboot) as well as restarting the splunk forwarder on the nodes. It seems like it's date related but none of the other sourcetypes seem to be affected.

0 Karma
1 Solution

mattymo
Splunk Employee
Splunk Employee

My wild stab in the dark, based on the horrors of experience, is that the props.conf does not have timestamp format explicitly set, and maybe what we want to be Nov 1, 2016, is showing up in January 11th, 2016?

Can you share your props config for this sourcetype?

- MattyMo

View solution in original post

mattymo
Splunk Employee
Splunk Employee

My wild stab in the dark, based on the horrors of experience, is that the props.conf does not have timestamp format explicitly set, and maybe what we want to be Nov 1, 2016, is showing up in January 11th, 2016?

Can you share your props config for this sourcetype?

- MattyMo

tbourne
Engager

That would be awesome. Here's a sample line:

04-11-2016 09:15:29,482|http-apr-8080-exec-3| INFO|ConfigFilters||sessionManagement/template?name=portal/Login built in 0.0151612 seconds ().
0 Karma

mattymo
Splunk Employee
Splunk Employee

Add Data Wizard FTW!

[ tbourne ]
SHOULD_LINEMERGE=false 
NO_BINARY_CHECK=true
TIME_FORMAT=%d-%m-%Y %H:%M:%S,%f
TIME_PREFIX=^
TZ=UTC
MAX_TIMESTAMP_LOOKAHEAD=30

This should clean up your timestamping and get you some perf gains. Best practice is to set linebreaker and time formatting params explicitly on all sourcetypes, as making splunk auto discover makes it work harder and ensures you avoid this pain in the future.

http://docs.splunk.com/Documentation/Splunk/6.5.0/Data/Configuretimestamprecognition

ADD DATA WIZ FTW!

- MattyMo
0 Karma

tbourne
Engager

mmodestino,

You're exactly right! The time format in the logs is dd-mm-yyyy and it would appear that splunk is expecting mm-dd-yyyy. I am a bit puzzled as to why it chose November to do this (as opposed to July or October or whatever). Maybe it's a different default in splunk 6.5.0 versus the 6.4.x and previous. When I look back in January I see interlaced data for 11-01-2016 and 01-11-2016.

From props.conf:

[application_log]
NO_BINARY_CHECK = 1
maxDist = 75
pulldown_type = 1
REPORT-myname=applicationlogmap

[applicationlogmap]
DELIMS="|"
    FIELDS="TimeStamp","ThreadID","LogLevel","ClassName","LogHash","Msg"

I'll read up on how to specify the time/date format and get this resolved. Thanks so much for taking a wild stab! I'll mark your answer as accepted as soon as I verify this works.

0 Karma

mattymo
Splunk Employee
Splunk Employee

My pleasure!

If you share a sample event I can assist with the config.

- MattyMo
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...