.kvp file forwarding from the server but not appearing in Splunk Indexer

[RAYUDU_NARA]

Hi,

Suddenly data not appearing in the indexer and dash board but it is forwarding from the server. No configuration changed but earlier it is forwarding and stopped suddenly at end of the month at last hour.

Is there any Indexer configuration limit or what is the issue ?

[richgalloway]

Are you using SSL? That the forwarding stopped at the end of the month makes me think a certificate expired.

[RAYUDU_NARA]

Hi Richgalloway,

How we will check whether SSL expired or not in Splunk forwarder ? If it is expired, how we will get renewal this ? We install forwarder downloading from Splunk.com .

[richgalloway]

If you or your admin did not specifically set up SSL then you are not using it.
That means something else changed at the end of the month. Perhaps someone pushed out a firewall change before leaving on Friday.
Have you checked Splunk's logs? Maybe something is preventing it from writing to the index.

[RAYUDU_NARA]

Hi Richgalloway,

This issue in windows forwarder.

.kvp file forwarding if we place without timestamp in the file.
But same file if we place with timestamp it is forwarding but not appearing in Splunk.
But from the server all Splunk logs, metric logs, system event logs everything appearing. Only script data is not appearing in Splunk indexer.

appearing: number_of_Images=20
not appearing : timestamp="06/07/2017 13:26:05" number_of_Images=20

[richgalloway]

What are your props.conf settings for that file's sourcetype?

[davebrooking]

Mmmm, the events stopped indexing at the end of the month. I suspect there may be a timestamp mismatch and the events for 1st July were indexed with a date of 7th January, the events for 2nd July were indexed with a date of 7th February. Will these events magically start indexing tonight at midnight?

Dave

[RAYUDU_NARA]

Yes Dave, last month same issue. stopped at 31st may midnight 11pm and started at 6th june 12am. But this month stopped at 11pm june but started yet.

[davebrooking]

The documentation covers timestamp recognition. I suspect you will need to specifically define a TIME_FORMAT attribute for this data, so that Splunk can correctly interpret the timestamp in the event.

Dave

[RAYUDU_NARA]

Hi Dave,
Like below we are data forwarding and appearing in Splunk indexer.

data stopped at 11pm end of the month and started at 12am like below.

Feb - 1st no data , from 2nd we have data
Mar - 1st and 2nd no data , From 3rd we have data
April - 1st - 3rd no data , from 4th we have data
May - 1st - 4th no data , from 5th we have data
June - 1st - 5th no data, , from 6th we have data
July - 1st - 6th no data – hope we have data from this mid night 12am

Here how we can fix the issue, this is in Production Environment.

[richgalloway]

I think @davebrooking has hit the nail on the head. Splunk defaults to US date format (mm/dd/yyyy) and is easily confused by dd/mm/yyyy dates. The best fix, which really should be done by everyone for every sourcetype, is to specify a TIME_FORMAT attribute in props.conf.

[RAYUDU_NARA]

Hi richgalloway,

Is the TIME_FORMAT we need to change to mm/dd/yyyy or dd/mm/yyyy ?

my soucename is Import_Count-kvp, so need to place in the props.conf like below,

[Import_Count-kvp]
TIME_FORMAT=mm/dd/yyy

And in one server we don't have source name we are monitoring with Indexer name splunk, so for this props.conf like,

[splunk]
TIME_FORMAT=mm/dd/yyy

[richgalloway]

Use TIME_FORMAT = %d/%m/%Y, assuming your dates are in that format (day/month/year).

[RAYUDU_NARA]

time format is correct but It is not working richgalloway.

[RAYUDU_NARA]

Is this configuration need to be done at forwarder level or Indexer level ?

[richgalloway]

The indexer level.

[RAYUDU_NARA]

Hi Richgalloway,

It is working fine now. Thank you.

Regards,
Rayudu