Hi all,
I want the "date" field to be used as timestamp. However, in some of the events this field is missing and so while loading the data, some events go missing. How do I work with the timestamp so that all of my data gets loaded(fields with date field as null are also loaded)?
My event looks like :
{
author: {
id: 631AAF84885D8AA48F4876100F92CEEB
location: New York, New York, United States
num_reviews: 1
username: Danny K
}
date: December 20, 2012
date_stayed: December 2012
id: 147785077
num_helpful_votes: 0
offering_id: 120556
text: We arrived late at night and immediately were treated with everything we needed by NIkki and Adam. We were here for three nights and there wasn't anything we needed that the hotel wasn't able to help us with. They even provide a whole bunch of complimentary travel items you might forget (phone chargers, tooth brushes, contact solution etc.). They also have a very nice restaurant and bar in lobby which was very enjoyable to visit. I highly recommend the George hotel.
title: “Unbelievable customer service”
via_mobile: false
}
Hi Dravebrooking,
Actually some of the event were too big and were getting truncated. Therefore, the field which was used for timestamp was giving me a null value and some of the records were missing. I fixed the problem by adding "TRUNCATE=0" in props.conf. I am now able to see all the events correctly.
Hi Dravebrooking,
Actually some of the event were too big and were getting truncated. Therefore, the field which was used for timestamp was giving me a null value and some of the records were missing. I fixed the problem by adding "TRUNCATE=0" in props.conf. I am now able to see all the events correctly.
Could you provide your current props.conf for this sourcetype?
My understanding is that all events should be indexed irrespective of whether the event contains a field that Splunk can identify as timestamp or not. The indexing process goes through a number of steps to try and identify which date/time to use, the process is described in the documentation. Your 'missing' events may have a timestamp of when the input was indexed by Splunk, which according to the documentation is the timestamp of last resort.
There are also other settings in the conf files that may affect whether indexing can detect your timestamps, many of which are shown in the pages following the above documentation reference.
If the date in your event is null, what timestamp would you like it to be associated with the event? Should it be the time the event was indexed?
Can you also post an example of an event that is missing the date field? Is the date stayed field populated or not?