How to write regex to identify and use current timestamp for event if the timestamp field in data is missing?

[saileec]

Hi all,

I want the "date" field to be used as timestamp. However, in some of the events this field is missing and so while loading the data, some events go missing. How do I work with the timestamp so that all of my data gets loaded(fields with date field as null are also loaded)?

My event looks like :

{ 
    author: {  
      id:  631AAF84885D8AA48F4876100F92CEEB 
      location:  New York, New York, United States 
      num_reviews:  1 
      username:  Danny K 
   } 
    date:  December 20, 2012 
    date_stayed:  December 2012 
    id:  147785077 
    num_helpful_votes:  0 
    offering_id:  120556 
    text:  We arrived late at night and immediately were treated with everything we needed by NIkki and Adam. We were here for three nights and there wasn't anything we needed that the hotel wasn't able to help us with. They even provide a whole bunch of complimentary travel items you might forget (phone chargers, tooth brushes, contact solution etc.). They also have a very nice restaurant and bar in lobby which was very enjoyable to visit. I highly recommend the George hotel. 
    title:  “Unbelievable customer service” 
    via_mobile:  false 
}

[saileec · Accepted solution]

Hi Dravebrooking,

Actually some of the event were too big and were getting truncated. Therefore, the field which was used for timestamp was giving me a null value and some of the records were missing. I fixed the problem by adding "TRUNCATE=0" in props.conf. I am now able to see all the events correctly.

[somesoni2]

Could you provide your current props.conf for this sourcetype?

[davebrooking]

My understanding is that all events should be indexed irrespective of whether the event contains a field that Splunk can identify as timestamp or not. The indexing process goes through a number of steps to try and identify which date/time to use, the process is described in the documentation. Your 'missing' events may have a timestamp of when the input was indexed by Splunk, which according to the documentation is the timestamp of last resort.

There are also other settings in the conf files that may affect whether indexing can detect your timestamps, many of which are shown in the pages following the above documentation reference.

If the date in your event is null, what timestamp would you like it to be associated with the event? Should it be the time the event was indexed?

Can you also post an example of an event that is missing the date field? Is the date stayed field populated or not?