Splunk Search

Incorrect Event Date Issue

usdreamz
New Member

We have Splunk free version protected by IBM Tivoli Access Manager. SPlunk indexes the access logs from access manager.
There are no logs in the system before Sep 2013 since system is just implemented.
Whenever I run a search in Splunk for events e.g. from Feb 2013 onwards the my access gets logged in access manager log with following string

splunk/en-US/app/search/flashtimeline?q=search%20*&earliest=1360573200&latest=1384074000

Splunk indexes this as event occurred in Feb 2013 (as per my example above) and show this under Feb 2013 events while the actual timestamp in the log is todays date . Why Splunk is treating the above as Feb 2013 event and how to fix this issue?

Tags (2)
0 Karma

usdreamz
New Member

Thanks for quick response. Please see more information below.

The raw log in the access manager is mentioned below. Every event / log in the access manager starts with and ends with

The event log in access manager for the search performed in Splunk

 <event rev="1.2">
    <date>2013-12-11-21:16:04.828-09:00I-----</date>
    <outcome status="0">0</outcome>
    <component rev="1.2">http</component>
    <event_id>xxx</event_id>
    <action>xxx</action>
    <location>accessmanagerserver</location>
    </originator>
    <accessor name="">
    <user_location>xxxxx</user_location>
    <user_location_type>xxxx</user_location_type>
    </accessor>
    <target resource="5">
    <object>/splunk/en-US/api/search/jobs/1386828913.220/summary?min_freq=0.5&#x0026;earliest_time=1233478800&#x0026;latest_time=1235898000&#x0026;output_time_format=%Y-%m-%dT%H:%M:%S.%Q%z&#x0026;_=1386828964438</object>
    <object_nameinapp>/splunk/en-US/api/search/jobs/1386828913.220/summary?min_freq=0.5&amp;earliest_time=1233478800&amp;latest_time=1235898000&amp;output_time_format=%25Y-%25m-%25dT%25H%3A%25M%3A%25S.%25Q%25z&amp;_=1386828964438</object_nameinapp>
    </target>
    <resource_access>
    <action>httpRequest</action>
    search/jobs/1386828913.220/summary?min_freq=0.5&amp;earliest_time=1233478800&amp;latest_time=1235898000&amp;output_time_format=%25Y-%25m-%25dT%25H%3A%25M%3A%25S.%25Q%25z&amp;_=1386828964438
    <method>xxxxx</method>
    <response>xxxx</response></resource_access>
    <data>
    GET ?min_freq=0.5&#x0026;earliest_time=1233478800&#x0026;latest_time=1235898000&#x0026;output_time_format=%25Y-%25m-%25dT%25H%3A%25M%3A%25S.%25Q%25z&#x0026;_=1386828964438 
    search/flashtimeline?auto_pause=true&#x0026;q=search%20host%3D%22webseal2%22
    </data>
    </event>

Please see the event parsed and indexed by Splunk. I am not sure why only part of the event is parsed here. This behavior is only observed for the searches performed in Splunk and logged in access manager logs and indexed by Splunk. The access logs for other applications in access manager are indexed by splunk as well and it works well in the above format (i.e. complete event with start and end with event tag). Why Splunk is parsing / filtering only some part of the complete event?

GET min_freq=0.5&#x0026;earliest_time=1233478800&#x0026;latest_time=1235898000&#x0026;output_time_format=%25Y-%25m-%25dT%25H%3A%25M%3A%25S.%25Q%25z&#x0026;_=1386828964438 

search/flashtimeline?auto_pause=true&q=search%20host%3D%22webseal2%22


Prop file has following properties

BREAK_ONLY_BEFORE = <event rev="1.2">
BREAK_ONLY_BEFORE_DATE = True
MAX_EVENTS = 256
MUST_BREAK_AFTER = </event>
0 Karma

Lucas_K
Motivator

Your events are not breaking correctly and as such the timestamp is not being extracted properly either.

Try something like this in your props.conf

BREAK_ONLY_BEFORE=
MAX_TIMESTAMP_LOOKAHEAD=35
NO_BINARY_CHECK=1
SHOULD_LINEMERGE=true
TIME_FORMAT=%Y-%m-%d-%H:%M:%S.%3N
TIME_PREFIX=

0 Karma

Lucas_K
Motivator

What do your raw full events look like from access manager?

If its plain text like you provided without timestamps, then reason is due to splunk guessing that the url epoch time is a timestamp (which it is not!).

Do you have a props configured for this source type?

I suggest you revisit how those logs are being parsed. Easiest way is to take a sample and put it through the gui data input (manager/settings -> data inputs -> add data -> choose your sample file). Play with your data inside that until you get the timestamp extraction right.

I'm guessing if what you provided is the actual raw event you'll need to use something like DATETIME_CONFIG=current to add the time at which the event was seen.

0 Karma

lukejadamec
Super Champion

That is odd. The event you posted looks like an event that is recorded in the _internal index.

Does your Splunk Access Role include _internal as one of the default search indexes?

0 Karma

usdreamz
New Member

index="main" hold these events.

0 Karma

lukejadamec
Super Champion

Which index holds this event?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...