Getting Data In

Why is the log time different than the system time?

pdgill314
Path Finder

I have been beating my head against the wall on this one for a few days now. I have tried every suggestion I can find...

I have logs that are being generated in Eastern Time on a server. That server's date config is UTC. My Splunk indexers are in UTC. My timezone for my user is in Eastern Time, yet, the logs always show up 4 hours behind.

Example log:

2018-05-22T13:01:06.882,GMT-04:00 DEBUG "ajp-bio-127.0.0.1-8009-exec-405" []controller.CourseSectionManagementController.processCourseSection() line 70: CourseSectionManagementService Full Request org.apache.catalina.connector.RequestFacade@2ad67b7a 
2018-05-22T13:01:06.973,GMT-04:00 DEBUG "ajp-bio-127.0.0.1-8009-exec-405" [] service.impl.CourseSectionManagementServiceImpl.processDeleteCourseSectionRequest() line 89:  Message to be sent back to ABCD <200,<?xml version="1.0" encoding="UTF-8" standalone="yes"?>

What Splunk sees:

_time 2018-05-22T09:01:06.882-04:00

5/22/18
9:01:06.882 AM  
2018-05-22T13:01:06.882,GMT-04:00 DEBUG "ajp-bio-127.0.0.1-8009-exec-405" [] controller.CourseSectionManagementController.processCourseSection() line 70: CourseSectionManagementService Full Request org.apache.catalina.connector.RequestFacade@2ad67b7a 

_time 2018-05-22T09:01:06.973-04:00

5/22/18
9:01:06.973 AM  
2018-05-22T13:01:06.973,GMT-04:00 DEBUG "ajp-bio-127.0.0.1-8009-exec-405" [] service.impl.CourseSectionManagementServiceImpl.processDeleteCourseSectionRequest() line 89:  Message to be sent back to ABCD <200,<?xml version="1.0" encoding="UTF-8" standalone="yes"?>

Props.conf file:

[Console]
KV_MODE = xml
DATETIME_CONFIG = NONE
TZ = America/New_York
TZ_ALIAS = GMT=UTC
TIME_FORMAT =  %Y-%m-%dT%H:%M:%S.%Q,%Z%:z
MAX_TIMESTAMP_LOOKAHEAD = 36
BREAK_ONLY_BEFORE_DATE = true

inputs.conf:

[monitor:///apps/tomcat-instances/*/logs/console/Console.log]
ignoreOlderThan = 7d
sourcetype = Console
index = sis
disabled = 0

Any suggestions on what I might be missing?

Thank you.

0 Karma
1 Solution

somesoni2
SplunkTrust
SplunkTrust

I would make following changes to props.conf (which goes on Indexer/Heavy forwarder whichever comes first in the data flow),

[Console]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?=\d{4}-\d{2}-\d{2}T)
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N,GMT%:z
MAX_TIMESTAMP_LOOKAHEAD = 34

Your raw data has time zone in them, so no need to explicitly set it. Your data is not XML so, removing KV_MODE attribute there. (plus it should be set on search heads).

View solution in original post

xpac
SplunkTrust
SplunkTrust

Just to add, if Splunk recognizes a timezone information in the timestamp, any TZ = setting will be ignored/overruled.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

I would make following changes to props.conf (which goes on Indexer/Heavy forwarder whichever comes first in the data flow),

[Console]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?=\d{4}-\d{2}-\d{2}T)
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N,GMT%:z
MAX_TIMESTAMP_LOOKAHEAD = 34

Your raw data has time zone in them, so no need to explicitly set it. Your data is not XML so, removing KV_MODE attribute there. (plus it should be set on search heads).

pdgill314
Path Finder

Thank you for replying. Giving that a shot now.

I tried adding what I had for the props.conf onto the indexers, and it worked to correct the time, but merged three events into one, so hopefully your suggestion should help.

There is XML data there but it is in a SOAP envelope, e.g.:

   ...
            <imsx_codeMajor>success</imsx_codeMajor>
            <imsx_severity>warning</imsx_severity>
            <imsx_description>success</imsx_description>
        </imsx_statusInfo>
    </ns2:local>
    ,{}>  

Thanks.

0 Karma

pdgill314
Path Finder

Ended up using this in the props.conf on my indexers:

[Console]
SHOULD_LINEMERGE = false
TZ = America/New_York
TZ_ALIAS = GMT=UTC
LINE_BREAKER = ([\r\n]+)(?=\d{4}-\d{2}-\d{2}T)
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%Q,%Z%:z
MAX_TIMESTAMP_LOOKAHEAD = 34

Looks like it is working correctly now.

Thanks for your help!

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Since your data is not pure XML (has some non-xml strings as well), the KV_MODE=xml would not work (which should be set on Search Heads).

0 Karma

pdgill314
Path Finder

Ah. Ok. Thanks for that.

Should I even have a Props.conf for this app on the Universal Forwarder (server generating logs)?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...