Getting Data In

Why is the timestamp not recognized for one particular event from our Microsoft TMG Proxy Logs?

btiggemann
Path Finder

Hi Splunkers,

I have a strange problem with Microsoft TMG, Splunk can't find the time stamp on one particular event. On all other events the time stamp is running fine.

For this event Splunk finds the time stamp automaticly:

172.31.192.191, anonymous, -, -, 11/10/2014, 16:19:36, -, KST032, -,
172.31.209.144, 172.31.209.144, 443, 1, 59, 3391, https, -, GET, http://172.31.209.144/, -, -, 12202,
-, [Enterprise] Standardregel, Req ID: 0b150cc4; Compression: client=No` server=No` compress rate=0% decompress rate=0% ; FBA cookie: exists=no` valid=no` updated=no` logged off=no` client type=unknown` user activity=yes, Internal, Local Host, 0x0, Denied, -, -, -, -, -, -, -, -,
-, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, Web Proxy, 172.31.209.144, 35716, -

For this event Splunk is not able to find the time stamp:

172.21.29.67, VERWALTUNG\user, Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML` like Gecko) Chrome/38.0.2125.111 Safari/537.36, -, 11/10/2014, 16:19:38, -, KST032, http://webradio.ffh.de/top40, 172.31.209.226, 172.31.209.226, 9090, 62, 648, 1003, http, -, POST, http://webradio.ffh.de/custom/getAllSonginfos.php, application/json, Upstream, 200, -, HTTP und HTTPS erlaubt, Req ID: 0b150ca5; Compression: client=No` server=No` compress rate=0% decompress rate=0%, Internal, Internal, 0x580, Allowed, 11/10/2014 15:19:38, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, None, webradio.ffh.de, 56419, -

Because of that issue the line breaking is going wrong and some events are combined together which is wrong.

We tried to use a REGEX that takes out the time stamp after the 4th comma, but it was not successful because there's also a comma between date and time.

I thought there's a problem with the "`" character in the user agent field. This is not the issue, I have tried to remove this character in one log event, but the time stamp problem is still there.

Does any body see a difference between the logs? What could be the issue?

I would be very thankful if you can help me.

Best regards

Tags (2)
0 Karma

tom_frotscher
Builder

Hi, i think this might be caused by the following configuration default:

MAX_TIMESTAMP_LOOKAHEAD

This is the configuration, how far splunk is seaching in the event to find a timestamp and it defaults to 150 characters. If you count the characters to the end of your date in the event where the timestamp is not correctly recognized, you exceed the 150 chars.

Can you try to set this value up, maybe to 250? You should add this to the stanza of your sourcetype in the props.conf.

Greetings

Tom

btiggemann
Path Finder

OH yes, we will try this. I have seen this value but I haven't counted the characters... Thanks a lot

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...