All Apps and Add-ons

The app is indexing event, before the TMG has write out the line of the event completely

shbagautdinov
Path Finder

Hello Mikael! Thank you for your Add-on!
Hello Guys,
I have installed an universal forwarder on my TMG 2010 Server, and I have configured to input log files from a local directory on this server. The logs are writting to this local folder by TMG constantly.
I see fresh events in my splunk web-interface. But some of events are corrupted and the consists of part of a line.
I think the root cause of this issue in indexing the events before TMG has finished to write event completely.
What setting I should change to fix this problem?

0 Karma
1 Solution

woodcock
Esteemed Legend

shbagautdinov
Path Finder

Hi, woodcock!
Thank you for your knowledge!

0 Karma

shbagautdinov
Path Finder

Hi,
1) yes, I have copied it to apps folder and I have create "local" folder with inputs.conf

## Forefront TMG Firewall logs
## Modify paths to fit your needs 
[monitor://C:\Program Files\Microsoft Forefront Threat Management Gateway\Logs\*_FWS_*.w3c] 
sourcetype = microsoft:forefront:tmg:fw 
disabled=0 
index=forefront_tmg

## Forefront TMG Proxy logs
## Modify paths to fit your needs
[monitor://C:\Program Files\Microsoft Forefront Threat Management Gateway\Logs\*_WEB_*.w3c] 
sourcetype = microsoft:forefront:tmg:proxy 
disabled=0
index=forefront_tmg

2) yes, I have copied it to apps folder. with default settings

I have another test deployment and for a test I have inputed the copied big log file of the yesterday to my test splunk instance. And all events are correct and field extraction works fine.

In my main installation which is indexes hot log files from TMG it is about 99.5% of events are correct and field extraction work. But there are 0.5 % of events which are partialy indexed and field extraction not work correctly.
How can I get 100% of events will be indexed correct on my main instance?

0 Karma

mikaelbje
Motivator

Ok, thanks for checking that out.

What Splunk version are you on?

Regarding the partially indexed events - are you sure these are not just the headers or the #Fields line from the beginning of the file?

You may also try to use MonitorNoHandle:// instead of monitor://
In this case you must specify a single file, not a path nor wildcards.

You may also test the "time_before_close" parameter in your inputs.conf: https://docs.splunk.com/Documentation/Splunk/6.5.1/Admin/Inputsconf

There are probably a few other things to try as well. Maybe someone else will chime in with some tips.

shbagautdinov
Path Finder

Hi, Mikael,
Fantastic! It Works!
I have added time_before_close=60 into inputs.conf and now all events are indexing correctly!
Thank you for your time!

0 Karma

mikaelbje
Motivator

Hi,

I haven't seen this before.

  1. Did you push the TA-Microsoft_Forefront_TMG add-on to the Forwarder in question?
  2. Did you install the TA-Microsoft_Forefront_TMG add-on on your Indexers?

Both are a requirement.

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...