How come Windows Security events are taking 15-20 minutes to appear on Splunk?

[spodda01da]

We have configured a universal forwarder on 4 Domain Controllers in our environment.

Now, we receive security events in real time on 3 Domain Controllers. The 4th DC has a lag of around 20 minutes to appear.

I am wondering if anyone has come across this issue or is there any configuration which I might have missed out.

Thanks,

[maraman_splunk]

HI,

causes & solutions could be multiple
check that you are not limiting bandwith (maxkbps=0 or set a value) (see https://docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/Troubleshootingeventsindexingdel... )
also make sure you have evt_resolve_ad_obj = 0 in the input
+ if ever you have some kind of AV software running on the server, make sure you have followed the doc about exclusion of files AND processes for splunk
+ use a recent version of UF and Splunk_TA_windows

there could also be ressources issues on the AD server (ie be at the limit of what the server can log)

[ddrillic]

I usually start with the following to see the indexing time delay (if any) -

<base search> 
| eval diff= _indextime - _time 
| eval diff = diff/60
| table _time diff

[spodda01da]

Thanks, I have been monitoring for couple of hours and see the time difference hovering between 18-28 minutes.

[jbrocks]

If you just configured the Windows Log collection with the TA, it might be possible (depending on your configurations in inputs.conf) that the Windows TA starts indexing from the oldest Windows Events.

e.g. if your inputs.conf includes:

start_from = oldest
current_only = 0

Windows Event Logs can be very large, so it might take some time to index all the old log files. In your case I would just wait for one or two days and that check the latency again. If this is not the problem and you also have problems with other logs latency, it can be also problems with the hardware references: https://docs.splunk.com/Documentation/Splunk/7.2.4/Capacity/Referencehardware but I can only suggest from far. Hope this helps!