Splunk Search

Why are my access_logs inconsistent?

zadenaji
Explorer

My access_logs files are not being pulled constantly. There are large gaps between the pulling of logs.

The logs are being updated within the server path (timestamp shows this), but they are not all being pulled. The source files being pulled tends to be inconsistent; on some occasions it will pull all the proper source files, on other occasions, it will pull 1-2 source files (all from same path).
![alt text][1]
I attached an image as well (for visual reference).

By the way, our inputs.conf file has a very basic setup:
[monitor:path*.log]
sourcetype =log
index = name

Any help would be greatly appreciated.

alt text

0 Karma

hettervik
Builder

Hi,

Normally applications have more traffic during normal working hours due to employees and/or customers actually using the applications and related infrastructure at these times. I can see from your screenshot that the peaks in your log retrieval are indeed approximately during the normal working hours. 14. Oct and 15. Oct is a Saturday and a Sunday (weekend), so it makes sense that there is less activity during these days.

If this doesn't explain the behavior, perhaps check the index time lag of the logs, as explained in the following thread.

https://answers.splunk.com/answers/48731/determining-logging-lag-and-device-feed-monitoring.html

cpetterborg
SplunkTrust
SplunkTrust

As @hettervi has said, this seems normal. You don't have gaps, you just have less data during certain hours.. The screenshot that you provided shows data at all hours across the time period of the search. It is probably just the nature of the business days as he has suggested.

zadenaji
Explorer

@cpetterborg + @hettervi thank you both for your answers. I think the photo I initially attached was not the best example. I attached another one (screenshot-4) that gives a better explanation. I will look into that article you attached @hettervi.

Thanks again

0 Karma

hettervik
Builder

Have you checked the index time lag? I linked to another discussion on index time lag in my original answer. If the index time lag of your logs is consistent and low, it is probably your logging system that is inconsistent, not Splunk. Might be worth to check out?

cpetterborg
SplunkTrust
SplunkTrust

I would look at whether or not new data is actually being sent to Splunk. There are a couple of things I'd look at.

  1. restart the forwarder - probably not the case, but it is a quick thing to do
  2. check on the rotation of the log data and see if the log may be waiting for there to be data beyond the end of the data pointer that the forwarder had for the file. If the forwarder thinks that it doesn't need to send more data because the logrotate didn't do something that it should, or the CRC of the first 256 bytes is the same, it's going to think that the file is the same file that was there before the rotate and it's going to wait to see new data beyond the pointer in the fishbucket.

Gaps in the data are not going to be that it just doesn't send for a short time and them pick up at some time in the future without filling in the data between the connect times. Splunk wants to finish the job it started. So my guess would be something along the lines of #2 above.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...