Splunk Forwarder only sends a few Data from monitored logfile

[toolanalyzer]

Greetings,

I using a Splunk-Enterprice trail version and installed Forwarders on 3 different VMs. The Indexer Server is running on its own VM.
After i configure the forwarders, they worked fine for a few days but since one day the forwarders send only a few entries from the configured logfiles, although the files have many more entries.

The connection to the indexer server works fine and also the splunkd logs on the forwarders do not return any error messages.
The forwarder's metric log sends permanent entries, so there is no connection problem to suspect.

I haven't found any other issues where the forwarder works but sends only a few log entries, so I hope you can help me with this.

[toolanalyzer]

Hi pyro_wood,

thank you for your answer. The log files are Log4j webapp logs from a tomcat server application and the entries look like this one:

03-11 14:23:48,601 [pool-15-thread-1][] DEBUG ion.communication.AbstractHTTPRequestExecutor (   ) - Received response: HTTP/1.1 200 OK

The first time I used Splunk it worked without any problems and logging hasn't changed since then, so I can't imagine it would be a problem with line breaking.

The following picture shows the incoming log entries of the last 4 hours on the Splunk Index Server:

The gaps are up to 1 hour, in which no entries were received and should not be present since the log file is very busy and writes several log entries in every minute. But as you can see, sometimes the Indexer reseive

[alemarzu]

Hi there @toolanalyzer

What about the new events, are they getting indexed properly ?

[toolanalyzer]

Hi Alemarzu, when events arrive at the index server they are properly indexed. But of e. g. 10,000 events per hour only 1000 events arrive at the server. It feels like the forwarder works but only runs for a few minutes per hour.

[alemarzu]

useACK is enabled in the outputs.conf on your UF ?

[horsefez]

Hi toolanalyzer,

I like that you did troubleshoot this before posting a question on answers, which helps us in understanding your problem.
Unfortunately for me it is difficult to understand your situation. It would be awesome if you could provide maybe some sample data ... like how the logs look like you are going to ingest.
Thank you.

PS: I'm going to make one assumption that this could be a problem due to proper Line Breaking

[toolanalyzer]

Hey pyro_wood, it is log4j logs from a webapp log of a tomcat application server and the entries look like this:

03-11 14:39:19,922 [pool-15-thread-86][] INFO  es.EnergyConcernsLastServiceAccessServiceImpl (   ) - Utilize AuditTrail regarding EnergyConcerns

The timeline for the last 4 hours on the Splunk UI look like this:

There are gaps where no entries be received for about 1 hour although the log file writes several entries per minute.