Getting Data In

Can you help me avoid data loss from my universal forwarder?

dailv1808
Path Finder

Hi Splunker!

i am using a universal forwarder to monitor and forward data (log file) to my Splunk. i have observed a loss of data...I.e. certain events are missing.

Periodic data loss

What is recommended to avoid any data loss during this period?

is there is any way to acknowledge whether complete data is being sent by the forwarder ??

0 Karma
1 Solution

inventsekar
Ultra Champion

Hi, the image shows some gaps in logs transmission, because, the UF to indexer can not / need not send logs "all" the times (the logs thruput can vary depending on many factors like network connectivity, how busy the UF/HF/indexers are, etc.. )
may we know how you found out that there are data loss? just by this pic or you really missed logs when you run search queries?!?! is the UF sending lots of logs / how much daily license volume from this UF per day?!?!

View solution in original post

0 Karma

ddrillic
Ultra Champion

Splunk's best practices is to enable load balancing and indexer acknowledgement.

0 Karma

inventsekar
Ultra Champion

Hi, the image shows some gaps in logs transmission, because, the UF to indexer can not / need not send logs "all" the times (the logs thruput can vary depending on many factors like network connectivity, how busy the UF/HF/indexers are, etc.. )
may we know how you found out that there are data loss? just by this pic or you really missed logs when you run search queries?!?! is the UF sending lots of logs / how much daily license volume from this UF per day?!?!

0 Karma

dailv1808
Path Finder

Hi,
I checked the log file in the directory of the host installed UF. Some events have not been collected to splunk.
Data generated on UF is very large and continuous. I don't know exactly how much per day.

0 Karma

inventsekar
Ultra Champion

on splunkd.log at the UF, do you see any errors?!

0 Karma

dailv1808
Path Finder

I see following INFO notice in splunkd.log
"INFO ThruputProcessor - Current data throughput (262 kb/s) has reached maxKBps. As a result, data forwarding may be throttled. Consider increasing the value of maxKBps in limits.conf."

Then, i set maxKMps=unlimit, the problem have solved.
But I dont know why. Can y explain for me?
More info: The UF is monitoring the file is rotate. filename.log have new content added to them and rotate around 5-10 minute to filename.log.1, filename.log.2, vv...

0 Karma

inventsekar
Ultra Champion

As you are a new user to Splunk Answers, you can upvote the answers/comments,
if this answer resolved your query, you can select this answer and "accept" it as the answer, so that this question will be moved to answered queue. Happy Splunking!

pls check the info from limits.conf file

[thruput]
maxKBps = 
* The maximum speed, in kilobytes per second, that incoming data is 
  processed through the thruput processor in the ingestion pipeline.
* To control the CPU load while indexing, use this setting to throttle
  the number of events this indexer processes to the rate (in
  kilobytes per second) that you specify.
* NOTE:
  * There is no guarantee that the thruput processor 
    will always process less than the number of kilobytes per
    second that you specify with this setting. The status of 
    earlier processing queues in the pipeline can cause
    temporary bursts of network activity that exceed what
    is configured in the setting. 
  * The setting does not limit the amount of data that is 
    written to the network from the tcpoutput processor, such 
    as what happens when a universal forwarder sends data to 
    an indexer.  
  * The thruput processor applies the 'maxKBps' setting for each
    ingestion pipeline. If you configure multiple ingestion
    pipelines, the processor multiplies the 'maxKBps' value
    by the number of ingestion pipelines that you have
    configured.
  * For more information about multiple ingestion pipelines, see 
    the 'parallelIngestionPipelines' setting in the 
    server.conf.spec file.
* Default (Splunk Enterprise): 0 (unlimited)
* Default (Splunk Universal Forwarder): 256

vishaltaneja070
Motivator

Hello @dailv1808

Try using Indexer Acknowledge to avoid data lose. please find the below link for your reference:

https://docs.splunk.com/Documentation/Forwarder/7.2.3/Forwarder/Protectagainstthelossofin-flightdata
0 Karma

dailv1808
Path Finder

Thank for your reply.
I want to know the exact source of the problem when I do not use the Index ACK. Why lost data?
Did Forwarder get too much data, cause a queue to overflow make lost data? OR other reason vv....

0 Karma

vishaltaneja070
Motivator

Hello @dailv1808

You will get this full information in splunkd.log on Universal Forwarder. The location of log is \var\log\splunk

Please accept the answer if the above solution worked for you.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...