All Apps and Add-ons

eventgen events stopped being indexed

splunk_zen
Builder

The event generation was flawlessly working for weeks but went fully quiet until a sole burst on yesterday at 4pm
The configuration file was not touched (generation frequency still the same) so what can cause the indexing to stop ?
Enabled the following but can only see the events going into the queue but nothing being indexed

debug = true
verbose = true

Taking a look into eventgen_main (there's nothing in eventgen_error )

2018-08-14 15:06:44 eventgen        INFO     MainProcess Start '2' generatorWorkers for sample 'test_sample.txt'
2018-08-14 15:06:44 eventgen        INFO     MainProcess Worker# 0: Put 50 events in queue for sample 'test_sample.txt' with et '2018-08-14 15:00:44.141400' and lt '2018-08-14 15:06:44.141451'
2018-08-14 15:06:44 eventgen        INFO     MainProcess Worker# 1: Put 50 events in queue for sample 'test_sample.txt' with et '2018-08-14 15:00:44.141400' and lt '2018-08-14 15:06:44.141451'
2018-08-14 15:06:44 eventgen        INFO     MainProcess Worker# 0: Put 50 events in queue for sample 'test_sample.txt' with et '2018-08-14 15:00:44.141400' and lt '2018-08-14 15:06:44.141451'
2018-08-14 15:06:44 eventgen        INFO     MainProcess Worker# 1: Put 50 events in queue for sample 'test_sample.txt' with et '2018-08-14 15:00:44.141400' and lt '2018-08-14 15:06:44.141451'
2018-08-14 15:12:44 eventgen        INFO     MainProcess Start '2' generatorWorkers for sample 'test_sample.txt'

Looking at splunkd,

08-14-2018 15:19:41.588 +0000 INFO  LicenseUsage - type=Usage s="/opt/splunk/var/log/splunk/test_service.log" st=test_service_log h="ip-172-31-36-143" o="" idx="default" i="EAE584D7-DBF7-4B6F-819B-36BAD9EEE258" pool="auto_generated_pool_enterprise" b=84 poolsz=53687091201
08-14-2018 15:19:41.588 +0000 INFO  LicenseUsage - type=Usage s="/opt/splunk/var/log/splunk/test_service.log" st=test_service_log h="ip-172-31-36-143" o="" idx="default" i="EAE584D7-DBF7-4B6F-819B-36BAD9EEE258" pool="auto_generated_pool_enterprise" b=84 poolsz=53687091201
08-14-2018 15:20:42.620 +0000 INFO  LicenseUsage - type=Usage s="/opt/splunk/var/log/splunk/test_service.log" st=test_service_log h="ip-172-31-36-143" o="" idx="default" i="EAE584D7-DBF7-4B6F-819B-36BAD9EEE258" pool="auto_generated_pool_enterprise" b=172203 poolsz=53687091201

Can you give me some pointers on what may be happening ?

kthammireddygar
Path Finder

if nothing changed on eventgen Config, it may be that there is no disk or disk full.

0 Karma

jkat54
SplunkTrust
SplunkTrust

search index=_internal log_level=warn* OR log_level=error see if that reveals more details.

If I had to guess, something could be sending everything to null queue.

Is it by chance "bursting" every time you restart splunk?

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...