All Apps and Add-ons

eventgen events stopped being indexed

splunk_zen
Builder

The event generation was flawlessly working for weeks but went fully quiet until a sole burst on yesterday at 4pm
The configuration file was not touched (generation frequency still the same) so what can cause the indexing to stop ?
Enabled the following but can only see the events going into the queue but nothing being indexed

debug = true
verbose = true

Taking a look into eventgen_main (there's nothing in eventgen_error )

2018-08-14 15:06:44 eventgen        INFO     MainProcess Start '2' generatorWorkers for sample 'test_sample.txt'
2018-08-14 15:06:44 eventgen        INFO     MainProcess Worker# 0: Put 50 events in queue for sample 'test_sample.txt' with et '2018-08-14 15:00:44.141400' and lt '2018-08-14 15:06:44.141451'
2018-08-14 15:06:44 eventgen        INFO     MainProcess Worker# 1: Put 50 events in queue for sample 'test_sample.txt' with et '2018-08-14 15:00:44.141400' and lt '2018-08-14 15:06:44.141451'
2018-08-14 15:06:44 eventgen        INFO     MainProcess Worker# 0: Put 50 events in queue for sample 'test_sample.txt' with et '2018-08-14 15:00:44.141400' and lt '2018-08-14 15:06:44.141451'
2018-08-14 15:06:44 eventgen        INFO     MainProcess Worker# 1: Put 50 events in queue for sample 'test_sample.txt' with et '2018-08-14 15:00:44.141400' and lt '2018-08-14 15:06:44.141451'
2018-08-14 15:12:44 eventgen        INFO     MainProcess Start '2' generatorWorkers for sample 'test_sample.txt'

Looking at splunkd,

08-14-2018 15:19:41.588 +0000 INFO  LicenseUsage - type=Usage s="/opt/splunk/var/log/splunk/test_service.log" st=test_service_log h="ip-172-31-36-143" o="" idx="default" i="EAE584D7-DBF7-4B6F-819B-36BAD9EEE258" pool="auto_generated_pool_enterprise" b=84 poolsz=53687091201
08-14-2018 15:19:41.588 +0000 INFO  LicenseUsage - type=Usage s="/opt/splunk/var/log/splunk/test_service.log" st=test_service_log h="ip-172-31-36-143" o="" idx="default" i="EAE584D7-DBF7-4B6F-819B-36BAD9EEE258" pool="auto_generated_pool_enterprise" b=84 poolsz=53687091201
08-14-2018 15:20:42.620 +0000 INFO  LicenseUsage - type=Usage s="/opt/splunk/var/log/splunk/test_service.log" st=test_service_log h="ip-172-31-36-143" o="" idx="default" i="EAE584D7-DBF7-4B6F-819B-36BAD9EEE258" pool="auto_generated_pool_enterprise" b=172203 poolsz=53687091201

Can you give me some pointers on what may be happening ?

kthammireddygar
Path Finder

if nothing changed on eventgen Config, it may be that there is no disk or disk full.

0 Karma

jkat54
SplunkTrust
SplunkTrust

search index=_internal log_level=warn* OR log_level=error see if that reveals more details.

If I had to guess, something could be sending everything to null queue.

Is it by chance "bursting" every time you restart splunk?

0 Karma
Get Updates on the Splunk Community!

Join Us for Splunk University and Get Your Bootcamp Game On!

If you know, you know! Splunk University is the vibe this summer so register today for bootcamps galore ...

.conf24 | Learning Tracks for Security, Observability, Platform, and Developers!

.conf24 is taking place at The Venetian in Las Vegas from June 11 - 14. Continue reading to learn about the ...

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...