All Apps and Add-ons

Problem with Hydra scheduler: sprayReadyJobs exception

halr9000
Motivator

I am troubleshooting my VMware app configuration, and am faced with this error in splunkd.log:

014-04-08 19:24:52,596 ERROR [ta_vmware_collection_scheduler://puff] Problem with hydra scheduler ta_vmware_collection_scheduler://puff:
 Something went unstable with a Hydra asset, typically due to a REST timeout or misconfiguration, rebuilding and validating entities
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/SA-Hydra/bin/hydra/hydra_scheduler.py", line 1718, in run
    collection_manifest.sprayReadyJobs(self.node_manifest)
  File "/opt/splunk/etc/apps/SA-Hydra/bin/hydra/hydra_scheduler.py", line 512, in sprayReadyJobs
    raise ForceHydraRebuild
ForceHydraRebuild: Something went unstable with a Hydra asset, typically due to a REST timeout or misconfiguration, rebuilding and validating entities

Data does not appear to be coming in from the data collection node (DCN)

0 Karma
1 Solution

halr9000
Motivator

Received an answer from the lab:

This is hydra's internal rebuild. It is run whenever it is incapable of collecting data. This usually means you have one DCN and thatn DCN became unresponsive. Scheduler had no one to give jobs to and rebuilt itself.

Essentially if we can't really do anything we just turn ourselves off and on again.

So it appears that the error indicates that Hydra is restarting itself because the DCN is inaccessible. Root cause could be network or system related on the DCN.

View solution in original post

halr9000
Motivator

Received an answer from the lab:

This is hydra's internal rebuild. It is run whenever it is incapable of collecting data. This usually means you have one DCN and thatn DCN became unresponsive. Scheduler had no one to give jobs to and rebuilt itself.

Essentially if we can't really do anything we just turn ourselves off and on again.

So it appears that the error indicates that Hydra is restarting itself because the DCN is inaccessible. Root cause could be network or system related on the DCN.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...