All Apps and Add-ons

Monitoring console, health check not responsive - Could not create search.

tvergov
Explorer

Hello Splunkers 🙂

Here is my case that is driving me crazy already. I'll not going too deep in details so will try to make small overview on the situation.

When i added SH and HF to a single instance environment. The single instance was left as indexer role and License master and deployment server. Also added 2 more indexes and 3 apps deployed and after 2 days the single instance (indexer) stopped responding as usual. I mean it's accessible via web interface but searches are not working but still getting data.
I decided that since it's older version than the other 2 newly added server to upgrade and to solve both problems in one shot.
Upgrade successful but same problem. I didn't know where to start but i saw the whole operating system was causing problems since the file system has permission problems.
Then
I decided to drop this server from the picture and focus on the Search Head where to move the Deployment server and license master roles. But before i get there the instance started to act weird the same way as the indexer.

Let me explain what's weird:
When i go to monitoring console and check overview it doesn't show anything except licence usage, disk usage and indexing rate and even that is not showing evertime and more like in rare cases.
Health check is not working and stops at 5 or 7% after the very first step.
the splunkd.log is not showing anything that can make sense to troubleshoot.
beside that the search is working just fine on the SH and the indexer is getting data in very successfully.

As far as i can see Monitoring console is not responding and cannot load any of the searches and ends up with "Could not create search".

Let me know what are your thoughts on this and any advice what to troubleshoot to make things as healthy as possible.

best,
T

0 Karma

tvergov
Explorer

additional to that this is typical behavior when the server doesnt have enough resources like enough CPU power or RAM available. add more before doing more digging like wasting time in troubleshooting.

0 Karma

tvergov
Explorer

Updates:
I believe the issue is due to incorrect props or transforms for one of the TAs that i was implementing out of the box.
I removed SH (deleted) and stopped incoming 9997 port from the app. I did that yesterday but still the indexer was unresponsive but today without any more actions taken it's acting normal but i'll not start the inputs prior finding out what's the root cause.

As i said speculating the TA add-on for one of the firewalls i'm testing as new inputs comming from the HF.

Here is picture taken yesterday giving me signal that something with the data is not right. Even more strange is that this is reported on the SH where there is no data incoming.... alt text

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...