Monitoring Splunk

Deployment Server seen as Unavailable/Offline by other members

romulusc
New Member

Hi,

I'm currently stuck on an issue where my Deployment Server seems to not be communicating with the same group as the other members. The server is listed as "Offline" or "Unavailable" when accessing the Monitoring Console. I believe this may be effecting clients communicating with the deployment server receiving apps and config information. I tried to see if any services are currently not running on the Deployment Server but nothing has stood out. The Deployment Server only lists itself when I access the Monitoring Console (it's in Distributed Mode). Not sure what other details are needed as I am kind of new at this but included some pictures to help describe my situation.

Any assistance is much appreciated. Thank you.

alt text

alt text

Labels (1)
0 Karma

woodcock
Esteemed Legend

On the MC, go to Settings -> Distributed search -> Search peers and see what it tells you. Try disabling the existing entry for DS and try re-adding it. Try adding it by IP address instead of hostname (perhaps you have a DNS problem).

0 Karma

woodcock
Esteemed Legend

Don’t use telnet to test SSL/TLS connections; run this from the DS client:

openssl s_client -connect deploymentserverhostname:8089

Alternatively this:

/opt/splunk/bin/splunk cmd openssl s_client -connect deploymentserverhostname:8089
0 Karma

romulusc
New Member

@woodcock

So I did that from one of the other clients and the License server and got the certificate thumbprint and the other details which should signify a successful connection using SSL/TLS

So this means there shouldn't be a communication issue right?

0 Karma

woodcock
Esteemed Legend

You should either get a password prompt or a CONNECTED response.

0 Karma

Vijeta
Influencer

@romulusc Check the splunkd logs or write this query and see if you find any issues. (host should be your Deployment server hostname)

index=_internal sourcetype=splunkd host=<your deployment server>
0 Karma

romulusc
New Member

Yeah I entered that query and received over 320,000+ events...I think that's a little much don't you think?
I tried using the "Interesting fields" options on the left to group "deploy-server" but that still yielded 10,000+ events just for today I think. Anything in particular I'm looking for?

0 Karma

woodcock
Esteemed Legend

Check splunkd.log it is usually quite specific.

0 Karma

romulusc
New Member

Sorry but is there a specific area I am looking for in this log? Also is the one you're referring to in the \var\log directory?

0 Karma

woodcock
Esteemed Legend

Here: /opt/splunk/var/log/splunk/splunkd.log
Go there and find the specific logs and report back.

0 Karma

romulusc
New Member

Again I do not know what I am looking for in this extensive log. I see a bunch of lines like this

06-11-2019 12:42:08.685 -0400 INFO
ClientSessionsManager:Listener_AppEvents
- Received count=7 AppEvents from DC ip=10.101.201.4
name=EA8EBD62-4291-44AB-BD9C-DE50036B75BF
06-11-2019 12:42:09.376 -0400 INFO
ClientSessionsManager:Listener_AppEvents
- Received count=7 AppEvents from DC ip=10.102.201.3
name=757FF50E-A587-4ADD-8C7A-3DF94FE6A3D9
06-11-2019 12:42:38.508 -0400 INFO
ClientSessionsManager:Listener_AppEvents
- Received count=3 AppEvents from DC ip=10.102.204.1
name=F3273119-1099-4842-9D5D-0794B2CBE9B1
06-11-2019 12:42:38.784 -0400 INFO
ClientSessionsManager:Listener_AppEvents
- Received count=3 AppEvents from DC ip=10.101.204.2
name=8DB575D9-937A-4BD8-B8A7-71C964BD9FBF
06-11-2019 12:44:14.292 -0400 INFO
ClientSessionsManager:Listener_AppEvents
- Received count=7 AppEvents from DC ip=10.102.201.4
name=B1285E03-0720-4A53-BFF4-F52C17BBC4D3
06-11-2019 12:44:43.722 -0400 INFO
ClientSessionsManager:Listener_AppEvents
- Received count=3 AppEvents from DC ip=10.101.204.3
name=CD69ADA1-AE46-4551-A27D-6975F3694934
06-11-2019 12:44:54.706 -0400 ERROR
ExecProcessor - message from
""C:\Program
Files\Splunk\etc\apps\Splunk_TA_microsoft_ad\bin\runpowershell.cmd"
nt6-health.ps1" Get-Item : Cannot find
path 06-11-2019 12:44:54.706 -0400
ERROR ExecProcessor - message from
""C:\Program
Files\Splunk\etc\apps\Splunk_TA_microsoft_ad\bin\runpowershell.cmd"
nt6-health.ps1"
'HKLM:\System\CurrentControlSet\Services\NTDS\Parameters'
because it does not 06-11-2019
12:44:54.706 -0400 ERROR ExecProcessor
- message from ""C:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft_ad\bin\runpowershell.cmd"
nt6-health.ps1" exist. 06-11-2019
12:44:54.706 -0400 ERROR ExecProcessor
- message from ""C:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft_ad\bin\runpowershell.cmd"
nt6-health.ps1" At C:\Program
06-11-2019 12:44:54.706 -0400 ERROR
ExecProcessor - message from
""C:\Program
Files\Splunk\etc\apps\Splunk_TA_microsoft_ad\bin\runpowershell.cmd"
nt6-health.ps1"
Files\Splunk\etc\apps\Splunk_TA_microsoft_ad\bin\powershell\nt6-health.ps1:66
06-11-2019 12:44:54.706 -0400 ERROR
ExecProcessor - message from
""C:\Program
Files\Splunk\etc\apps\Splunk_TA_microsoft_ad\bin\runpowershell.cmd"
nt6-health.ps1" char:15 06-11-2019
12:44:54.706 -0400 ERROR ExecProcessor
- message from ""C:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft_ad\bin\runpowershell.cmd"
nt6-health.ps1" + $SchemaInfo =
Get-Item 06-11-2019 12:44:54.706
-0400 ERROR ExecProcessor - message from ""C:\Program
Files\Splunk\etc\apps\Splunk_TA_microsoft_ad\bin\runpowershell.cmd"
nt6-health.ps1"
"HKLM:System\CurrentControlSet\Services\NTDS\Parameters"
06-11-2019 12:44:54.706 -0400 ERROR
ExecProcessor - message from
""C:\Program
Files\Splunk\etc\apps\Splunk_TA_microsoft_ad\bin\runpowershell.cmd"
nt6-health.ps1" +

06-11-2019 12:44:54.706 -0400 ERROR
ExecProcessor - message from
""C:\Program
Files\Splunk\etc\apps\Splunk_TA_microsoft_ad\bin\runpowershell.cmd"
nt6-health.ps1"
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
06-11-2019 12:44:54.706 -0400 ERROR
ExecProcessor - message from
""C:\Program
Files\Splunk\etc\apps\Splunk_TA_microsoft_ad\bin\runpowershell.cmd"
nt6-health.ps1" + CategoryInfo

: ObjectNotFound:
(HKLM:\System\Cu...NTDS\Paramete
06-11-2019 12:44:54.706 -0400 ERROR
ExecProcessor - message from
""C:\Program
Files\Splunk\etc\apps\Splunk_TA_microsoft_ad\bin\runpowershell.cmd"
nt6-health.ps1" rs:String)
[Get-Item], ItemNotFoundException
06-11-2019 12:44:54.706 -0400 ERROR
ExecProcessor - message from
""C:\Program
Files\Splunk\etc\apps\Splunk_TA_microsoft_ad\bin\runpowershell.cmd"
nt6-health.ps1" +
FullyQualifiedErrorId :
PathNotFound,Microsoft.PowerShell.Commands.GetIt
06-11-2019 12:44:54.706 -0400 ERROR
ExecProcessor - message from
""C:\Program
Files\Splunk\etc\apps\Splunk_TA_microsoft_ad\bin\runpowershell.cmd"
nt6-health.ps1" emCommand
06-11-2019 12:44:54.706 -0400 ERROR
ExecProcessor - message from
""C:\Program
Files\Splunk\etc\apps\Splunk_TA_microsoft_ad\bin\runpowershell.cmd"
nt6-health.ps1" 06-11-2019
12:44:54.711 -0400 ERROR ExecProcessor
- message from ""C:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft_ad\bin\runpowershell.cmd"
nt6-health.ps1" You cannot call a
method on a null-valued expression.

Does this mean anything to you?

0 Karma

satyaallaparthi
Communicator

Hello,

Can you go to monitoring console-->Settings and general setup.. Distributed and reset---> apply and restart server.. might be that will work.

Thanks,

0 Karma

romulusc
New Member

Am I doing the "Reset All Settings" on the Deployment Server? That'll remove all of the Forward Management and all that stuff correct?

0 Karma

satyaallaparthi
Communicator

can you share the screen shot if u can?
yes in deployment server..monitoring console..general setup and distributed and then reset there and apply settings..

0 Karma

romulusc
New Member

Well I did that and it is still showing the same status. And yes I stopped and started the Splunk service

0 Karma

satyaallaparthi
Communicator

did you check all firewalls or ports or communication between those server.. do telnet and see if you can find something..

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...