Nest Add-on for Splunk: Issue with Zombies/invalid procs

[jlanders]

Didn't see a link in the description for code hosting/issue submission so here you go...

The script to pull data had issues on my system because of a lingering process that was invalid. The script attempts to find the splunkd process ID but stops at the first one and does no further checks. This resulted in me getting locked out from the API and data not pulling. I added a little check in the script, using the same os.kill concept used in the check_splunk method. This is around line 117 in the script. I'm still testing to ensure it fixes my problem but wanted to bring it to your attention!

for pid in pids:
    cmd = open(os.path.join('/proc', pid, 'cmdline'), 'rb').read()
    try:
        if 'splunkd' in cmd:
            try:
                # Do a test, see if the process is valid
                os.kill(int(pid), 0)
                splunk_pid = pid
                break
            except OSError:
                # Continue to the next pid
                continue
        else:
                splunk_pid = "SPLUNK NOT RUNNING"
    except IOError: # proc has already terminated
       continue

[jlanders · Accepted solution]

I also ended up having to modify the if/then check to match my environment more:

if 'splunkd' in cmd:

to:

if 'splunkd' in cmd and '-p' in cmd and '8089' in cmd and 'start' in cmd:

This is likely because my test system runs Splunk and the Splunk UF so I needed to differentiate between the two. Anyway, the combination of updating the if statement and adding the try/except to validate the process ID seems to have fixed the issue for me.

[aaraneta_splunk]

Hi @jlanders - Thanks for bringing this issue to light. Did you want to leave your question open for now to see if the add-on author wants to chime in? Or would you want me to convert your comment as an answer to accept?

[jlanders]

Sure, you can convert it. I mostly just wanted to bring up the bug with the author.

[aaraneta_splunk]

I'm going to leave this unaccepted for now in case other users want to chime in. But after a little while, either you (or I) can accept your answer if there's no further activity. Thanks again 🙂