Python Error in restarting Splunk

[teddyidc1101]

upon restarting splunk forwarder, we are encountering the error below..

python: can't open file '/opt/splunkforwarder/lib/python2.7/site-packages/splunk/clilib/cli.py': [Errno 2] No such file or directory

what could be the reason and fix for this?

[xpac]

Hey, try running this:

/opt/splunkforwarder/bin/splunk validate files

It will check your install for missing files, and give you a hint what is wrong.

[teddyidc1101]

By the way, here are the items we looked into to check...did we miss anything?
- Checked the logs and confirmed that the events stopped coming in at 5/21 around 12:30MN
- Check on crash logs…none found
- Checked forwarder management, the server was not a client
- Executed Splunk start forwarder instead of Splunk restart - failed
- Obtained the following logs in /opt/splunkforwarder/var/log/splunk of the server : python.log – not found for splunkforwarder, splunkd-utility.log, splunkd.log
- When we checked the actual file the restart is looking for, file was not found, as indicated in the error does not exist -
python: can't open file '/opt/splunkforwarder/lib/python2.7/site-packages/splunk/clilib/cli.py': [Errno 2] No such file or directory
**Does this file really exist?

What is the best way to revive the UF.**

[MuS]

Did you see my comment I posted just one minute before you posted your comment?

[teddyidc1101]

yeah i did...did we miss any steps with the one listed above?
the client installed the UF in their servers. we dont have access on them. but upon checking via splunk search, UF version is 7.0.

checking the UF , we dont have the Python in there but what the team did was linked python and python 2 to the /opt/splunkforwarder/bin. Is this ok?

upon checking in server via command "whereis python" , it listed /opt/splunkforwarder/bin/python.
is this ok?

[MuS]

you did what???
Please revert those changes. The universal forwarder does not need Python by default. It only needs Python if you want to run a python script using the universal forwarder.

Still this is really weird .....

[teddyidc1101]

ok thanks...the linking happend 5/10 and UF stopped in 5/21, so i am not sure if they are connected at all...if we remove the link, does it impact the start/restart of the splunk instance? will it restart successfully? is reinstalling the forwarder an option here?

[xpac]

Yeah, that's sounds rather strange... If I were you, I'd wipe the complete /opt/splunkforwarder directory and reinstall the forwarder from scratch. Be aware, however, that this might lead to duplicate events for some time.

[teddyidc1101]

ok thanks @xpac...i'm already considering it as i'm running out of resources to look into 🙂 thanks so much for sharing you expertise!

[teddyidc1101]

I triend running the command and its saying that
Result: Validating installed files against hashes from '/opt/splunkforwarder/splunkforwarder-7.0.0-c8a78efdd40f-linux-2.6-x86_64-manifest'
All installed files intact.

[MuS]

Wait a second, you get this error /opt/splunkforwarder/lib/python2.7/site-packages/splunk/clilib/cli.py': [Errno 2] No such file or directory ... but this is a universal forwarder ?!
Splunk universal forwarder does not ship Python, the only Splunk software that ships Python is the full enterprise install.

How did you install this version of Splunk; What package did you use and what command to install it?

cheer, MuS

[MuS]

did you use the /opt/splunkforwarder/bin/splunk start command?

[teddyidc1101]

yes we did but still getting the error.