Splunk App for Unix: Confusion

[OldManEd]

I am having a problem with my Splunk environment and resource allocation. I’ve been using the *nix app that someone loaded on my Search Head a long time ago to try and analyze what’s going on. It was version 4.5. From what I’ve read, there is a newer, better version of the app called “Splunk App for Unix”. From the documentation I read that I can’t do an update from 4.5 to 5.0.1, but I can run them both in parallel, point the new version to the newer data to get historical data and then kill 4.5 when I’m done.

I loaded version 5.0.1, but I’m having multiple issues with the app.

First, when I go to the “Metrics” page, I’m seeing multiple entries for the name of my Search Head. How and what do I reconfigure to clean this up?

Second, when I select “Select All” under “Host Filter”, in the “Metrics Viewer” side of the page, the time frame only goes to “Last 24 Hours”. Also, only data for 1 server shows data although I have 3 listed.

Under “Settings” I reset “Unix index(es)” by adding “index=main”. But that didn’t help at all. Is it possible to reset this to whatever index the *nix version 4.5 kept its data in? And what would that be?

~Ed

[markdaku]

I'll be entirely honest.

The Splunk app for UNIX is really badly broken. The list of issues with this app is far to long to even start to discuss here. In general this app is not usable by the casual user. It's far to heavy on all parts of splunk from the browser to the indexer and search head.

The TA is pretty good. It gathers data fairly well. My recommendation is to install the TA and then build your own dashboards from the data. There are a few issues with the TA as it tends to toss data out. For example the cpu.sh provides good data, but for some reason only the averages across all cpu's are reported. There are other areas of lost data but nothing to serious.

Search is your friend here. Leverage the work done on inputs from the TA and just delete the actual front end app.

[araitz]

I'm sorry you are having problems with the app. If you could provide any level of specificity regarding what problems you have had, that would be great. We are ready to accept your criticism if you can provide us with specific, actionable feedback.

[the_wolverine]

It was extremely confusing to find two new apps residing in the splunk_app_for_nix: SA-nix and Splunk_TA_nix. Previously SA and TA are installed on indexers in a SH_POOL environment.

In this app:
"The Splunk Supporting Add-on for Unix and Linux provides search macros, reports, and other configurations for the Splunk App for Unix and Linux. A Splunk App for Unix and Linux instance must have the Supporting Add-on installed in order to function correctly."

What the documentation fails to mention is that the SA- and TA are INSTALLED AUTOMATICALLY by the splunk_app_for_nix. This was extremely frustrating to figure out where they were coming from. Please update the documentation -- it would be a problem for organizations who have tightly controlled configuration management to have software installed automatically.

[OldManEd]

I guess I was confused from the documentation where I read the following;

From version 4.6.x and earlier to version 5.0.1

There is no supported upgrade path from version 4.6 of the Splunk App for Unix and Linux to this version. However, you can run both version 4.6 and this version simultaneously, if you so choose.

Anyway, is there a process to remove the older *Nix application or do I simply delete in from the app directory?

[araitz]

Nope, the old and new versions of the app are not compatible with each other. The problem is that the old version of the app ships with its own inputs, while the new version of the app depends on knowledge and inputs from the TA. I thought the docs were pretty clear, but in any case, when you install the app on the search head, it auto installs the necessary versions of the SA and TA. You need the TA on the indexers and forwarders as well, as it contains input-time and index-time configurations.

[OldManEd]

NEW ISSUE:

I thought older versions of Splunk *nix could live together. I'm seeing this notice when I try to run the installed older version app. Any ideas on what's going on here?

The app "Splunk Add-on for *Nix" is installed on this system.

The Splunk *nix App and the "Splunk Add-on for *Nix" app cannot exist together on the same Splunk instance.

Please click on Manage Apps to disable the conflicting app, then remove "Splunk Add-on for *Nix" from $SPLUNK_HOME/etc/apps and restart Splunk.

[OldManEd]

OK, I’m getting “wrapped around the axle here” with the documentation and could use a little clarification. Just to verify,

The main app, “Splunk App for Unix and Linux”, gets loaded on the Splunk Search Head and is a stand-alone.

The “Splunk Add-on for Unix and Linux” gets loaded on the instance indexers and forwarders and is a stand-alone.

The “Splunk Supporting Add-on for Unix and Linux” gets loaded on the indexers only and is used with “Splunk Add-on for Unix and Linux”.

Are these assumptions correct?

[araitz]

Ah yes. Yeah, you'll need to install the add-on on your indexers and enable the inputs you want, either through the conf files, the setup.sh command line script, or the TA's very basic UI.

[OldManEd]

Oh heck, I tried what you suggested and no, I only saw the search head under "hosts". But, going over the documentation, I never loaded the "add-on" to the indexers. I'll try that next.

But I'm still only seeing data for the last 24 hours for the search head.

[araitz]

I believe you want index=os. By default, that is where the Unix Add-on sends data.

If you go to the search interface and search for index=os, do you see your indexers?

[OldManEd]

UPDATE: I was able to go to Settings>Categories off the top tabs and under "groups, delete the server entries I did not want. This seemed to work fine. I was also able to create new groups with a unique name and add my search head to it. The problem now is I can't see any of the indexers associated with this instance. Can someone tell me how to accomplish that?