App and Add-on for cassandra cluster monitoring: Why are cassandra logs not generated or forwarded?

[jigarashah]

HI, I am trying to configure the App for cassandra cluster monitoring and Add-on for cassandra cluster monitoring to monitor cassandra cluster. I have universal forwarder on each node.

I have installed Add-on for cassandra cluster monitoring on universal forwarder.

I get following log in splunkd.log

01-16-2017 13:58:15.883 +0000 INFO  ExecProcessor - New scheduled exec process: /opt/splunkforwarder/etc/apps/Splunk_TA_cassandra/bin/cache.pl
01-16-2017 13:58:15.883 +0000 INFO  ExecProcessor -     interval: 60000 ms
01-16-2017 13:58:15.883 +0000 INFO  ExecProcessor - New scheduled exec process: /opt/splunkforwarder/etc/apps/Splunk_TA_cassandra/bin/compactionhistory.pl
01-16-2017 13:58:15.883 +0000 INFO  ExecProcessor -     interval: 60000 ms
01-16-2017 13:58:15.883 +0000 INFO  ExecProcessor - New scheduled exec process: /opt/splunkforwarder/etc/apps/Splunk_TA_cassandra/bin/cpu_perf.pl
01-16-2017 13:58:15.883 +0000 INFO  ExecProcessor -     interval: 60000 ms
01-16-2017 13:58:15.883 +0000 INFO  ExecProcessor - New scheduled exec process: /opt/splunkforwarder/etc/apps/Splunk_TA_cassandra/bin/mem_perf.pl
01-16-2017 13:58:15.883 +0000 INFO  ExecProcessor -     interval: 60000 ms
01-16-2017 13:58:15.883 +0000 INFO  ExecProcessor - New scheduled exec process: /opt/splunkforwarder/etc/apps/Splunk_TA_cassandra/bin/netstats.pl
01-16-2017 13:58:15.883 +0000 INFO  ExecProcessor -     interval: 60000 ms
01-16-2017 13:58:15.883 +0000 INFO  ExecProcessor - New scheduled exec process: /opt/splunkforwarder/etc/apps/Splunk_TA_cassandra/bin/process.pl
01-16-2017 13:58:15.883 +0000 INFO  ExecProcessor -     interval: 60000 ms
01-16-2017 13:58:15.883 +0000 INFO  ExecProcessor - New scheduled exec process: /opt/splunkforwarder/etc/apps/Splunk_TA_cassandra/bin/readwrite.pl
01-16-2017 13:58:15.883 +0000 INFO  ExecProcessor -     interval: 60000 ms
01-16-2017 13:58:15.883 +0000 INFO  ExecProcessor - New scheduled exec process: /opt/splunkforwarder/etc/apps/Splunk_TA_cassandra/bin/status.pl
01-16-2017 13:58:15.883 +0000 INFO  ExecProcessor -     interval: 60000 ms

I don't know if its issue with the forwarder which is not sending data to server or if the server is not receiving it.

I think server is receiving other data fine when I search for

host=<<host name>> index=_internal

I do get results but I don't see any of Cassandra source / sourcetype

Is the Add-on for cassandra cluster monitoring working? where does it write logs? in Perl script, i just see print(). does this mean its only STDOUT? how do i check where is the issue? in forwarder or at receiver?

[hunters_splunk]

Hi jigarashah,

You must install the Cassandra add-on on a heavy forwarder or indexer because the add-on contains perl script, which is not supported on a universal forwarder.

For more information about where to install Splunk add-ons, please refer to documentation:
http://docs.splunk.com/Documentation/AddOns/released/Overview/Wheretoinstall

Hope this helps. Thanks!
Hunter

[jigarashah]

Ah...Did see works with Splunk Enterprises....

Anyways, installed on primary search node (which is also indexer). Gets installed but still no logs.

Tried all steps in http://docs.splunk.com/Documentation/AddOns/released/Overview/Troubleshootadd-ons

Still no luck. Dont know if addon is not able to execute perl scripts or not able to push data...no 'cassandra' logs in splunkd.log either.

Really appreciate your help...why debugging splunk addon behaviour is so complicated ?